...
- What are the clever solutions to submitting N different DAG jobs with each having different parmeters?
- T10t34
- J220200-003000
- bin, working, data
- J220600-003000
- bin, working, data
- ...
- J220200-003000
- T10t35
- J170743-393000
- bin, working, data
- J171241-383000
- bin, working, data
- ...
- J170743-393000
- ANSWERS:
- INCLUDE syntax for DAGs
- include syntax for submit files
- make a template of files
- use a PRE script that populates things
- usedagdir
- T10t34
- It seems that when using DAGs the recommended method is to define variables in the DAG script instead of submit scripts. This makes sense as it allows for only one file, the DAG script, that needs to be edited to make changes. But, is there a way to get variables into the DAG script from the command line or environment or an include_file or something?
- ANSWER: There is an INCLUDE syntax but there is no command-line or environment variable way to get vars into a DAG.
- We are starting to run 10s of jobs in CHTC requiring 40GB as part of a local DAG. Are there any options we can set to improve their execution chance. What memory footprint (32, 20, 16, 8GB) would significantly improve their chances.
- ANSWER: only use +LongJobs if the job needs more than 72 hour, which is the default "walltime".
- How can we set AWS Tags with condor_annex? We'd like this to track jobs and set billing tags.
- Launch Templates didn't work.
- Possibly use aws-user-data to condor_annex
...