...
- explicity list every file/directory in transfer_input_files (it doens't grok regexps). This would be a large list . E.g.
- transfer_input_files = "working/VIP_iter0.gridwt, working/VIP_iter0.pb.tt0, working/VIP_iter0.psf.tt0, working/VIP_iter0.psf.tt1, working/VIP_iter0.psf.tt2, working/VIP_iter0.sumwt.tt0, working/VIP_iter0.sumwt.tt1, working/VIP_iter0.sumwt.tt2, working/VIP_iter0.weight.tt0, working/VIP_iter0.weight.tt1, working/VIP_iter0.weight.tt2"
- Can transfer_input_fies take a manifest? E.g a file containing the list of files to transfer
- Make a temporary director on the submit host, and transfer that (possibly tarring it up)
- Set the inputs and outputs for both data and working as a variables in the unified DAG file. The task.sh script uses rsync to merge the various data_inputs together into one data directory and the various working_inputs together into one working directory. Then at the end, task.sh moves data to data-<dagstep> and working to working-<dagstep> and the appropriate dirs/files from these are transferred back to the submithost. The result of all this is that the data needed as an input for a step (E.g. Task08) may need to be combined from multiple places (initial data and data output from Task07)
To Do
- DONE: write .log, .out and .png files one level up so they are not in the working directory and therefore not copied to execute hosts.
- DONE: add rm -f *.last to the sh script?
- Figure out how to not copy SYSPOWER in the MS. Presumably we can just cp /dev/null SYSPOWER/table.f0 and cp /dev/null SYSPOWER/table.f0i
- DONE: Task24 and Task25 swapped with 8cores so they need to run with fewer cores. So I may need to make another variable to pass to the sh script for this.
- Task19 needs to be unwraveled from NRAO filesystems.
- I don't like using the name Task as that has meaning to CASA. A better term might be Step as in DAG Step.
- Look into running Task22 at the same time as another Task. If it is possible, perhaps this can be done for some other tasks as well. Note, I specificly avoided using the work parallel here so as not to confuse this idea with using mpicasa.
- Figure out how to not copy SYSPOWER in the MS. Presumably we can just cp /dev/null SYSPOWER/table.f0 and cp /dev/null SYSPOWER/table.f0i
Task01
Doesn't alter the MS
...