...
- DONE: write .log, .out and .png files one level up so they are not in the working directory and therefore not copied to execute hosts.
- DONE: add rm -f *.last to the sh script?
- DONE: Task24 and Task25 swapped with 8cores so they need to run with fewer cores. So I may need to make another variable to pass to the sh script for this.
- DONE: re-create my DOT graph after finishing task01-25-parallel-dag4. Also make a PDF instead of a PS.
- DONE: Craft parallel-dag5 with the concept of running tclean at CHTC and everything else locally.
- DONE: I don't like using the name Task as that has meaning to CASA. A better term might be Step as in DAG Step.
- DONE: Figure out how to not copy SYSPOWER in the MS. Presumably we can just cp /dev/null SYSPOWER/table.f0 and cp /dev/null SYSPOWER/table.f0i I am doing this with my dag5 test now.
- Task19 needs to be unwraveled from NRAO filesystems. Actually maybe not. This can be a task we always run here. But it does need to move into a VLASS area instead of Josh's home acocunt.
- Setup testpost-serv-1 with Lustre access over IB so we can start submitting to CHTC.
- Need to make the MS in my DAG script a variable. Right now I specify data/VLASS.../table.f23_TSM1
Possible Improvements
- Task04 Task03 could perhaps be run concurrently with Task01 as long as Task05 Task04 is run after both.
- Task13 could perhaps be run concurrently with Task12.
- Task14 Task13 could perhaps be run concurrently with Taks12 Taks11 and Task13 Task12 as long as Task15 Task14 is run after both Task12 Task11 and Task14Task13.
- Task22 Task21 could perhaps be run concurrently with Task12 Task11 or later as long as Task23 Task22 is run after both Task22 Task21 and Task13Task12.
Task01 - Step01
Doesn't alter the MS
...
This task could possibly run at the same time as Task01 except that I have combined this with Task05 Task04 which requires both Task01 and Task04Task34.
run_tclean( 'iter1', robust=-2.0, uvtaper="3arcsec" )
...
Could this run in parallel with one or more previous run_tclean calls like Task12Task11?
run_tclean( 'iter0d', datacolumn='corrected', cfcache=cfcache_nowb, calcres=False )
...
This task could possibly run at the same time as Task11 and/or Task12 or Task13 except that I have combined this with Task15 Task14 which requires both Task14 Task13 and Task12Task11.
run_tclean( 'iter1b', datacolumn='corrected', robust=-2.0, uvtaper="3arcsec" )
...
As far as I can tell at this point, ../data has not changed since Task11 Task10 (applycal).
Could this run in parallel with one or more previous run_tclean calls like Task16Task15?
run_tclean( 'iter2', datacolumn='corrected' )
...
This is just some python that deletes VIP_iter2.psf.* and copies VIP_iter0d.psf.* to VIP_iter2.psf.*. It is inefficient to ever make this task be its own DAG. I suggest it always be in the same DAG as Task22Task21.
- input: VIP_iter2.psf.*, VIP_iter0d.psf.*
- output: VIP_iter2.psf.*
...