I list each task here and what is necessary to run them in HTCondor. I am assuming this will be running without a shared filesystem and also without access to NRAO filesystems. So any call to /lustre/aoc or /users/<username> or other such things need to be altered to be site agnostic.
Every DAG or task creates .log, .out and maybe .png files that we want to keep. Also, .last files like tclean.last are often created. These are not necessary but can be usefull for debugging things. I assume that almost all tasks require the Measurement Set (MS). I question what tasks actually modify the MS. run_tclean() defaults to using the corrected datacolumn. Does that mean it is changing this column?
Task01
run_tclean( 'iter0', cfcache=cfcache_nowb, robust=-2.0, uvtaper='3arcsec', calcres=False )
- input: data
- Input: cfcache_nowb='/mnt/scratch/cfcache/cfcache_spw2-17_imsize16384_cell0.6arcsec_w32_conjT_psf_wbawp_False.cf'
- output: VIP_iter0.*
Task02
run_tclean( 'iter0b', cfcache=cfcache_nowb, calcres=False )
- input: data
- input: cfcache_nowb='/mnt/scratch/cfcache/cfcache_spw2-17_imsize16384_cell0.6arcsec_w32_conjT_psf_wbawp_False.cf'
- VIP_iter0b.* (but what else uses iter0b?)
Task03
mask_from_catalog(inext=inext,outext="QLcatmask.mask",catalog_search_size=1.5,catalog_fits_file='../VLASS1Q.fits')
- input: data
- input: VLASS1Q.fits
- output: mask_from_cat.crtf, VIP_QLcatmask.mask
Task04
run_tclean( 'iter1', robust=-2.0, uvtaper="3arcsec" )
- input: data
- output: VIP_iter1.*
Task05
replace_psf('iter1','iter0')
This is just some python that deletes VIP_iter1.psf.* and copies VIP_iter0.psf.* to VIP_iter1.psf.*. It is inefficient to ever make this task be its own DAG. I suggest it always be in the same DAG as Task04.
- input: VIP_iter0.psf.*, VIP_iter1.psf.*
- output: VIP_iter1.psf.*
Task06
run_tclean( 'iter1', robust=-2.0, uvtaper="3arcsec", niter=20000, nsigma=5.0, mask="QLcatmask.mask", calcres=False, calcpsf=False )
- input: data
- input: VIP_iter1.*, VIP_QLcatmask.mask
- output: VIP_iter1.*
Task07
run_tclean( 'iter1', calcres=False, calcpsf=False, savemodel='modelcolumn', parallel=False )
- input: data
- input: VIP_iter1.*
- output: VIP_iter1.*
- output: data
Task08
flagdata(vis=vis, mode='rflag', datacolumn='residual_data',timedev='tdev.txt',freqdev='fdev.txt',action='calculate')
replace_rflag_levels()
flagdata(vis=vis, mode='rflag', datacolumn='residual_data',timedev='tdev.txt',freqdev='fdev.txt',action='apply',extendflags=False)
flagdata(vis=vis, mode='extend', extendpols=True, growaround=True)
- input: data
- output: tdev.txt,. fdev.txt
Task09
statwt(vis=vis,combine='field,scan,state,corr',chanbin=1,timebin='1yr', datacolumn='residual_data' )
- input: data
- output: data
Task10
gaincal(vis=vis,caltable='g.0',gaintype='T',calmode='p',refant='0',combine='field,spw',minsnr=5)
- input: data
- output: data