Following the successful conclusion of the VIP project we now have a model for processing a statistically significant percentage of the sky. There are two tracks we need to follow to get to operational scale imaging:
- Implement the example script within the CASA Pipeline framework
- Demonstrate a model for at-scale processing, where at scale is roughly 1000 square degrees imaged per month.
There will be at least 3 different examples of the workflow being implemented at any given time in the path to operations.
- Scripted Pipeline: Example python script provided by VIP. This is the plan of record and should be kept up to date with the other forms until deprecated. (https://gitlab.nrao.edu/jmarvil/vlass-imaging-project/)
- CASA Pipeline: standard pipeline in both a pythonic and XML based form
- External processing script: initially based off of (1), hopefully it will converge with (2) but may need to be a separately maintained entity due to external constraints.
There are 3 main tracks of development
- Characterize resource and runtime characteristics and develop external processing mode
- Runtime and memory per tclean call
- Questions regarding creation vs distribution of cfcache
- Questions of decomposition axis (major/minor cycle, per SPW gridding, per SPW+W gridding)
- Questions regarding distribution of CASA software stack (per sq dg * per SPW per W gridding = 30 PBytes of CASA tar ball distribution)
- Port scripted pipeline to CASA pipeline format
- CASA 6 based ?
- Questions regarding pybdsf incorporation
- Implement CASA pipeline based execution within SSA workflows
- Pybdsf inclusion
- Tracking split calibrated MS temporary secondary products
- Tracking state of external processes (TBD)