...
HTCondor area: High Throughput Computing with HTCondor
Attendees: James,
Agenda (Primarily
...
focused on roadmap)
- Coordination/Logistics
- Agree to label all tickets within various groups with a VLASS-SE-OPS label plus state labels described here: Jira Labels and Component Lists
- Schedule (broad strokes, need to flesh out details)
- Roadmap (thanks to Mark Whitehead) page set up here following proposed DMS model: SE Imaging Operations Roadmap
- Need to work with Pipeline and SSA to further flesh out Roadmap, goal is to have a first full draft done by end of month.
- Pipeline
- Is the plan to make a separate task or leverage existing task
- how to manage pybdsf and other external python apps (largest concern), could use sequence manager class to iterate over tclean in different ways, may be able to use
- Can each tclean() call be checkpointed and restarted.
- Packaging question w.r.t. pipeline for external packages
- Can some external scripts be reimplemented in CASA pipeline or always externally called scripts
- Weblog completely uncertain
- SSA
- Split pipeline between data selection and mstransform and imaging (should this be done in the pipeline recipe, ie hifv_splitms() or some such)
- HTCondor effort, when to start, how to coordinate, how to set up test environment (want to start in June, need to discuss what it should look like)
- How to incorporate into workflow manager vs vlass manager
- How to track SE-continuum products and their possible inclusion in SE-cube imaging (relevant in the short term, inputs to Cube need to be known so they can drive continuum products (IDs, name formats etc)
- SCG
- Return to effort to create an external imaging approach with decomposition along the major/minor cycle and SPWs. Imaging with HTCondor
- Assist other groups, set up test environments
- Coordinate with CHTC NRAO-CHTC HTCondor collaboration
- Can we split the pipeline in to the PSF and MASK steps and the final imaging steps, has utility for the
...