...

HTCondor area: High Throughput Computing with HTCondor

Attendees: James, 

Agenda (Primarily a getting organized meeting)

• Purpose of group meeting
  • Coordinate all DMS effort to support VLASS SE imaging internally and externally
• Coordination/Logistics
  • Track everything here or just major targets and let groups track their own deliverables and JIRA tickets (I'd prefer the latter)
    • Answer: Track effort globally for confluence and then link into JIRA via label
  • Use of tags to make clearing house pages easy (ie select for all tickets in SSA/SCG/CASA/CASA-pipeline/ARDG with tag 'XXXX'
    • Answer: need a label james will come up with.
• Schedule (broad strokes, need to flesh out details)
  • Demonstrate 100 sq degrees via HTCondor external to pipeline by October 1st.
  • VIP script implemented as pipeline by ?  (would like October) (Likely Q2 FY21)
  • SSA natively uses HTCondor for SE job submissions by ?  (would like September or October) (Likely Q1FY21)
• Pipeline
  • Is the plan to make a separate task or leverage existing task
  • how to manage pybdsf and other external python apps (largest concern), could use sequence manager class to iterate over tclean in different ways, may be able to use
  • Can each tclean() call be checkpointed and restarted.
  • Packaging question w.r.t. pipeline for external packages
  • cfcache tracking is a workflow issue not a pipeline issue
  • Can some external scripts be reimplemented in CASA pipeline or always externally called scripts
  • Weblog completely uncertain 
• SSA
  • Split pipeline between data selection and mstransform and imaging (should this be done in the pipeline recipe, ie hifv_splitms() or some such)
  • HTCondor effort, when to start, how to coordinate, how to set up test environment (want to start in June, need to discuss what it should look like)
  • How to incorporate into workflow manager vs vlass manager
  • How to track SE-continuum products and their possible inclusion in SE-cube imaging (relevant in the short term,  inputs to Cube need to be known so they can drive continuum products  (IDs, name formats etc)
• SCG
  • Return to effort to create an external imaging approach with decomposition along the major/minor cycle and SPWs. Imaging with HTCondor
  • Assist other groups, set up test environments
  • Coordinate with CHTC NRAO-CHTC HTCondor collaboration
  • Can we split the pipeline in to the PSF and MASK steps and the final imaging steps, has utility for the 

...