Versions Compared

Old Version 6

changes.mady.by.user James Robnett [X]

Saved on

compared with

New Version Current

changes.mady.by.user James Robnett [X]

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

HTCondor area: High Throughput Computing with HTCondor


Attendees: James, Brian

Agenda (Primarily focused on roadmap)

  • Coordination/Logistics
  • Schedule (broad strokes, need to flesh out details)
    • Roadmap (thanks to Mark Whitehead) page set up here following proposed DMS model:  SE Imaging Operations Roadmap
    • Need to work with Pipeline and SSA to further flesh out Roadmap, goal is to have a first full draft done by end of month.  
  • Pipeline
    • Need list of existing tickets tied to SE imaging (I can go in and add labels)
    • Is the plan to make a separate task or leverage existing task
    • how to manage pybdsf and other external python apps (largest concern), could use sequence manager class to iterate over tclean in different ways, may be able to use
    • Can each tclean() call be checkpointed and restarted.
    • Packaging question w.r.t. pipeline for external packages
    • Can some external scripts be reimplemented in CASA pipeline or always externally called scripts
    • Weblog completely uncertain 
  • SSA
    • Need list of existing tickets tied to SE imaging (I can go in and add labels)
    • Split pipeline between data selection and mstransform and imaging (should this be done in the pipeline recipe, ie hifv_splitms() or some such)
    • HTCondor effort, when to start, how to coordinate, how to set up test environment (want to start in June, need to discuss what it should look like)
    • How to incorporate into workflow manager vs vlass manager
    • How to track SE-continuum products and their possible inclusion in SE-cube imaging (relevant in the short term,  inputs to Cube need to be known so they can drive continuum products  (IDs, name formats etc)
  • SCG
    • Return to effort to create an external imaging approach with decomposition along the major/minor cycle and SPWs. Imaging with HTCondor
    • Assist other groups, set up test environments
    • Coordinate with CHTC NRAO-CHTC HTCondor collaboration
    • Can we split the pipeline in to the PSF and MASK steps and the final imaging steps, has utility for the 

...