You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

This page captures the high level planning for each wave of SRAO development.  Objectives are captured as Epics in Jira and a brief description is below.

Wave 3

Requirements To Pipeline and CASA

The SRDP requirements from the pipeline are divided into four main categories, VLA calibration, VLA spectral line calibration, VLA Imaging, and ALMA Imaging.

VLA Calibration Improvement

During Wave 2, we were able to achieve some of our highest priorities like enabling multi-band calibration for the pipeline. However the infrastructure work was more extensive than initially estimated and other development items could not be scheduled as a result. As such, we highlight what was done in Wave 2, carryover priorities from Wave 2 and new priorities. We also identify some stretch goals that may have some work done toward requirements definition/implementation during Wave 3 and could be Wave 4 priorities.

Highest Priorities

  • Completion of AQUA reports for VLA/VLASS*

  • Enhanced DTS issue identification and flagging (CASR-563)*

  • Improved calibrator RFI flagging heuristics (PIPEREQ-12; PIPE-939)*

  • Gain table flagging extensions

    • Ripple detection (bad switch problem) (PIPEREQ-11)

    • Gain outlier detection (PIPEREQ-10)

  • Target RFI Flagging (CASR-540)

  • Per-baseband delay solutions (PIPE-670)*

  • Spectral window mapping for gain calibration (PIPE-671)*

    • enables phase calibration if not enough S/N due to faint calibrator*

  • Multi-band weblog updates (PIPE-935)

  • Detect high delays (and scatter) on reference antennas (PIPE-1032)

Medium Priorities

  • Allow different bandpass calibrators per band (PIPE-669)

  • Antenna position corrections in a post-observation time window (CASR-556)

  • Fluxboot gain solution flagging (PIPEREQ-9)

  • Fluxboot QA metric adjustments (PIPEREQ-8)

  • Enable compression correction for S-band in PI pipeline

Low Priorities/If Time available

  • Generalize the hifv_circfeedpolcal pipeline task for all bands (polarization calibration)

  • Fix pipeline bugs to allow for experimental P-band calibration with the pipeline

* - denotes carryover requirement from last wave planning

VLA Spectral Line Calibration

Little progress was made toward spectral line calibration during Wave 2 implementation, but progress has been made in defining requirements. The VLA pipeline currently has a number of shortcomings with regard to spectral line calibration that require manual intervention. The primary goal is to develop a spectral line reduction recipe that can be run on datasets by PIs, as a recalibration request, or by the DAs for SRDP processing.

Highest Priorities

  • RFI flagging on uv-grid (CASR-496)

    • avoids flagging of bright spectral lines

  • Enable data weights to be determined from switched power (CASR-476)*

  • Design and set up independent spectral line calibration recipe that could be used by end users, triggered using recalibration workflow, or by SRDP DAs

  • Per-baseband delay solutions (PIPE-670) – cross-list with standard calibration*

Medium Priorities

  • Use MS tool hanning smoothing to enable per spw hanning smoothing (CASR-523; PIPE-672)*

  • Spectral window mapping for gain calibration (PIPE-671) – cross-list with standard calibration*

    • enables phase calibration if not enough S/N on narrow spws*

  • Cont.dat interpretation to treat non-specified spws as continuum-only

Low Priorities/If Time Available

  • Enable pipeline to use multiple MS files to perform continuum processing/imaging on one

    • ALMA has similar requirement for this cycle

  • Carry out spectral line optimized calibration on the other when spectral lines are a science goal set in PST – relies on a connection of OPT database to processing environment via SSA

* - denotes carryover requirement from last wave planning

It would be preferable to know if a dataset is meant to be used for spectroscopy and if so, what spectral windows are intended for spectroscopy. In the absence of this information, which needs to be passed from the PST into the metadata of the pipeline environment, we need the ability to process the entire dataset under the assumption that it is spectral line, while also creating a continuum-optimized reduction. Within this effort we will be exploring the possibility of using the switched power measurements to set the weights for the VLA data, which may be useful for both calibration and spectral line imaging.

VLA Imaging Pipeline

The work on a basic continuum imaging pipeline has been completed and continuum images can now be produced by the pipeline with proper imaging heuristics. However, the current pipeline has shortcomings that need to be improved upon. Automasking is not currently used (it was descoped and we bring it back in as a new priority) and cleaning is done without a mask to a high-threshold (5-sigma). Furthermore, work was not able to be done to detect mosaicked fields for the VLA. Finally, we need to enable a basic cube imaging pipeline.

Highest Priorities

  • Evaluate automasking heuristics for VLA continuum imaging pipeline

    • Main work will be in determining the optimal settings for the automasking

  • Detect mosaicked fields and perform joint deconvolution with mosaic gridder*

Low Priorities/If Time Available

  • Define requirements for the continuum imaging pipeline to use outlier fields to remove effects of bright sources rather than using large images

  • Develop heuristics for using wproject gridder for single pointing S and L-band images

  • Enable basic cube imaging

ALMA User-Defined Imaging


The ALMA User-Defined Imaging recipe is functioning well and we are now focused on adding additional features to the service. The primary new feature is user-specified angular resolution, the pipeline is now capable of this using the hifas_imageprecheck task, and work to the archive interface needs to be done to allow for parameter input and pass it to the pipeline.

Highest Priorities

  • Line + continuum imaging (CASR-509)*

  • Characterize taperability of ALMA data to validate limits of uv-tapering applied.

  • Define requirements to enable self-calibration*

  • Image QA using continuum-subtracted MFS images*

Medium Priorities

  • Enable experimental self-calibration of ALMA datasets.

Low Priorities/If Time Available

  • Define requirements to enable multi-configuration imaging with ALMA (12m-only)


Requirements To SSA

Highest Priorities

  • CARTA Integration

  • ALMA Image Product Ingestion

  • Image Product Display Improvements

  • AUDI Interface Enhancements

  • Large Project Support

  • Catalog Support

Medium Priorities

  • Data delivery enhancement

  • Astroquery support

  • Observatory Database

Low Priorities

  • New Archive Frontend

  • Image cutout service

  • Cache System


Detailed Requirements

  • Authentication & Authorization Improvements
    • Lock/Unlock icons of user interface should be accurate
    • Release status of ALMA files should be shown per-MOUS, not per-Project
    • ALMA MOUSes can be public, unreleased or released, not just public or private
    • Thumbnails/weblogs should only be shown if they are public, or if the user is authorized to see them

  • ALMA User-Defined Imaging: Additional options in the pipeline and interface
    • requirements and suggested implementation under discussion
      • Add additional call to MFS imaging on continuum-subtraced data
        • Characterization of continuum subtraction quality/reliability
        • must not overwrite mfs images of continuum generated earlier in PL recipe
    • Interface
      • rest frequencies for commonly used transitions selectable (interface with splatalogue)
      • support the multiple resolution modes
      • multiple sources per request
      • multiple cube requests
      • Scrape hifa_imageprecheck weblog output to beam sizes for different resolution selections
      • Enable user to manually edit their continuum regions
        • Use cont.dat from archive as starting point (if available)
        • show plot of continuum regions
        • Ability to Pan and Zoom in plot
          • when possible use CARTA to extract spectra from archive datacubes within masks
          • plot spectra in window
        • regions defined as start and end in LSRK frequency
          • organize in rows
          • add/delete rows
          • expand manually with mouse
      • Enable user to request delivery of calibrated MS with imaging request
    • Infrastructure
      • Split workflow for Restore and Imaging
        • Enables Restore in same CASA version as reduction (e.g., 5.1.1) and newest for imaging (5.6.1)
      • Add option to deliver calibrated MS in addition to images to user
  • Improved Download Support: Cutouts for images and data selection for MS downloads
    • Archive restore workflows should use XML template recipes for VLA and ALMA delivery
      • templates provided in CASA 6.1.2
      • enable intent selection, target selection, frequency averaging, and time averaging
    • batch request capability
      • upload formatted catalog, obtain cutouts against selected image collections (e.g., ALMA, VLASS)
  • Metadata Improvement: Improved meta-data for products in the archive.
    • Migrate to a refreshed archive-new interface
      • Ability add/hide presentation of some metadata associated with project/observation/image
    • Thumbnails for archived image products
      • Clickable to obtain large thumbnail
      • Determine thumbnail generation mode
        • crawl and process
        • scrape weblogs
    • QA State for different telescopes
        • ALMA
        • QA2 per MOUS
        • QA0 per EB
        • SRDP Imaging
        • VLA/SRDP QA
      • Weblog from Calibrations linked and viewable through archive
        • VLA
        • ALMA Calibration
        • ALMA Imaging (If separate from Calibration run)
        • ALMA SRDP Imaging
      • VLA Receiver Band Metadata correction
        • Fix older EVLA data (listed as subband...)
      • Angular resolution
        • Displayed per MOUS for ALMA
        • Tabulated approximately per VLA band/configuration
      • Conform to standard results table from System Requirements document
      • Search operates properly on images using their field of view
      • Search operates properly for mosaic observations (EBs and MOUSes)
        • Results should be returned if search position is within mosaic
    • CARTA- Archive Integration: Integration of the CARTA viewer with the Archive.
      • Support starting CARTA backend (require to be logged-in) and user views through new web browser tab
      • User should be able to select an image product in the archive and open it in CARTA
      • User should see a progress bar so they know what is happening
        • important for large cubes that are not available within seconds
      • Ability to open second image from archive within CARTA and view or overlay
        • Multiple image selection to make available in CARTA session
      • Regions support is required
        • Upload to CARTA
        • Download from CARTA to personal computer
      • region information should be available to archive interface for cutout or search functionality
      • Ability to upload their own image
      • Overlay catalogs  (available in archive e.g., NVSS, VLASS, etc or external catalog available in CARTA)
      • external catalogs available in CARTA (e.g., 2MASS, IRAS, GAIA, FIRST, NVSS, IRAS)
      • external image servers available to CARTA (e.g., DSS, SDSS, 2MASS, IRAS, WISE)
      • Upload user's catalog
      • Enable CARTA to be used behind the scenes to extract data from Cubes (e.g., spectra) for use in interface
      • Enable user to download fits or other image format from CARTA view
    • Catalog Support: Support for searchable scientific catalogs in the Archive
      • Catalogs should be searchable from a subset of search filters
      • Return results able to be used as query for project/observation/image searches
      • Ability to hide columns
      • Interface for projects to upload catalogs for ingestion
    • Image Ingestion: If not already complete the ingestion and service of VLASS and ALMA images.
      • ALMA
      • ALMA cubes and continuum images associated with each MOUS need to be ingested
      • ALMA image products must be displayed to users and downloadable
      • Thumbnails should be present for ALMA image products
      • Viewable in a CARTA session
      • VLASS
      • VLASS QL images from the entire first epoch must be ingested
      • VLASS images must be displayed and downloadable
      • VLASS images require thumbnails and high-resolution png images
      • Viewable in CARTA session
      • VLA Images
      • When VLA imaging pipeline comes on line products must be ingested
      • Image products have same requirements as ALMA image products
      • Ancillary products for all imaging products should be made available for download
    • Large Project Support: Support for the ingest and serving of products from Large projects as collections.
      • VLA Sky Survey
      • Epochs 1.1, 1.2, 2.1 images ingested and served
      • proper position search within the images and SDMs for VLASS
      • Projects should be viewable as a 'collection' to see only the data from that project.
      • project specific metadata should be searchable
      • Interface to support upload and ingestion of large project data
    • Observatory Calibrator Database: Development of a centralized calibrator database for the VLA.
      • Calibrator Flux densities
      • Per VLA Band
        • Central frequency
        • bandwidth
      • spectral index and curvature
      • Date
      • Calibrator models
      • Time-dependent models
      • Polarization models
      • models with spatial and spectral information
      • Antenna position database

    • Performance Improvement: Efforts to make the system more responsive and decrease product latency.
      • Optimized imaging
      • Multiple sources per MOUS grouped into single request
      • Multiple cubes specified per source(s) grouped into single request
      • Reduce restore load if user comes back to specifc MOUS several times for different jobs
      • Skip continuum finding step when possible
          • Data from current ALMA cycle should use the archived cont.dat when possible
          • Older data should always make use of current continuum finding routines
          • Must check that source(s)/spw selected for imaging has continuum defined in cont.dat
            • Data that are size mitigated might not have a region defined for all sources/spws
      • Restore
        • Reduce latency for popular data sets
        • Restore datasets with same CASA version as calibrated with

Relevant Documents

  • No labels