SRDP Wave 2 Conceptual Requirements

PREPARED BY	ORGANIZATION	DATE
John Tobin	NRAO	03 September 2019

Change Record

VERSION	DATE	REASON
0.1	12 August 2019	Initial Version
0.2	03 September 2019	Minor Revisions
0.3	19 September 2019	Minor Revisions based on stake holder comments

Table of Contents

1	Introduction	2

1.1	Purpose	2

1.2	Scope	2

1.3	References	2

1.4	Abbreviations and Acronyms	2

2	VLASS PRODUCT DEVELOPMENT PLAN	2

2.1	The path to the First Generation of SE Stokes I Continuum Imaging	2

2.1.1	SE Calibration Pipeline Validation	2

2.1.2	Number of w-planes	3

2.1.3	Primary Beams	3

3	Introduction	3

Introduction

Purpose

This is a Science Ready Data Products (SRDP) Project Office document describing the conceptual requirements for the wave 2 capabilities of SRDP. SRDP releases functionality in a rolling wave approach with major features released on an approximately yearly basis. SRDP is currently in its Pilot phase for VLA operations and calibrated data are being delivered from the NRAO archive-new interface to PIs and archival users. Wave I will begin toward the end of 2019 with the same features as the Pilot, but available to the general community. With the launch of SRDP operations for Wave I, it is time to plan the Wave 2 requirements to be developed over the course of late 2019 to mid-2020.

Scope

This document details the conceptual requirements for the second wave of major SRDP functionality. The conceptual requirements are sub-divided into functionality that is primarily related to CASA and ALMA/VLA pipeline functionality and requirements that are primarily related archive or system level functionality. Interdependencies will be pointed out, for example, it may be needed for the archive interface to pass a parameter to the pipeline in order to activate/deactivate functionality. All items listed are high priority for the project and the prioritization of implementation will be coordinated with DMS, CASA, SSA, and the Pipeline Team.

References

[SRDP System concept]

[SRDP System Requirements]

Abbreviations and Acronyms

AAT-PPI Archive Access Tool – Post-Processing Interface

AoD Astronomer on Duty

AWS Amazon Web Services

CASA Common Astronomy Software Applications

CIS Computing and Information Systems

DA Data Analyst

DMS Data Management and Software

DO NRAO Observatory Director’s Office

HPC High Performance Computing

OPT VLA Observation Preparation Tool

OTFM On-The-Fly Mosaic

PD Project Director

PM Project Manager/Project Management

PS Project Scientist/System Engineering

QA Quality Assurance

RFI Radio Frequency Interference

RMS root-mean-squared noise

SB Scheduling Block

SE Systems Engineer

SIS Scientific Information Systems

SPW Spectral Window

SRDP Science Ready Data Products

SSA Science Support and Archive

VLA Karl G. Jansky Very Large Array

VLASS Karl G. Jansky VLA Sky Survey

SRDP Requirements from Pipeline

VLA Calibration Improvement

The VLA pipeline, while producing valid calibrations, does not always result in a science ready data product. Often, additional flagging is required due to the presence of RFI and/or system issues that the pipeline does not currently have the heuristics to flag. The QA process conducted by the SRDP Operations catches these issues and applies additional flags to datasets; however, when these issues can be identified by the pipeline, they should be flagged automatically. As such, the primary focus of the VLA pipeline will be on calibration to provide a foundation for users and further SRDP functionality.

In SRDP Wave 2, we aim to implement enhanced flagging heuristics to catch system issues and RFI.
We are aiming to identify a wider variety of DTS issues (bad deformatters) in the VLA pipeline and more gracefully recover from bad deformatters on the reference antenna. We also plan to enable new flagging heuristics on the antenna-based gaintables to reject outliers based on their gain solutions. Some of these can result from RFI, while others can be due to a variety of system issues. These heuristics could be further used to refine the flux calibration accuracy. The VLA PI pipeline will also be adapted to better support multi-band data taken as part of the same EB.

VLA Spectral Line Calibration

The VLA pipeline cannot currently process spectral line data properly through the current recipe. This is because all data are Hanning smoothed, subjected to RFI flagging, and run through statwt. If there are bright spectral lines (e.g., masers) RFI flagging will flag these channels and weaker spectral line emission that gets through RFI flagging can be down-weighted by statwt. Thus, the current pipeline does not provide users with a viable data product for spectral line observations. The pipeline is able to handle spectral line datasets through a modified recipe and with user-provided input in the form of a cont.dat file. However, this is not optimal and the VLA pipeline should be able to calibrate spectral line data as part of the standard pipeline.

As such, the pipeline needs to know upon importing of the data, what spws are intended to be spectral line and which are intended to be continuum such that they are handled correctly by each pipeline task. Within this effort we will be exploring the possibility of using the switched power measurements to set the weights for the VLA data, which may be useful for both calibration and spectral line imaging.

VLA Imaging Pipeline

Building on the foundation of improved pipeline flagging heuristics, the VLA will be provided with a basic imaging pipeline in Wave 2. This imaging pipeline will build off the existing imaging tasks that have been developed as general purpose imaging pipeline tasks that can be successfully used with VLA data. The initial imaging pipeline will function at Ku-band and above to reduce the likelihood of bright sources outside the primary beam that cannot be properly imaged without very large fields or outlier fields. It will also be continuum only as the VLA pipeline does not adequately support spectral line data at present.

Automasking has been developed and can be used with VLA data, but further investigation is required to determine if its settings are optimal for VLA data. As such, we will initially deploy the imaging pipeline to use a sensitivity threshold for the masking at the 5 or 6 sigma level.

ALMA Optimized imaging

The initial optimized imaging capability provided in the Pilot and Wave I enabled users to create cubes of specific size and channel width. Now in Wave 2 we aim to expand on these capabilities with the infrastructure that has been built. We will enable user-selectable resolution to select between various tradeoffs between surface brightness sensitivity and angular resolution. They will be able to select between line-only and continuum+line imaging, preserving the continuum if desired. We aim to enable users to modify the continuum ranges determined by the pipeline and enable users to request continuum imaging-only; at present they must request a cube in order to get a continuum image. Finally, we aim to provide the user with the ability to request self-calibration of their requested data, based on the continuum image (subsequently applied to line data). We will work with the ALMA Pipeline Working Group on the implementation of this feature.

SRDP requirements from the archive

ALMA image products

To facilitate data discovery, the ALMA image products need to be ingested and served from the NRAO archive. Thumbnail images should be provided for each imaging product. Image products (cube or continuum) should be viewable web browser using CARTA (see section 3.1.6).

CARTA Integration

Once ALMA and VLA image products are ingested into the archive, the ability to explore the data online, without downloading extremely large datacubes must be supported. CARTA has this functionality already. There are three basic levels of integration that are required:

The users should be able to select an image in the archive interface and have it open in a CARTA session in their web browser without waiting an unreasonable amount of time.

The user shall be able to select a second image and have it open/overlaid in the same CARTA session.

The user shall be able to select a region in the CARTA session and use this information in the archive interface (either for cutouts or as a search parameter).

Optimized imaging Interface Improvements

New functionality to support continuum-only, cube imaging without a separate continuum image, continuum+line imaging, and variable resolution. Moreover, several additional user experience improvements can be added such as a button to either open Splatalogue in a new tab for the frequency range of the current SPW or an in-line display of molecular lines in the given spectral range from Splatalogue. Also, in addition to a longer list of spectral lines, we should enable users to select from a dropdown list of commonly used rest frequencies within the given SPW, emulating something similar to the ‘Quick Picker’ functionality of Splatalogue.

Large Project Support

Large projects from the VLA and ALMA should be able to upload their data products to the archive and have them served with the ability to have project-specific metadata.

Catalog Display and Search

Catalogs need to be ingested into the archive and should be able to be searched upon. Catalogs should also be able to be overlaid within CARTA seamlessly.

Improved product Metadata

QA state

The archive must display the QA state for the data products contained within the archive, encompassing the differing QA systems for ALMA, SRDP imaging on ALMA data, and SRDP products for the VLA.

Weblog

The weblog from product calibrations for both ALMA and VLA calibration and imaging should be available for immediate viewing from a link.

Proprietary state

The archive needs to better track if a dataset is proprietary or not to reduce confusion on which data are available for restoration/imaging.

Image cutouts

The image products hosted within the archive from ALMA, VLASS, and soon the VLA, as well as large projects require a cutout service to provide only the sub-regions required by the user. A single-image cutout service could be useful, but a more useful interface would allow a user to upload a catalog and based on this catalog return cutouts from specific collections within the archive around their requested source position.

Data Delivery

The archive will enable enhanced data selection and manipulation prior to download. For example, only download particular SPWs and sources and possibly time averaging and spectral averaging. CASA 5.6.X has an XML template recipe available to support this capability.

Cache system

The cache system for calibrated measurement sets that was descoped for Wave I requires implementation in Wave 2. This will lead to a better user experience with optimized imaging where subsequent requests can be completed more quickly without the need to conduct a restoration.

Observatory Database

An observatory database will be required to fulfill the end goal of SRDP from the proposal process to data processing. This database needs to supply calibrator flux densities in each frequency band, valid uv-ranges, calibrator models when available, flux calibrator models, polarization calibrator models, and support time-dependent models for polarization and flux density.