Versions Compared

Old Version 2

changes.mady.by.user Robert Treacy

Saved on

compared with

New Version Current

changes.mady.by.user Robert Treacy

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Quarterly summaries of the progress and status of the SRDP Project.  Intended for internal consumption and distributed to the NRAO coord list.

Table of Contents
maxLevel2

Expand
titleMarch 4, 2019

March 4, 2019

Project

  • Scope for Wave 1 was realigned to accommodate RealFast [VLA commensal transient project], which displaced some previously scheduled work within DMS.  The scope change for RealFast, along with other schedule delays have restructured the SRDP Pilot a bit, with a potential delayed rollout of the workspace infrastructure to the Wave 1 release.
  • The first of the Wave 1 capabilities is nearing completion. The NRAO Archive will soon allow authorized users to download calibrated Measurement Sets for ALMA Cycle 5 and Cycle 6 data (this capability already exists for VLA).  The calibrated MSs are automatically restored from the ALMA pipeline generated calibrations upon demand.  Scientific validation is proceeding with detailed comparison of restored products to what is provided by NAASC via LAVA process.
  • The delivery of a next generation Proposal Submission Tool (ngPST) is a key enabler of true SRDP operations.  The SRDP project has agreed to include the creation of this tool as part of the SRDP project deliverables.  Project management plan is under development, a change request (to the SRDP scope) will be submitted once scope of ngPST work has been understood.

Heuristics

  • VLASS RFI flagging heuristics are being evaluated for inclusion in the PI pipeline along with other VLASS functionality.
  • The pipeline tasks necessary to conduct ALMA imaging based on PI-specified parameters request through the NRAO Archive are undergoing final testing for validation.

Operations

  • A draft SRDP operations plan was distributed to the SRDP management team at the end of December, the document is in stages of final draft.
  • Three applicants for the Socorro Scientific Data Analyst position were shortlisted and interviewed by phone. Two have been invited for in-person visits the first week of March. (One has since withdrawn.)
  • An NRAO-internal review (internal to the Observatory, not the project) of the SRDP Operations Plan is being organized for late in the month of March; a charge has been set, a review plan is in draft, selection of a chair and committee is in progress

Implementation

  • Improved VLA flux calibration will be brought into the Standard PI Pipeline for CASA 5.6.
  • Pipeline work to support PI directed ALMA cube imaging is ongoing, Support for PI directed ALMA cube imaging in the AAT and integration with the pipeline work is moving into the work queue
  • A testing environment setup is in progress, and a detailed test plan has been developed, first set of test procedures are under development for OUS restores
Expand
titleJune 4, 2019

June 4, 2019

Project

The SRDP project is approaching Pilot Operations (June 18)

  • Download of calibrated data from the Archive (ALMA, VLA)
  • Improved quality assurance of VLA pipeline results, including correction and re-execution of the pipeline.
  • Mid-Pilot release (mid-July) brings PI driven ALMA cube generation.

User facing capabilities for Wave 1 (Dec 2019) are the same as Pilot, starting to look at Wave 2 (Dec 2020) targets. 

Telescope Time Allocation Tools now an initiative within the SRDP project (see below)

Heuristics

  • Completed validation testing of pipeline and new archive for SRDP pilot. Validation testing now proceeding on CASA 5.6 for ALMA Cube Imaging and VLA pipeline with improved flux calibration.
  • Planning for Wave 2 capabilities proceeding, aiming to improve VLA pipeline spectral line handling and extend capabilities of ALMA cube imaging (e.g., resolution vs. surface brightness sensitivity tradeoffs) to enable users to further 'optimize' their images.
  • Studying common calibration issues with VLA data and those revealed in pilot to improve quality of pipeline calibration
  •  Mark Mark Whitehead has replaced Rafael Hiriart on the stakeholder requirements committee.  We thank Rafael for his contributions and welcome Mark to the committee.

Operations

  • A new data analyst (Aaron RobothamLawson) has recently started in Socorro. A second offer has been made (to replace Alex Sobotka, who is leaving in the summer).
  • The SRDP Operations review was held on 28th March. 51 RIDs were filed by the review panel and addressed by the Operations Team. The result was a partial pass, the reviewers requested a second review of the operations plan after the pilot was complete, but agreed that the planning was sufficient to proceed with the pilot.

  • Operations testing began following a TRR on 3rd May.

Telescope Time Allocation Tools

  • Change Request and Charter approved by key stakeholders
  • TTA Tools Concept and Software Requirements Document through internal review
  • Level 1 requirements capture and conceptual architecture design now underway.
  • Next milestone: Conceptual Design Review (Q1 FY20)

Implementation

  • Archive 3.6.0/3.6.1 was delivered to support OUS restores
  • Most of the pipeline work to support optimized imaging was completed, as was the CLI to support testing
  • Scripts were built to support pilot testing and early operations as a workaround to workflow implementation (scheduled later)
  • As this was the first integration period, interoperability issues were discovered and resolved

  • L1 test procedures were developed and are currently being executed in preparation for pilot operations

Information Services

  • CIS has provisioned two NGAS servers for use as a primary NRAO archive store in Charlottesville (primary and backup).  This store will be used as the backing store for ALMA products produced as part by the SRDP project.
  • RFQ in process for replacing the CV-Lustre solution with increased capacity and expanding the NAASC-Lustre system to meet the working storage needs for SRDP and other projects. Purchase Requisitions have been issued in NM for similar expansions needed by SRDP.
  • In addition, CIS participated in efforts to provide an architectural diagram for current NAASC IT operations, one that includes the new SRDP dedicated hardware (see issue SRDP-192 in open-jira).
Expand
titleSep 4, 2019

Sep 4, 2019

Project

As announced at the Coord Meeting SRDP is now a program (there are multiple projects that are under the SRDP umbrella).  For clarity we are going to change the name of the original SRDP Project to Science Ready Archive and Operations (SRAO), this is just a name change there is no change to the original scope of the project.

Science Ready Archive and Operation

  • SDRP Operations started in mid-June. The delivery of the first calibrated VLA dataset meeting science ready standards was on June 24th.
  • Since June 24th, 35 SRAO-calibrated datasets have been delivered (in Ku-band and higher).
  • Calibration restoration workflows for users of both ALMA and the VLA have also been enabled in the AAT since mid-June.
  • Archive 3.7.0 supporting the Mid-Pilot release is currently in validation.
    • Delayed several weeks due to an underestimate of the effort needed to deliver the software.
    • The ALMA PI Driven Imaging Graphical User Interface is operational and is being validated.
    • The VLA PI pipeline is in final validation and on track to be released jointly with the ALMA Cycle 7 pipeline.
    • The “Product System” within the archive, although invisible to the users, addresses significant technical debt and will facilitate delivering different products more quickly in the future.
  • Future Development
    • The Workspace Infrastructure is the current planning focus, with a target for delivery to the Wave 1 deployment (Nov. 2019).  This will retire scripted workflows, which were implemented within the archive as a temporary solution.
    • A conceptual description of Wave 2 developments is in final review, this begins the second wave of development leading to a November 2020 deployment.

VLA Sky Survey

  • VLASS Epoch 1 Observing and Quick Look imaging completed
  • Validation of CASA 5.6 based Calibration and Single Epoch Imaging underway.
  • Meeting with Center for High Throughput Computing (CHTC) organized by DMS to begin addressing the gap in computing resources for SE imaging.
  • A new Data Analysist (Trent Seelig) has recently started in Socorro (backfill for Alex Sobotka departure).

Next generation Telescope Time Allocation Tools

  • Effort is currently focusing on the requirements and beginning to develop the conceptual architecture.
  • Oct. target for Conceptual Design Review is no longer valid, will re-plan once project velocity better established.
Expand
titleDec 20, 2019

Dec 20, 2019

Science Ready Archive and Operation

  • Quality assured SRAO VLA calibrations extended to include X-band
    • 95 SRAO VLA calibrations delivered between Sept 1 and Nov 30th.
  • Version 3.7 of the NRAO Archive has been released, importantly this provides the support for User Driven Imaging of ALMA data to the Archive.
  • Planning and development for the next wave of functionality (Wave 2) is underway.  Headline capabilities are:
    • Integration of the CARTA viewer with the NRAO Archive
    • Workspace infrastructure in the archive to better manage capabilities and product creation.
    • Support for catalogs of science sources support in the archive
    • Extension of current VLA calibration and ALMA imaging capabilities.

VLA Sky Survey

  • Job ad has been posted for Data Analyst position (backfill for Angie Vargas January departure. Thanks Angie!)
  • Publication describing the survey has been accepted by PASP.
  • Reprocessing is underway for VLASS1.2 images that were affected by the correlator backend errors during observing
  • SIS addition of “persistent” storage to relive pressure on lustre allowing increased processing.

Next generation Telescope Time Allocation Tools

  • TTA Tools Concept is iterating between Project Scientist and System Architect, following architect’s review
Expand
titleMar 23, 2020

March 23, 2020

Science Ready Archive and Operation

  • 178 SRDP VLA calibrations delivered between Nov 30th and Feb 28th
  • ALMA User-driven imaging (AUDI) started in mid-December, 10 jobs completed successfully as part of our "soft" rollout.
  • Updated Archive 3.7 with bug fixes (now on 3.7.4)
  • Workspace CDR complete within DMS, now under implementation
  • New heuristics for wave 2 are likely to be delayed, effort focused on VLASS Imaging Project.


VLA Sky Survey

  • Open Data Analyst positions (2) currently in phone interviews:
    • VLASS (backfill for Angie Vargas who left in January)
    • SRDP / general DA position (replacement for Nick Ferraro, who will leave in July).
  • Initiated VLASS Imaging Project to address VLASS SE continuum imaging issues.
  • Preparing for initial data release:
    • Reprocessing of VLASS 1.2 images for correlator back-end errors is complete, now reprocessing VLASS 1.1 images
    • Reprocessing to correct for input parameter bug ongoing.
  • Preparing for VLASS Observing 2.1 (May)
    • Validation plan for VLASS 2.1 complete
    • Validated VLASS with realfast setup does not drop BDFs.  No correlator errors found in the data.
    • Draft of the QuickLook Quality Assurance memo currently in review by the VLASS team.
  • Grant for RADIAL project support was submitted

 

Telescope Time Allocation Tools

  • SRAO project planning and DMS documents in final draft for CoDR, April 15-16
Expand
titleJun 16, 2020

June 16, 2020

Science Ready Data Products

  • Four Data Analysts have accepted offers; two for work in Charlottesville and two for work in Socorro however COVID-19 has delayed onboarding
  • New Mexico Data Analysts are being cross-trained to support all efforts across SRDP program areas
  • Windows on the Radio Universe supplemental proposal submitted to NSF 

Science Ready Archive and Operation

  • 134 SRDP VLA calibrations processed since last quarter
  • 16 ALMA reimaging jobs processed since last quarter 

VLA Sky Survey

  • VLASS Imaging Project complete, VLASS memo published and delivered to DMS describing procedures and recipe for implementation of SE Continuum Imaging.
    • Test plan to pursue options for bright source mitigation for single epoch images developed
  • Preparation for VLASS Epoch 2.1 Observing
    • Pipeline validation using CASA 5.6.3 is complete
    • Updated VLASS Manager validated and deployed
    • VLASS Manager (1.2.1) with (shiny new interface) is in validation
    • Final validation for online system and workflow nearly complete 

Telescope Time Allocation Tools

  • CoDR held (virtually) Apr 15-16, received favorable committee reports with recommendations, response to committee is in progress, most recommendations and RIDs are in closeout
  • Evaluation of effort to continue vs. resources available resulted in a decision to postpone further work for a minimum of six months
Expand
titleSep 25, 2020

September 25, 2020

Science Ready Data Products

  • Non-technical introduction to SRDP Program presented as part of ODI series.

Related Staffing News

  • 2 SSA, 2 CASA, 1 NMS, and 1 Pipeline software engineers started or hired. 
    • Increased SSA capacity necessary prerequisite for TTA Tools work starts early FY21
  • 3 New Data Analysts started in Socorro
    • Kaitlyn Raub, and Edward Starr will be focused on SRAO work.
    • Sergio Garzo will be focused on VLASS effort.

Science Ready Archive and Operation

  • 76 SRDP calibrations and 6 ALMA reimaging jobs were processed.
  • Workspaces development underway, running behind due to priority conflict with VLASS.
  • ALMA User-Defined Imaging extension was validated for Fall 2020 pipeline release.  Users will be able to request higher or lower angular resolution for imaging within the NRAO archive imaging tool
  • VLA pipeline validation was completed for Fall 2020 release, upcoming release will feature multi-band calibration and basic continuum imaging pipeline (internal testing).

VLA Sky Survey

  • AAT (3.8.2) and VLASS Manager (1.2.1) software was delivered to support VLASS Epoch 2.1.
  • VLASS 2.1 observing/calibration/imaging is underway (and progressing well).
  • VLASS Manager 1.2.2 is now in place (many bug fixes and improved features over 1.2.1)
  • The VLASS Science colloquium was held on 20th August in conjunction with ODI to present VLASS science to participants in the RADIAL program.
  • Assessment and mitigation strategies for image-plane polarization correction is proceeding with promising results.
  • VLASS higher fidelity single epoch continuum demonstration imaging currently executing on Amazon Web Services.

Telescope Time Allocation Tools

  • Recruiting for scientist to support the existing proposal process and contribute to the new tool specification and testing is ongoing (closes mid-October).
Expand
titleDec 3, 2020

December 3, 2020

Science Ready Archive and Operation

  • 69 SRDP calibrations and 1 ALMA reimaging jobs were processed.
  • Validation of CASA 6.1.2 and Pipeline
    • Validated for VLA pipeline processing (for production and PIs).  Release first week of December (A Configuration).
    • Prototype VLA Imaging pipeline for testing and validation (not user facing yet)
    • Includes bug fixes with respect to 6.1 for VLA data processing, specifically for polarization and Total Electron Content data retrieval.
    • User specified angular resolution for ALMA user directed imaging. (Not yet available for users, needs interface modification in archive tool).
  • Two new releases of the Archive Interface, resolving VLASS operations issues and preparing for retirement of the old Archive.

VLA Sky Survey

  • Observation of the first half of the second VLA Sky Survey epoch is complete! Thanks to all of the teams that supported this observing. A few statistics from the observing:
    • A total of 178 scheduling blocks were observed.
    • Of these 149 (84%) were fully successful, and another 13 (7%) were partially usable.
    • Approximately 10% of observations were affected by a known CBE error accounting for 17 of the 29 partial and failed observations.
  • As of Dec 3, a single observation from VLASS 2.1 remains to be calibrated. Approximately 2250 Quick Look images remain to be created and Quality assured.
  • Image-plane position corrections have been applied to all VLASS 1.2 Quick Look images and validated.  This completes the re-processing of VLASS 1.2 Quick Look products.
  • CIRADA published their source catalog based on VLASS epoch 1 Quick Look images, and put their cutout server (serving VLASS 1.1, 1.2 and 2.1 images) online (http://cutouts.cirada.ca). 
  • The first NRAO press release based on VLASS data went out ("VLA Sky Survey Reveals Newborn Jets in Distant Galaxies" Nov 19th).
  • Progress toward VLASS Single Epoch Imaging
    • SE processing tests performed comparing mosaic gridder results with AW project gridder results for relative image quality and accuracy.
    • Implementation of VLASS Imaging Project workflow into pipeline proceeding on schedule. 

Telescope Time Allocation Tools

  • Implementation scheduled to begin Jan, 2021.
Expand
titleMarch 3, 2021

March 3, 2021

Science Ready Archive and Operation

  • VLA C-band and multiband data (using C-band and above) added to SRDP calibrations.
  • Flare of the VLA flux calibrator 3C138 starting mid-December was detected during SRDP operations (observations using 3C138 as the primary flux calibrator have been temporarily removed from SRDP processing pending updates to its variability).
  • During this period:
  • 318 SRDP calibrations performed (41% of pipelined datasets).
  • 9 ALMA reimaging jobs completed.
  • Two releases of the Archive Tool were validated and deployed.  These releases developed infrastructure for the CARTA viewer, and added user specified angular resolution to the ALMA User Driven Imaging (AUDI) capability.
  • Increased advertisement of AUID Capabilities:
  • AUDI advertised on Facebook Astronomers group
  • AUDI demo video created (link to video)
  • Presentation given to the ALMA Ambassadors
  • Rebooted DMS Workspaces project to improve the archive infrastructure. 

VLA Sky Survey

  • VLASS 2.1 QuickLook imaging complete.  We are now halfway through observing and QuickLook Processing for the survey!
  • Investigation ongoing as to why a small fraction of images (< 0.7%) have beam sizes that seem unphysically small in one or both axes
  • Additional servers to support VLASS processing all installed at NMT, disk drive issues have been solved, commissioning for use by VLASS is imminent.
  • Completed 100-square degrees of Single Epoch Demonstration imaging, analysis currently underway.

Telescope Time Allocation Tools

  • The TTA tools kickoff meeting was held on 9 February, 2021 to officially start the development on the new software.
  • Completed the recruitment of a new scientific staff member to support the existing proposal process and contribute to the new tool specification and testing.  Allison Costa will start July 19 located in Charlottesville.
Expand
titleJune 9, 2021

June 9, 2021

Science Ready Archive and Operations

  • 242 SRDP calibrations performed (39% of pipelined datasets).
  • 15 ALMA re-imaging jobs completed.
  • Testing of VLA imaging pipeline.
  • CARTA implementation for the NRAO Archive currently in validation.
  • First HT-Condor calibration workflows demonstrated by SSA team.  This important capability allows for remote execution of pipelines (c.f. RADIAL).


VLA Sky Survey

  • Preparation for VLASS 2.1 Single Epoch Continuum Imaging (Workflow and pipeline validation)
  • Preliminary investigation of Single Epoch Coarse Cubes
  • Validation of Single Epoch Continuum Images (memo in prep)

Telescope Time Allocation Tools

  • Implementation of "walking skeleton" is progressing, revisiting some technology choices to better support maintainability requirements.
Expand
titleSeptemper 16, 2021

September 16, 2021

Science Ready Data Products

  • Significant time spent recovering from infrastructure outages – NGAS server outage, VLA power failure, NAASC cooling failure, DSOC UPS upgrade. 

Science Ready Archive and Operations

  •  SRDP VLA calibrations performed (16% of pipelined datasets)
    •  Lower fraction than usual partly due to observations for the X-proposals
  •  7 ALMA User Directed Imaging jobs completed
    • Developed additional spatial smoothing heuristics for the AUDI interface.  Now being implemented for productions
  • VLA legacy data now available through the new NRAO Archive interface
  • Automated imaging and calibration for VLA SRDP are implemented in the SSA workspaces framework, allowing greater flexibility and maintainability 

VLA Sky Survey

  • Production Single Epoch Calibration and Continuum Imaging underway!
    • The VLASS Single Epoch continuum imaging pipeline has been validated and handed over for operations
    • Demonstrated using SSA Workspaces to execute VLASS imaging on a remote (NMT) cluster. 
  • Preparation for VLASS 2.2 observing
    • Observing tests for updated CBE/CMIB
    • Scheduling Block preparation
  • NMS updated software to address CBE crashes; updated MCAF, Telcal, and other software to avoid changes during Epoch 2.2 observing
    • Decided to allow observation of flux calibrator in the middle of or end of an SB (enabled by fix of CBE crashes)
      • allows us to use 3C286 as primary flux calibrator for larger fraction of SBs
      • allows expanded LST start ranges, increasing flexibility of scheduling

Telescope Time Allocation Tools

  • Shifted gears from working on a high-level “walking skeleton” to a specific VLA Use Case that has more depth to enable the team to move forward.
  • Started development of the User Interface in parallel to the back-end work.
  • Developing framework for detailed software requirements: refining concepts in the system description, work on the subsystem description, and algorithm development.



Expand
titleDecemper 09, 2021

December 9, 2021

Science Ready Data Products

  • A combined pipeline release supporting ALMA Cycle 8, VLA and SRDP functionality was updated in September ahead of VLASS [2021.2.0.128 (CASA 6.2.1)]
    • Supports enhanced RFI flagging (derived from VLASS) for VLA pipeline.
  • The NRAO Archive now includes GBT, VLBA, and historical VLA data.

Science Ready Archive and Operations

  • The CARTA viewer is now available in NRAO archive and can be used on VLASS images and ALMA images created with AUDI.  This allows users to view, and inspect image data directly in the web browser.
  • 150 SRDP VLA calibrations were performed (32% of pipelined datasets).
  • 18 ALMA User Directed Imaging jobs were completed and successfully archived.

VLA Sky Survey

  • VLASS Manager, Archive, and Workspace software all updated and validated for VLASS 2.2, included improvements to QL, CARTA, and Single Epoch Continuum Imaging (SECI).
  • Observing, QL Calibration & Imaging is underway for VLASS 2.2 and going smoothly (90% through B Config.). 
    • No sign of the recurring "missing BDF" issue or other errors that plagued previous observing epochs. (Congratulations NM Systems Group!)
  • VLASS Single Epoch Calibration and Continuum Imaging are underway for VLASS 2.1, with SECI executions being executed remotely (on cluster served at NMT).
  • Commissioning of the VLASS SECI "QA decision" workflow is ongoing; SE calibration and imaging are not yet at their expected steady-state/long-term rate as we are still in learning stages.
  • VLASS coarse cube imaging pipeline prototype was developed and handed off to Pipeline Group for implementation

Telescope Time Allocation Tools

  • Developed software to handle the basic concepts of a Proposal and generating details about how to observe, Observation Specifications, for a VLA Use Case.
  • Developed the User Interface for Proposals in parallel to the back-end work.
  • Developed concepts and prototype code that use observing, scheduling, and calibration strategies to generate Observation Specifications.


Expand
titleMarch xx, 2022

March xx, 2022

Science Ready Data Products

  • A

Science Ready Archive and Operations

  • T

VLA Sky Survey

  • VLASS

Telescope Time Allocation Tools

  • D