Page History

Overview

The NRAO Archive contains data from as far back as 1976, and currently contains just over 3 petabytes of information. Preserving this data and providing tools for the efficient utilization of the data products by the full multi-messenger community is a key responsibility of the DMS department. Starting in FY2023, SSR working collaboratively with DMS will launch a new project to expand the use of the NRAO Archive as a Research Instrument. This project will focus on user specified reprocessing, interoperability with other science archives, and increasing the supported type of projects. We expect this project to extend at least into 2024 and perhaps to 2025.

Scope Statement

The archive project will improve the user experience with the NRAO archive for the raw (visibility or otherwise) and image data from the VLA, VLBA, ALMA, and other telescopes, while not accruing significant new technical debt and retiring technical debt where necessary to improve the user experience. Interdependencies between the Workspaces system and the archive will be addressed within this project, but significant changes to Workspaces are out of scope, though limited enhancements are possible.

The archive predates architectural analysis at the NRAO. Therefore, this project will also begin with a limited architectural review with the aim of capturing the architectural drivers, quality attributes, and overall architecture of the archive, in order to inform the rest of the project.

Key Goals

• Improve the scripting interface to the NRAO archive
  
  • Allow downloading of data
  • Display additional metadata (and/or refactor database) to enable better filtering of data
• Download improvements (inclusion of wget commands to retrieve data).
• Search improvements.
  • Position search fixes.
  • Frequency-dependent FOV search
  • Search based on integration time.
• VLA-metadata display improvements (helps enable VUDI)
  
  • scan list improvements
  • sources,
  • frequency setup(s)
  • total integration times)
  • QA state
  • weblogs
• VLBA data presentation and display
  
  • Completion of VLBA data collection: Fixing corrupted Mark IV files, and adding a few missing GMVA, HSA, and Global VLBI data.
  • Downloading large numbers (>500) of correlation files for VLBA projects in bulk via the 'Select All' option.
  • Fixing the missing or incomplete metadata of some VLBA+EVLA (i.e. +Y1, +Y27) projects, without which download is impossible. 
• Improve restore capability
  
  • Increased robustness of ALMA MOUS ingestion of EBs and calibrations (ALMA Butler)
  • Allow subsets to be downloaded(select on target, intents, spectral windows)
  • Select CASA versions automatically
• Improve User-Defined Imaging
  • Improvements to AUDI (ALMA User-Defined Imaging) and reimplementation into Workspaces
  • Add VLA User Defined Imaging Capabilitiy
• Enable the retirement of the legacy AAT/PPI
• IDM? (Identity Management; might be mostly done?)
• Support ngVLA archive workspaces (what is this???)

Definition of Done

• Archive has the functional equivalency of the legacy AAT and the overall user experience is improved.
• As a user, I can more easily identify data I'm interested in without having to download it first on the basis of the metadata display and interactivity (e.g., CARTA).

Broad Themes/Epics

• Product Versioning
• Data Access:
  • ALMA Data (both visibility and image data)
  • Large Project Ingestion and Support
  • VLA/VLBA Image Archive
• Usability Improvements
  • DA Data Editing Tools
  • Metadata availability
  • VO / Scriptable Interfaces
• User Driven Imaging
  • VUDI

Scrum Team

• Product Owner, Archive: John Tobin
• Scrum Master: Daniel Lyons
• Team Members: Nathan Bokisch; Brittany Faciane, Charlotte Hausman; Daniel Lyons; Daniel Nemergut

Subject Matter Experts

• Mark Whitehead (Architecture)
• Robert Treacy (PMD)

Key Stakeholders ('Product Owner Village')

• John Tobin (NRAO Archive subsystem scientist)
• Amy Kimball (VLASS Ops)
• Adele Plunkett (ALMA Archive)
• Mark Lacy (VLASS, Workspaces)
• Emmanuel Momijan (New Mexico Ops)
• John Tobin (VLBA subsystem scientist)
• John Hibbard (ngVLA archive scientist)
• David Frayer (GBT)
• John Tobin (VLA)
• Stephan Witz (SSA Technologies/Tech Debt Avoidance)

Key Stakeholders

• Tony Remijan
• Jeff Kern

Links

NRAO Archive Tool Enhancements

New Archive, Legacy Archive Issues

Archive Improvement Projects

The goal of this page is to compile both issues with the New Archive (AAT) and features the Legacy Archive (LA) has but not the AAT. The issues posted will be reviewed by John Tobin and Anna Kapinska (archive sub-system scientists) for submission as Jira tickets or submission of a requested development project for SSA planning. See the example for an issue posting below, we ask that whomever records an issue put their name and date along side it such that we can reach out if further information is needed.

Example:

• The AAT does not allow me to do X, while in the LA this can be accomplished by doing Y. (John Tobin; 02/25/2022)
  • Example data 1
  • Example data 2

Real recorded issues start here:

• Changing the download default so that tar.gz is unselected by default (Emmanuel Momjian; 02/28/2022)
  • Stephan made a ticket for it (

    Jira
    server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7344

    ) so tar-ing is not selected by default.
  • He will provide wget examples to be added into the science plone pages

    Jira
    server DMS JIRA serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7421

  • He will also disable checksums by default (

    Jira
    server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7343

    ).
• In AAT, currently you can download a calibration .tar file (if available) by clicking the icon in the Cals column. It would be nice to also access this from the same place as other download formats: Add to clipboard > Download > Choose download format > "Calibration files only". This is intuitive because the .tar file is already listed at the bottom of the download popup for use with requesting Calibrated MS. Additionally, this could allow for downloading several calibration tars from the same project at once (or from several projects if that capability is added for all download formats - separate ticket?). (Edward; 03/01/2022) 

  Jira
  server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7353

• The AAT handles any data download request as 'pipeline processing' even if the request is for raw data (e.g., SDM-BDF). This might be a reason why it is significantly slower in the download process itself compared to the LA (reported by staff as well as users). Added to the frustration of external users is the lack of direct download to their lustre area (

  Jira
  server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7179

  ). For the former, it will be highly worthwhile to enable more straight forward downloading when raw data are requested, i.e., removing the extra layer of treating these as 'pipeline process', so the speed is (hopefully) more comparable to the legacy archive's download (Emmanuel Momjian; 03/13/2022)
• The AAT does not support scripting. There are now several requests to have this supported in the new archive. This is a topic that has already been discussed with SW. There are now several examples on the use cases. Adding this here for completeness (Emmanuel Momjian; 03/14/22). Requirements have been submitted by JT for the May 6, 2022, planning meeting: Scriptable Query Interface to NRAO Archive.
• Download number limit max will be 100 data sets, but download volume limited to 10TB. Stephan made this change already OTF!
  • Update (by Anna): there are now Select All requests, one for VLBA files within single segment (can be considered temporary until it gets superseded by the broader request)

    Jira
    server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7185

    , and a broader one accessible on many levels in the archive structure

    Jira
    server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7352

• VLA/VLBA Image archive contents should be migrated into AAT (https://archive.nrao.edu/archive/archiveimage.html) (04/28/2022: Frank Schinzel)
• (old) FITSAIPS files should be associated with actual project segments instead of being dumped into made-up segment  (Anna Kapinska: 02-May-2022) 
  • This is what legacy archive used to do too (segment x), but in the new archive it should be fixed. See ticket

    Jira
    server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7328

    for details (that ticket is to be closed as it only requested that the FITSAIPS are ingested into new archive, the segment association is additional thing and needs a new ticket)
• There are 51 stray FITSAIPS projects that are left to sort out and ingest into new archive to AAT 4.2.0

  Jira
  server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7426

  (Anna Kapinska: 03-May-2022)
• There are still UVFITS files missing in the new archive, e.g. project GP0024 (Anna Kapinska: 03-May-2022)
• For observing VLBA projects that incorporate other dishes (e.g. +Y1, +Y27 etc), the VLA files should be associated with the relevant VLBA segment and not considered as "mixed" projects. This has been descoped from ticket 

  Jira
  server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7185

  but needs to be picked up in future AAT releases (Anna Kapinska: 03-May-2022)

Issues already in JIRA/Confluence:

...

import FITSAIPS files without matching projects (51 files)

...

...

Add FITSAIPS files to new archive

...

...

...

...

...

...

Download number limit: 100 data sets, but download data volume limit to 10TB per request.

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

Subscans table holding both Radian and Degree positions

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

Incorrect metadata sometimes displayed for projects with duplicate proposal IDs between VLA and VLBA

...

...

...

...

...

...

...

...

Helpdesk ticket for a search return we do not have in AAT, but legacy archive provides.

...

Allow search filtering based on integration time.

...

...

Proposal Contact Author not added to data.nrao.edu database

...

...

...

For completeness, in the table below we give web pages that will not work once the legacy archive becomes internal only (we can move this to another confluence page if needed)

...

...

...

...