Page History

...

The NRAO Archive contains data from as far back as 1976, and currently contains just over 3 petabytes of information. Preserving this data and providing tools for the efficient utilization of the data products by the full multi-messenger community is a key responsibility of the DMS department. Starting in 2023, SSR working collaboratively with DMS will launch a new project to expand the use of the NRAO Archive as a Research Instrument. This project will focus on user specified reprocessing, interoperability with other science archives, and increasing the supported type of projects. We expect this project to extend at least into 2024 and perhaps to 2025.

...

• Scriptability of the archive query interface via VO-compliant Table Access Protocol (TAP).
• Download improvements (inclusion of wget commands to retrieve data).
• Search improvements.
• Position search fixes (see Known Issues below).
• Search based on integration time.
• Scanlist improvements
• VLA-specific
• Metadata display improvements (i.e., scan list, sources, frequency setup(s), total integration times).
• VLBA-specific
• Completion of VLBA data collection: Fixing corrupted Mark IV files, and adding a few missing GMVA, HSA, and Global VLBI data.
• Downloading large numbers (>500) of correlation files for VLBA projects in bulk via the 'Select All' option.
• Fixing the missing or incomplete metadata of some VLBA+EVLA (i.e. +Y1, +Y27) projects, without which download is impossible. 

The goal of this page is to compile both issues with the New Archive (AAT) and features the Legacy Archive (LA) has but not the AAT. The issues posted will be reviewed by John Tobin and Anna Kapinska (archive sub-system scientists) for submission as Jira tickets or submission of a requested development project for SSA planning. See the example for an issue posting below, we ask that whomever records an issue put their name and date along side it such that we can reach out if further information is needed.

Example:

• The AAT does not allow me to do X, while in the LA this can be accomplished by doing Y. (John Tobin; 02/25/2022)
  • Example data 1
  • Example data 2

Real recorded issues start here:

• Changing the download default so that tar.gz is unselected by default (Emmanuel Momjian; 02/28/2022)
  • Stephan made a ticket for it (

    Jira
    server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7344

    ) so tar-ing is not selected by default.
  • He will provide wget examples to be added into the science plone pages

    Jira
    server DMS JIRA serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7421

  • He will also disable checksums by default (

    Jira
    server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7343

    ).
• In AAT, currently you can download a calibration .tar file (if available) by clicking the icon in the Cals column. It would be nice to also access this from the same place as other download formats: Add to clipboard > Download > Choose download format > "Calibration files only". This is intuitive because the .tar file is already listed at the bottom of the download popup for use with requesting Calibrated MS. Additionally, this could allow for downloading several calibration tars from the same project at once (or from several projects if that capability is added for all download formats - separate ticket?). (Edward; 03/01/2022) 

  Jira
  server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7353

• The AAT handles any data download request as 'pipeline processing' even if the request is for raw data (e.g., SDM-BDF). This might be a reason why it is significantly slower in the download process itself compared to the LA (reported by staff as well as users). Added to the frustration of external users is the lack of direct download to their lustre area (

  Jira
  server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7179

  ). For the former, it will be highly worthwhile to enable more straight forward downloading when raw data are requested, i.e., removing the extra layer of treating these as 'pipeline process', so the speed is (hopefully) more comparable to the legacy archive's download (Emmanuel Momjian; 03/13/2022)
• The AAT does not support scripting. There are now several requests to have this supported in the new archive. This is a topic that has already been discussed with SW. There are now several examples on the use cases. Adding this here for completeness (Emmanuel Momjian; 03/14/22). Requirements have been submitted by JT for the May 6, 2022, planning meeting: Scriptable Query Interface to NRAO Archive.
• Download number limit max will be 100 data sets, but download volume limited to 10TB. Stephan made this change already OTF!
  • Update (by Anna): there are now Select All requests, one for VLBA files within single segment (can be considered temporary until it gets superseded by the broader request)

    Jira
    server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7185

    , and a broader one accessible on many levels in the archive structure

    Jira
    server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7352

• VLA/VLBA Image archive contents should be migrated into AAT (https://archive.nrao.edu/archive/archiveimage.html) (04/28/2022: Frank Schinzel)
• (old) FITSAIPS files should be associated with actual project segments instead of being dumped into made-up segment  (Anna Kapinska: 02-May-2022) 
  • This is what legacy archive used to do too (segment x), but in the new archive it should be fixed. See ticket

    Jira
    server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7328

    for details (that ticket is to be closed as it only requested that the FITSAIPS are ingested into new archive, the segment association is additional thing and needs a new ticket)
• There are 51 stray FITSAIPS projects that are left to sort out and ingest into new archive to AAT 4.2.0

  Jira
  server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7426

  (Anna Kapinska: 03-May-2022)
• There are still UVFITS files missing in the new archive, e.g. project GP0024 (Anna Kapinska: 03-May-2022)
• For observing VLBA projects that incorporate other dishes (e.g. +Y1, +Y27 etc), the VLA files should be associated with the relevant VLBA segment and not considered as "mixed" projects. This has been descoped from ticket 

  Jira
  server DMS JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId eb2e750b-a83a-387e-8345-36eee8a98f01 key SSA-7185

  but needs to be picked up in future AAT releases (Anna Kapinska: 03-May-2022)

Issues already in JIRA/Confluence:

...

import FITSAIPS files without matching projects (51 files)

...

...

Add FITSAIPS files to new archive

...

...

...

...

...

...

Download number limit: 100 data sets, but download data volume limit to 10TB per request.

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

• Enable the retirement of the legacy AAT/PPI

Definition of Done

• Archive has the functional equivalency of the legacy AAT and the overall user experience is improved.
• As a user, I can more easily identify data I'm interested in without having to download it first.
• Scanlists are improved

Broad Themes/Epics

• Product Versioning
• Data Access:
  • ALMA Data (both visibility and image data)
  • Large Project Ingestion and Support
  • VLA/VLBA Image Archive
• Usability Improvements
  • DA Data Editing Tools
  • VO / Scriptable Interfaces
• User Driven Imaging
  • VUDI

Scrum Team

• Product Owner, Archive: John Tobin
• Scrum Master: Daniel Lyons
• Team Members:
• SMEs:

Key Stakeholders

Mark Lacy (VLASS, Workspaces)

Emmanuel Momijan (New Mexico Ops)

...

...

...

...

...

...

...

...

Subscans table holding both Radian and Degree positions

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

Incorrect metadata sometimes displayed for projects with duplicate proposal IDs between VLA and VLBA

...

...

...

...

...

...

...

...

Helpdesk ticket for a search return we do not have in AAT, but legacy archive provides.

...

Allow search filtering based on integration time.

...

...

Proposal Contact Author not added to data.nrao.edu database

...

...

...

For completeness, in the table below we give web pages that will not work once the legacy archive becomes internal only (we can move this to another confluence page if needed)

...

...

...

...