Overview
The NRAO Archive contains data from as far back as 1976, and currently contains just over 3 petabytes of information. Preserving this data and providing tools for the efficient utilization of the data products by the full multi-messenger community is a key responsibility of the DMS department. Starting in FY2023, SSR working collaboratively with DMS will launch a new project to expand the use of the NRAO Archive as a Research Instrument. This project will focus on user specified reprocessing, interoperability with other science archives, and increasing the supported type of projects. We expect this project to extend at least into 2024 and perhaps to 2025.
Key Goals
- Scriptability of the archive query interface via VO-compliant Table Access Protocol (TAP).
- Download improvements (inclusion of wget commands to retrieve data).
- Search improvements.
- Position search fixes.
- Search based on integration time.
- Scanlist improvements
- VLA-specific
- Metadata display improvements (i.e., scan list, sources, frequency setup(s), total integration times).
- VLBA-specific
- Completion of VLBA data collection: Fixing corrupted Mark IV files, and adding a few missing GMVA, HSA, and Global VLBI data.
- Downloading large numbers (>500) of correlation files for VLBA projects in bulk via the 'Select All' option.
- Fixing the missing or incomplete metadata of some VLBA+EVLA (i.e. +Y1, +Y27) projects, without which download is impossible.
- Enable the retirement of the legacy AAT/PPI
- ALMA User Defined Imaging (AUDI) (Improvements to AUDI and reimplementation into Workspaces)
- ALMA Butler ? (Improved detection and ingestion of ALMA datasets for Restores and AUDI)
- IDM? (Identity Management; might be mostly done?)
- Support ngVLA archive workspaces
Definition of Done
- Archive has the functional equivalency of the legacy AAT and the overall user experience is improved.
- As a user, I can more easily identify data I'm interested in without having to download it first.
- Scanlists are improved
Broad Themes/Epics
- Product Versioning
- Data Access:
- ALMA Data (both visibility and image data)
- Large Project Ingestion and Support
- VLA/VLBA Image Archive
- Usability Improvements
- DA Data Editing Tools
- Metadata availability
- VO / Scriptable Interfaces
- User Driven Imaging
- VUDI
Scrum Team
- Product Owner, Archive: John Tobin
- Scrum Coordinator?: Daniel Lyons
- Team Members: Nathan Bokisch; Charlotte Hausman; Daniel Lyons; Daniel Nemergut
Subject Matter Experts
- Mark Whitehead (Architecture)
- Robert Treacy (PMD)
Key Stakeholders ('Product Owner Village')
- John Tobin (ALMA Archive subsystem scientist)
- Amy Kimball (VLASS Ops)
- Mark Lacy (VLASS, Workspaces)
- Emmanuel Momijan (New Mexico Ops)
- John Tobin (VLBA subsystem scientist)
- John Hibbard (ngVLA archive scientist)
- David Frayer (GBT)
- John Tobin (VLA)
- Stephan Witz (SSA Technologies/Tech Debt Avoidance)
Key Stakeholders
- Tony Remijan
- Jeff Kern
Links
NRAO Archive Tool Enhancements
New Archive, Legacy Archive Enhancement List