AAT/PPI 4.3 Series Release Notes

Introduction

Additional features this release include:

4.3.0.3

• Improvement to the TAP service
• Metadata additions to the search interface
• Improvements for tracking file ingestion into permanent storage
• Introduction of Full Curation for image product correction
• Bug corrections for observation metadata curation

4.3.0

• Automatic Error Recognition of Ingested Data

  • Missing Scans/BDFs should now be flagged on ingestion

• Improved display of archive ingestion/observation issues

  • Missing Scans/BDFs warnings should now be highlighted in yellow

• Improved Scan Lists

  • Introduction of pagination

  • Introduction of Spectral Window data for EVLA, ALMA, VLBA, and GMVA

  • Various display improvements and corrections for previously held observation data
     Please Note: support of these features will require extensive data reingestion.

• QA Notes for newly ingested non-VLASS Calibrations are available from the Archive

• QA Notes for newly ingested non-VLASS Images are available from the Archive

• X-band will no longer be listed as an observation band if it was only used for pointing for newly ingested EVLA data

• The Science Helpdesk is now accessible from the Archive

• Curator: NRAO Archive Reingestion System

• VLBI Ingestion Support

  • VLBA/GMVA file ingestion has been standardized to use the manifest driven system in use for other instruments

  • VLBA/GMVA file ingestion will be removed from the Legacy Archive and current AAT/PPI services

  • VLBA/GMVA ingestion is now controlled by AAT/WS via StackStorm

    • this includes:
      - directory monitoring for the difx area
      - directory renaming to reflect ingestion status
      - Workspaces HTCondor based ingestion workflow implementation for VLBA, GMVA and Mark4 data types.

Release Notes - 4.3.0.3 - 2024.05.02

Task

• [WS-2109] - Remove defunct CASA processing properties from CAPO

Story

• [WS-1612] - As a users, I would like to see the number of antennas in TAP search returns
• [WS-1617] - As a user, I would like to see max uv-distance in the TAP server returns
• [WS-1764] - As a user, I would like to see the number of antennas in the AAT search returns
• [WS-1768] - As a user, I want to only see the tt0 images in archive search returns.
• [WS-1770] - As a product owner, I need images to be reingested such that the tt0 image is the primary data product
• [WS-2103] - As a user, I would like to see updated spectral information in the TAP service

Bug

• [WS-1411] - File sizes in TAP database stored as int32 and many overflow
• [WS-2179] - ALMA scanlists showing NaN
• [WS-2315] - data-query-test not returning observations without SPW info

Release Notes - 4.3.0.2 - 2023.12.21

Story

• [WS-1833] - Reingest to exclude pointing bands from eb.band_code
• [WS-2100] - As a user, I want positions corrected so that I can search OTF observations
• [WS-2112] - As a user, I do not want to see targets in spectral setups that only have SYSTEM_CONFIGURATION intents
• [WS-2170] - As a user, I would like the min/max freq columns to be present in the scanlists until metadata reharvesting is complete

Bug

• [WS-1351] - Indexing of VLA low band projects
• [WS-1893] - As a product owner, I would like existing data to have their metadata updated to support refactored scanlists

Release Notes - 4.3.0.1 - 2023.12.08

Bug

• [WS-2163] - VLBA not ingesting idifits into NGAS

Release Notes - 4.3.0 - 2023.11.30

Story

• [WS-213] - As a user I want to see the QA notes for my calibration
• [WS-223] - As a user I want to see the QA notes for my imaging products
• [WS-338] - As a user I want to know that the QA notes are ingested so I can see them
• [WS-1350] - As the product owner, I would like to see ingestion changes to support better scanlist display
• [WS-1355] - Automated Error Recognition of Ingested Data
• [WS-1356] - Display of archive ingestion/observation issues
• [WS-1362] - As a scientist, I would like to see several changes to ingestion to make the scanlist display more useful
• [WS-1391] - As a user, I do not want to see a project labeled as containing X-band (or C-band) data if X band (or C-band) was only used for pointing
• [WS-1392] - As a data analyst, we need a set of data annotation tools for highlighting problems with the data itself
• [WS-1393] - As a data analyst, we need a new type of annotation to label data with notes about the pipeline calibration
• [WS-1407] - As a scientist, I would like to see several changes to make the scanlist information more useful
• [WS-1495] - As a user, I would like the scanlists to have pagination
• [WS-1571] - As a product owner, I want VLBI files treated similarly to other products for ingestion
• [WS-1648] - As a scientist, I would like max uv-distance archived when data is ingested
• [WS-1649] - As a scientist, I would like the number of antennas to be kept during ingestion
• [WS-1769] - As a proj. scientist, I want the primary image data product to be righter
• [WS-1794] - As a user, I would like to see changes to VLBA ingestion to support new information in scanlists
• [WS-1802] - As a user, I would like a link to the science helpdesk on the archive page
• [WS-1803] - As a user, I would like the ALMA scanlists to mirror the refactored EVLA scanlists
• [WS-1805] - As a product owner, I want ingest to be able to work with the VLBI manifest changes
• [WS-1806] - As a product owner, I want VLBA ingestion to happen automatically
• [WS-1807] - As a DA, I want to be able to ingest GMVA files as needed
• [WS-1808] - As a product owner, I want a workflow that performs VLBA/GMVA ingestion
• [WS-1840] - As a system administrator, I would like automatic cleaning of the NAASC downloads areas
• [WS-1910] - As a product owner, I want VLBA Mark4 data placed into NGAS to facilitate current functionality
• [WS-1966] - Update AAT's mark4import script for WS taking over VLBA ingestion
• [WS-1976] - As a project scientist, I would like some further refinements to the scanlist presentation before deployment
• [WS-2001] - As a project scientist, I would like the VLBA and ALMA scanlists to have the same functionality as EVLA

Bug

• [WS-1349] - Scan list does not show coordinates between dec= 00 and dec = -1 correctly
• [WS-1810] - As a user, viewing scanlists from data within 'username's data' I am taken back to the main page
• [WS-1887] - The new archive provides a bad link for info on some data quality notes
• [WS-1919] - As a user, I would like local delivery links for MSes and basic downloads to include the full path and not be clickable
• [WS-2085] - The observed polarizations do not appear in the VLBA scanlists
• [WS-2110] - AUDI test run won't validate (velocity width rejected)

Sub-Task

• [WS-1771] - Alter the logic for selecting the primary vs. ancillary for ingest_envoy manifest creation

Research Request

• [WS-1363] - As a product owner, I want a way to ensure that as new fields are added to the archive, we have a method for reingesting existing data to populate those fields
• [WS-1591] - As a developer, I want to understand how the legacy archive performs error recognition
• [WS-1650] - As a developer, I would like a good estimate of the work necessary to support the NRAO archive from Astroquery