AAT/PPI 4.3 Series Release Notes

Introduction

Additional features this release include:

4.3.4.1

• Fix bug where duplicate ALMA project codes prevent the My Data view from rendering for some users
• Fix bug in observation ingestion causing duplicate observations configurations to persist to the database which prevents spectral data import

4.3.3/4.3.4

• Fix for solr indexer skipping projects without acknowledgements
• Import of Observing Type information from PST to Archive database
• Ingestion of EVLA Standard Imaging Spectral Cubes
• Ingestion for VLASS SE Coarse Cube Image products
• Downloads for VLASS SE Coarse Cube Image products
• Search and display of VLASS SE Coarse Cube Products

4.3.2.1

• Fix for Acknowledgment system bug causing data curation to remove VLBA observations from the archive search index
• Fix for My Data loading issue
• Fix for passing user emails to Workspaces for restores
• Fix to restore delivery directory naming to use the Request Handler Id rather than the Workspaces workflow id

4.3.2

• Introduction of the Data Acknowledgment System
• Improves position search results allowing display and sorting on distance from target
• Clarification of search field wildcards
• Aggregated scan list view for ALMA MOUSes
• TAP system now provides information about data proprietary status and QA notes.
• Calibrated Measurement Set Restores are now processed through the Workspaces system for both EVLA and ALMA.
• The CASA Version Matrix now disallows selection of CASA versions older than the original processing version if it is known.

4.3.1.1

• Enables project level and since specific date indexing for calibrations and images, manual and automated
• Corrections for display of partially curated observations
• Corrections to collection flagging on observation ingestion

4.3.1

• Request Handler error message improvements
• New Archive product view, allowing stable reproducible links to individual archived products
• Source resolver tool improvements
• Scan list display improvements
• Selfcal product ingestion
• Improvements to the TAP service
• Partial Curation Calibration support 
• Implementation of the CASA Version Matrix for CASA version selection based on initial product and processing type

4.3.0.3

• Improvement to the TAP service
• Metadata additions to the search interface
• Improvements for tracking file ingestion into permanent storage
• Introduction of Full Curation for image product correction
• Bug corrections for observation metadata curation

4.3.0

• Automatic Error Recognition of Ingested Data

  • Missing Scans/BDFs should now be flagged on ingestion

• Improved display of archive ingestion/observation issues

  • Missing Scans/BDFs warnings should now be highlighted in yellow

• Improved Scan Lists

  • Introduction of pagination

  • Introduction of Spectral Window data for EVLA, ALMA, VLBA, and GMVA

  • Various display improvements and corrections for previously held observation data
     Please Note: support of these features will require extensive data reingestion.

• QA Notes for newly ingested non-VLASS Calibrations are available from the Archive

• QA Notes for newly ingested non-VLASS Images are available from the Archive

• X-band will no longer be listed as an observation band if it was only used for pointing for newly ingested EVLA data

• The Science Helpdesk is now accessible from the Archive

• Curator: NRAO Archive Reingestion System

• VLBI Ingestion Support

  • VLBA/GMVA file ingestion has been standardized to use the manifest driven system in use for other instruments

  • VLBA/GMVA file ingestion will be removed from the Legacy Archive and current AAT/PPI services

  • VLBA/GMVA ingestion is now controlled by AAT/WS via StackStorm

    • this includes:
      - directory monitoring for the difx area
      - directory renaming to reflect ingestion status
      - Workspaces HTCondor based ingestion workflow implementation for VLBA, GMVA and Mark4 data types.

Known Issues

4.3.1

• There is a bug in the Product View UI where if a user opens the download modal and the closes it the loading screen never returns. This can be bypassed by reloading the page. Investigation into the cause is ongoing.

Release Notes - 4.3.4.1 - 2025.03.06

Bug

• [WS-3235] - Duplicate projects causing My Data to fail loading
• [WS-3236] - Autoflush change to Ingestion is causing duplicate observation configurations

Release Notes - 4.3.3/4.3.4 - 2025.02.27

Story

• [WS-2374] - As an operations manager I need the Observing Types from the PST stored in the AAT
• [WS-2755] - As a DA, I would like to be able to ingest a VLASS SE cube into the archive
• [WS-2828] - As a user I would like to download a VLASS cube
• [WS-2963] - As a user, I want to see VLASS cubes displayed in the archive

Bug

• [WS-2941] - Indexer skipping products without acknowledgement entries

Research Request

• [WS-2937] - Research and document needs for implementing Collections

Release Notes - 4.3.2.1 - 2024.11.07

Story

• [WS-2712] - As a user, directory name for local delivery should match the RH request ID for restorations
• [WS-2841] - As an operations manager, I would like the e-mail addresses for restores to be propagated into workspaces

Bug

• [WS-2842] - Add new project acknowledgement pattern for NRAO/VLBA pairings

Release Notes - 4.3.2 - 2024.10.30

Task

• [WS-2701] - Make location service pull expected ALMA calibration files from ALMA database

Story

• [WS-1339] - Image Position Search returns should show angular distance (in arcsec) from position searched
• [WS-1348] - Create download acknowledgement database layer
• [WS-1412] - As a user, I would like the TAP return query to carry information about the proprietary state of data
• [WS-1804] - As a user, I would like consistently implemented 'implicit' wildcards and to know that wildcards are implicit with tool tops
• [WS-1852] - As a user, I would like the QA notes to appear as a field in the database queried for TAP
• [WS-2169] - As a user, I would like to choose which version of CASA to run for ALMA restores
• [WS-2198] - As a product owner, I want appropriate acknowledgements displayed to users before downloads begin
• [WS-2199] - As a product owner, I want an archive-service endpoint to retrieve a project's acknowledgments
• [WS-2200] - As a DA, I want a tool that allows me to set an acknowledgment on a project
• [WS-2387] - As a user, I would like older data to have the X-band code removed from bands when it was only used for pointing
• [WS-2397] - Make AAT submit EVLA Restore request to WS
• [WS-2413] - As a product owner, I would like the restore workflow to parse the casa logs to search for 'SEVERE' statements
• [WS-2509] - Create an aggregate scanlist link for ALMA MOUSes
• [WS-2510] - As a user, I would like the scanlist display to show spectral setup information with each EB, below the header info
• [WS-2539] - Observation/Project Position Search returns should show angular distance (in arcsec) from position searched
• [WS-2547] - As a product owner, I would like the appropriate acknowledgements set at the time of data ingestion
• [WS-2563] - As a product owner, I would like the display of acknowledgements adjusted
• [WS-2565] - As an NRAO partner, I want to update the USNO project codes for the acknowledgement system
• [WS-2573] - Preserve sort column across results views if possible
• [WS-2637] - As a user, I want the new ALMA restores integrated into the Archive

Bug

• [WS-2691] - Fix bug where CASA Version list for processing doesn't reload except on refresh in AAT

Epic

• [WS-1453] - Provide users with standard acknowledgements for data
• [WS-2184] - Conversion and improvements for EVLA Restores
• [WS-2185] - Conversion and improvements for ALMA Restores
• [WS-2193] - Implementation of CASA Version Matrix

Release Notes - 4.3.1.1 - 2024.08.08

Story

• [WS-2570] - As a archive maintainer, I need the Solr Indexer to support project level indexing for calibrations

Bug

• [WS-2560] - Fix product view UI to handle NaNs in json
• [WS-2562] - Ingest not flagging RealFast collection data

Release Notes - 4.3.1 - 2024.07.08

Story

• [WS-1405] - As a product owner, I would like the AAT to automatically select the appropriate version of CASA to restore my data
• [WS-1610] - As a user I would like to be able to access an AAT link that brings up a specific product so that I can integrate the archive with other systems or share the link with other users
• [WS-1934] - As a user, I would like a more intuitive way to use the source resolver
• [WS-2087] - As a user, I would like to see the source coordinates in the scanlist per-band tables
• [WS-2088] - As a user, I would like to see spectral resolution displayed in kHz when the channel width is < 0.1 MHz.
• [WS-2171] - As a project scientist, I need the selfcal outputs archived with the images from an EB
• [WS-2317] - As a project scientist, I would like the default CASA for VLA restores (and VUDI) able to be set independently
• [WS-2346] - As a user, I would like the t_min and t_max TAP returns to reflect the start/stoptime of a scan, not the whole observation
• [WS-2347] - As a user, I would like to see the scan number returned in the TAP interface
• [WS-2349] - As an operations manager, I would like calibration ingestion to track the CASA version used for processing
• [WS-2350] - As an operations manager, I would like calibrations curated to back populate the CASA version used during processing
• [WS-2389] - Provide default scheduling block types for non-SDM observations
• [WS-2409] - As a user, I would like project code to be returned in TAP searches

Bug

• [WS-1332] - Request Handler should provide user-friendly error message when access is not allowed to dataset

Research Request

• [WS-2318] - Design and document CASA Version Matrix

Release Notes - 4.3.0.3 - 2024.05.02

Task

• [WS-2109] - Remove defunct CASA processing properties from CAPO

Story

• [WS-1612] - As a users, I would like to see the number of antennas in TAP search returns
• [WS-1617] - As a user, I would like to see max uv-distance in the TAP server returns
• [WS-1764] - As a user, I would like to see the number of antennas in the AAT search returns
• [WS-1768] - As a user, I want to only see the tt0 images in archive search returns.
• [WS-1770] - As a product owner, I need images to be reingested such that the tt0 image is the primary data product
• [WS-2103] - As a user, I would like to see updated spectral information in the TAP service

Bug

• [WS-1411] - File sizes in TAP database stored as int32 and many overflow
• [WS-2179] - ALMA scanlists showing NaN
• [WS-2315] - data-query-test not returning observations without SPW info

Release Notes - 4.3.0.2 - 2023.12.21

Story

• [WS-1833] - Reingest to exclude pointing bands from eb.band_code
• [WS-2100] - As a user, I want positions corrected so that I can search OTF observations
• [WS-2112] - As a user, I do not want to see targets in spectral setups that only have SYSTEM_CONFIGURATION intents
• [WS-2170] - As a user, I would like the min/max freq columns to be present in the scanlists until metadata reharvesting is complete

Bug

• [WS-1351] - Indexing of VLA low band projects
• [WS-1893] - As a product owner, I would like existing data to have their metadata updated to support refactored scanlists

Release Notes - 4.3.0.1 - 2023.12.08

Bug

• [WS-2163] - VLBA not ingesting idifits into NGAS

Release Notes - 4.3.0 - 2023.11.30

Story

• [WS-213] - As a user I want to see the QA notes for my calibration
• [WS-223] - As a user I want to see the QA notes for my imaging products
• [WS-338] - As a user I want to know that the QA notes are ingested so I can see them
• [WS-1350] - As the product owner, I would like to see ingestion changes to support better scanlist display
• [WS-1355] - Automated Error Recognition of Ingested Data
• [WS-1356] - Display of archive ingestion/observation issues
• [WS-1362] - As a scientist, I would like to see several changes to ingestion to make the scanlist display more useful
• [WS-1391] - As a user, I do not want to see a project labeled as containing X-band (or C-band) data if X band (or C-band) was only used for pointing
• [WS-1392] - As a data analyst, we need a set of data annotation tools for highlighting problems with the data itself
• [WS-1393] - As a data analyst, we need a new type of annotation to label data with notes about the pipeline calibration
• [WS-1407] - As a scientist, I would like to see several changes to make the scanlist information more useful
• [WS-1495] - As a user, I would like the scanlists to have pagination
• [WS-1571] - As a product owner, I want VLBI files treated similarly to other products for ingestion
• [WS-1648] - As a scientist, I would like max uv-distance archived when data is ingested
• [WS-1649] - As a scientist, I would like the number of antennas to be kept during ingestion
• [WS-1769] - As a proj. scientist, I want the primary image data product to be righter
• [WS-1794] - As a user, I would like to see changes to VLBA ingestion to support new information in scanlists
• [WS-1802] - As a user, I would like a link to the science helpdesk on the archive page
• [WS-1803] - As a user, I would like the ALMA scanlists to mirror the refactored EVLA scanlists
• [WS-1805] - As a product owner, I want ingest to be able to work with the VLBI manifest changes
• [WS-1806] - As a product owner, I want VLBA ingestion to happen automatically
• [WS-1807] - As a DA, I want to be able to ingest GMVA files as needed
• [WS-1808] - As a product owner, I want a workflow that performs VLBA/GMVA ingestion
• [WS-1840] - As a system administrator, I would like automatic cleaning of the NAASC downloads areas
• [WS-1910] - As a product owner, I want VLBA Mark4 data placed into NGAS to facilitate current functionality
• [WS-1966] - Update AAT's mark4import script for WS taking over VLBA ingestion
• [WS-1976] - As a project scientist, I would like some further refinements to the scanlist presentation before deployment
• [WS-2001] - As a project scientist, I would like the VLBA and ALMA scanlists to have the same functionality as EVLA

Bug

• [WS-1349] - Scan list does not show coordinates between dec= 00 and dec = -1 correctly
• [WS-1810] - As a user, viewing scanlists from data within 'username's data' I am taken back to the main page
• [WS-1887] - The new archive provides a bad link for info on some data quality notes
• [WS-1919] - As a user, I would like local delivery links for MSes and basic downloads to include the full path and not be clickable
• [WS-2085] - The observed polarizations do not appear in the VLBA scanlists
• [WS-2110] - AUDI test run won't validate (velocity width rejected)

Sub-Task

• [WS-1771] - Alter the logic for selecting the primary vs. ancillary for ingest_envoy manifest creation

Research Request

• [WS-1363] - As a product owner, I want a way to ensure that as new fields are added to the archive, we have a method for reingesting existing data to populate those fields
• [WS-1591] - As a developer, I want to understand how the legacy archive performs error recognition
• [WS-1650] - As a developer, I would like a good estimate of the work necessary to support the NRAO archive from Astroquery