Introduction
The AAT/PPI (Archive Access Tool/Pipeline Processing Interface) is designed to be a replacement for the NRAO's current archive tool (https://archive.nrao.edu) that gives astronomers the ability to utilize the NRAO's processing clusters to manipulate observation data using CASA. This release provides the additional capability of providing ALMA OUS restores.
Additional features include:
- Responsive interface that works on mobile devices.
- Natural language (text based) searches of the observation's title, abstract, source names, authors, and more; e.g., search on 'nova'.
- Search results displayed as an Observations View (similar to the legacy archive) or a Projects View, which aggregates observations by project and also displays the title, abstract and other assorted details missing from the Observations View.
- Display a scans report for a given execution block, including the scan intents and subscans.
- The 'my data' feature: log in, press a button, and see a list of your observations.
- If you link your ALMA and NRAO accounts, you can use either to access your observations.
- For ALMA and VLA you can download your SDM as a measurement set and filter by scan intent.
- For VLA you can download a calibrated measurement set restored from a previously run calibration pipeline.
New Features
- refactoring of VLBA IDIFITS ingestion to support older format (multiple UV_DATA tables)
- support for VLASS quicklook image and VLBA IDIFITs and UVFITs reingestion, in order to improve metadata quality
- faster availability of restore functionality, as the search index updates are more complete upon calibration ingestion
- support for customization of cluster job limits (both time and memory) to allow greater flexibility of CIPL processing
- support for data quality notes
Known Issues
Release Notes - 3.9.0 - 2020-09-08
Epic
- [SSA-6355] - Prototype Legacy VLA Support
- [SSA-6434] - Bulk Ingestion of VLASS1.2 Quicklook Images
- [SSA-6437] - Can We Index/Display GBT Data?
- [SSA-6526] - Import Legacy Archive VLBA Metadata 2
Research Request
- [SSA-6356] - Prototype Legacy VLA Metadata Extraction & Persistence
- [SSA-6551] - Investigate responsiveness under high concurrency with Python web applications
Engineering Task
- [SSA-6071] - Ingestion Exit Status Values
- [SSA-6304] - VLBA Filegroup Problems
- [SSA-6310] - VLBA Duplicate External Name
- [SSA-6433] - Bulk Ingestion of VLASS1.2 Quicklook Images
- [SSA-6485] - database schema additions for RealFast metadata, revised
- [SSA-6497] - Prep Vlass 1.2 Cache
- [SSA-6511] - Build pytest structure for pyat
Feature
- [SSA-5928] - ALMA MOUS ID should be listed with Metadata
Bug
- [SSA-6165] - problem with VLA/11B-039 in archive-new
- [SSA-6458] - proprietary-setter script failure
- [SSA-6460] - Some VLBA ingestions not creating science_products_projects entry
- [SSA-6486] - scans link is bad
- [SSA-6512] - Disable selection of intents from MS downloads
- [SSA-6514] - Request hander displays links to individual files even when a tar file of data is produced
- [SSA-6517] - AUDI interface does not properly calculate end channels when chan is adjusted using arrows only
Release Notes - 3.9.0.1 - 2020-09-28
Epic
- [SSA-6555] - Interface Server Updates
Engineering Task
- [SSA-6581] - Oversee/Test OS Update of the Development Servers
- [SSA-6582] - Oversee/Test OS Update of the Testing Servers
- [SSA-6583] - Oversee/Test OS Update of the Production Servers
Release Notes - 3.9.1 - 2020-10-28
Epic
Engineering Task
- [SSA-6357] - Implement Legacy VLA Metadata and File Persistance
- [SSA-6358] - Implement Legacy VLA Indexing and Searching
- [SSA-6359] - Implement Legacy VLA File Retrieval
- [SSA-6668] - additional field for realfast data
Feature
- [SSA-5206] - integration testing for RealFast ingestion
- [SSA-5400] - support for realfast's image files
- [SSA-5752] - workflow modifications to support RealFast ingestion
- [SSA-6506] - Ingest RealFast metadata and products
- [SSA-6601] - Adapt workflows to work with CASA 6
- [SSA-6608] - new pipeline naming scheme
Bug
- [SSA-6639] - The 'Archive Access Tool' link on the data-test interface produces some jibberish and multiple error messages
- [SSA-6642] - unpinned zope versions
- [SSA-6657] - command-line script for "ingest-complete" is incompatible with Python 3.6
- [SSA-6674] - VLA pipeline fails to run in 3.9.1 tests
- [SSA-6683] - Realfast NGAS ingestion changes causing cal and image ingestion errors
Release Notes - 3.9.2 - 2020-12-16
Epic
- [SSA-5067] - AAT/PPI: GBT Support - Part I
- [SSA-5655] - Archive missing ALMA Projects, MOUSes and EBs
- [SSA-6376] - Are PSRFITS files execution blocks?
- [SSA-6423] - Prototype CARTA integration
- [SSA-6456] - Investigate Missing ALMA Metadata
Research Request
- [SSA-6282] - Investigate usage of fitsio rather than astropy
- [SSA-6377] - Are PSRFITS files execution blocks?
- [SSA-6457] - Investigate Missing ALMA Metadata
- [SSA-6744] - Investigate Missing ALMA Metadata
Engineering Task
- [SSA-6424] - Prototype CARTA integration
- [SSA-6525] - clean up files in postgres accounts
- [SSA-6686] - VLBA imports failing on corrupt source name
- [SSA-6692] - Add EVLA Only option to Indexer
- [SSA-6706] - VLBA import are failing on checksum
- [SSA-6710] - Rationalize MarkIV product metadata
- [SSA-6742] - Add GBT support to New Archive Front End and Service Layer
Feature
- [SSA-5068] - Support GBT metadata migration
- [SSA-5545] - Receiver bands 'Subband:XX' should correspond to VLA bands
Bug
- [SSA-5626] - EBs with missing band codes (EVLA)
- [SSA-6166] - data missing from archive-new
- [SSA-6245] - Ingestion failed for ALMA AUDI job 342391907
- [SSA-6614] - Proprietary period setter: tests and bug fix
- [SSA-6759] - proj_prop_period -h option returns all possible values
Release Notes - 3.9.3 - 2021-02-03
Epic
Engineering Task
- [SSA-6643] - Make clone-capo script more robust
- [SSA-6716] - Install traefik & redis on interface servers
- [SSA-6767] - VLBA Imports Failing on Corrupt Filegroup Structure
- [SSA-6772] - V2 Archive Rest API
- [SSA-6780] - Update VLBA indexing to ignore extremely large projects
- [SSA-6806] - Roll up VLBA file sizes to filegroup
- [SSA-6812] - Proposed new archive welcome message text
Feature
- [SSA-6690] - GUI Resolution Input
- [SSA-6691] - RH and WF Mods
- [SSA-6717] - Starting CARTA and service discovery
- [SSA-6718] - Stopping CARTA and cleanup
- [SSA-6766] - Large (>100 GB)SDM downloads failing
Bug
- [SSA-6752] - Frequencies in scan list for VLBA data have wrong units.
- [SSA-6779] - AUDI End channel dialog box almost unreadable in Safari
Release Notes - 3.9.4 - 2021-04-15
Epic
- [SSA-6719] - Support Mark 4 Files
Engineering Task
- [SSA-6724] - Design Ingestion Testing System
- [SSA-6740] - Refinement of VLASS SB selection Query
- [SSA-6801] - Filename handling extension for VLBA reingestion
Feature
- [SSA-5525] - Support download metrics
- [SSA-6721] - Implement Mark 4 Ingestion Queue System
- [SSA-6722] - Implement Mark 4 Ingestion CLI Modifications
Bug
- [SSA-6800] - deployed library versions not matching requirements.txt
- [SSA-6862] - recently ingested VLBA file sizes off
- [SSA-6867] - RealFast updates from validation testing.
- [SSA-6895] - Error in VLBA Project indexing Query
Release Notes - 3.10.0 - 2021-07-27
Engineering Task
- [SSA-6915] - Missing ALMA ASDM and calibration files
- [SSA-6934] - add locator to ingestion complete event
- [SSA-6953] - disable 'tar' option for GBT downloads
- [SSA-6969] - Refine ALMA OUS metadata definitions
- [SSA-6987] - legacy VLA indexer tweak, export -> visibility
- [SSA-7054] - mods for gunicorn