Introduction

The AAT/PPI (Archive Access Tool/Pipeline Processing Interface) is designed to be a replacement for the NRAO's current archive tool (https://archive.nrao.edu) that gives astronomers the ability to utilize the NRAO's processing clusters to manipulate observation data using CASA. This release provides the additional capability of providing ALMA OUS restores.

Additional features include:

  • Responsive interface that works on mobile devices.
  • Natural language (text based) searches of the observation's title, abstract, source names, authors, and more; e.g., search on 'nova'.
  • Search results displayed as an Observations View (similar to the legacy archive) or a Projects View, which aggregates observations by project and also displays the title, abstract and other assorted details missing from the Observations View.
  • Display a scans report for a given execution block, including the scan intents and subscans.
  • The 'my data' feature: log in, press a button, and see a list of your observations.
  • If you link your ALMA and NRAO accounts, you can use either to access your observations.
  • For ALMA and VLA you can download your SDM as a measurement set and filter by scan intent.
  • For VLA you can download a calibrated measurement set restored from a previously run calibration pipeline.

New Features

  • refactoring of VLBA IDIFITS ingestion to support older format (multiple UV_DATA tables)
  • support for VLASS quicklook image and VLBA IDIFITs and UVFITs reingestion, in order to improve metadata quality
  • faster availability of restore functionality, as the search index updates are more complete upon calibration ingestion
  • support for customization of cluster job limits (both time and memory) to allow greater flexibility of CIPL processing
  • support for data quality notes

Known Issues


Release Notes - 3.9.0 - 2020-09-08

Epic

  • [SSA-6355] - Prototype Legacy VLA Support
  • [SSA-6434] - Bulk Ingestion of VLASS1.2 Quicklook Images
  • [SSA-6437] - Can We Index/Display GBT Data?
  • [SSA-6526] - Import Legacy Archive VLBA Metadata 2

Research Request

  • [SSA-6356] - Prototype Legacy VLA Metadata Extraction & Persistence
  • [SSA-6551] - Investigate responsiveness under high concurrency with Python web applications

Engineering Task

  • [SSA-6071] - Ingestion Exit Status Values
  • [SSA-6304] - VLBA Filegroup Problems
  • [SSA-6310] - VLBA Duplicate External Name
  • [SSA-6433] - Bulk Ingestion of VLASS1.2 Quicklook Images
  • [SSA-6485] - database schema additions for RealFast metadata, revised
  • [SSA-6497] - Prep Vlass 1.2 Cache
  • [SSA-6511] - Build pytest structure for pyat

Feature

  • [SSA-5928] - ALMA MOUS ID should be listed with Metadata

Bug

  • [SSA-6165] - problem with VLA/11B-039 in archive-new
  • [SSA-6458] - proprietary-setter script failure
  • [SSA-6460] - Some VLBA ingestions not creating science_products_projects entry
  • [SSA-6486] - scans link is bad
  • [SSA-6512] - Disable selection of intents from MS downloads
  • [SSA-6514] - Request hander displays links to individual files even when a tar file of data is produced
  • [SSA-6517] - AUDI interface does not properly calculate end channels when chan is adjusted using arrows only

Release Notes - 3.9.0.1 - 2020-09-28

Epic

Engineering Task

  • [SSA-6581] - Oversee/Test OS Update of the Development Servers
  • [SSA-6582] - Oversee/Test OS Update of the Testing Servers
  • [SSA-6583] - Oversee/Test OS Update of the Production Servers

Release Notes - 3.9.1 - 2020-10-28

Epic

  • [SSA-5201] - Collection Support: realfast (ingestion)
  • [SSA-6585] - Implement Legacy VLA Support

Engineering Task

  • [SSA-6357] - Implement Legacy VLA Metadata and File Persistance
  • [SSA-6358] - Implement Legacy VLA Indexing and Searching
  • [SSA-6359] - Implement Legacy VLA File Retrieval
  • [SSA-6668] - additional field for realfast data

Feature

  • [SSA-5206] - integration testing for RealFast ingestion
  • [SSA-5400] - support for realfast's image files
  • [SSA-5752] - workflow modifications to support RealFast ingestion
  • [SSA-6506] - Ingest RealFast metadata and products
  • [SSA-6601] - Adapt workflows to work with CASA 6
  • [SSA-6608] - new pipeline naming scheme

Bug

  • [SSA-6639] - The 'Archive Access Tool' link on the data-test interface produces some jibberish and multiple error messages
  • [SSA-6642] - unpinned zope versions
  • [SSA-6657] - command-line script for "ingest-complete" is incompatible with Python 3.6
  • [SSA-6674] - VLA pipeline fails to run in 3.9.1 tests
  • [SSA-6683] - Realfast NGAS ingestion changes causing cal and image ingestion errors

Release Notes - 3.9.2 - 2020-12-16

Epic

  • [SSA-5067] - AAT/PPI: GBT Support - Part I
  • [SSA-5655] - Archive missing ALMA Projects, MOUSes and EBs
  • [SSA-6376] - Are PSRFITS files execution blocks?
  • [SSA-6423] - Prototype CARTA integration
  • [SSA-6456] - Investigate Missing ALMA Metadata

Research Request

  • [SSA-6282] - Investigate usage of fitsio rather than astropy
  • [SSA-6377] - Are PSRFITS files execution blocks?
  • [SSA-6457] - Investigate Missing ALMA Metadata
  • [SSA-6744] - Investigate Missing ALMA Metadata

Engineering Task

  • [SSA-6424] - Prototype CARTA integration
  • [SSA-6525] - clean up files in postgres accounts
  • [SSA-6686] - VLBA imports failing on corrupt source name
  • [SSA-6692] - Add EVLA Only option to Indexer
  • [SSA-6706] - VLBA import are failing on checksum
  • [SSA-6710] - Rationalize MarkIV product metadata
  • [SSA-6742] - Add GBT support to New Archive Front End and Service Layer

Feature

  • [SSA-5068] - Support GBT metadata migration
  • [SSA-5545] - Receiver bands 'Subband:XX' should correspond to VLA bands

Bug

  • [SSA-5626] - EBs with missing band codes (EVLA)
  • [SSA-6166] - data missing from archive-new
  • [SSA-6245] - Ingestion failed for ALMA AUDI job 342391907
  • [SSA-6614] - Proprietary period setter: tests and bug fix
  • [SSA-6759] - proj_prop_period -h option returns all possible values

Release Notes - 3.9.3 - 2021-02-03

Epic

Engineering Task

  • [SSA-6643] - Make clone-capo script more robust
  • [SSA-6716] - Install traefik & redis on interface servers
  • [SSA-6767] - VLBA Imports Failing on Corrupt Filegroup Structure
  • [SSA-6772] - V2 Archive Rest API
  • [SSA-6780] - Update VLBA indexing to ignore extremely large projects
  • [SSA-6806] - Roll up VLBA file sizes to filegroup
  • [SSA-6812] - Proposed new archive welcome message text

Feature

Bug

  • [SSA-6752] - Frequencies in scan list for VLBA data have wrong units.
  • [SSA-6779] - AUDI End channel dialog box almost unreadable in Safari

Release Notes - 3.9.4 - 2021-04-15

Epic

Engineering Task

  • [SSA-6724] - Design Ingestion Testing System
  • [SSA-6740] - Refinement of VLASS SB selection Query
  • [SSA-6801] - Filename handling extension for VLBA reingestion

Feature

  • [SSA-5525] - Support download metrics
  • [SSA-6721] - Implement Mark 4 Ingestion Queue System
  • [SSA-6722] - Implement Mark 4 Ingestion CLI Modifications

Bug

  • [SSA-6800] - deployed library versions not matching requirements.txt
  • [SSA-6862] - recently ingested VLBA file sizes off
  • [SSA-6867] - RealFast updates from validation testing.
  • [SSA-6895] - Error in VLBA Project indexing Query

Release Notes - 3.10.0 - 2021-07-27

Engineering Task

  • [SSA-6915] - Missing ALMA ASDM and calibration files
  • [SSA-6934] - add locator to ingestion complete event
  • [SSA-6953] - disable 'tar' option for GBT downloads
  • [SSA-6969] - Refine ALMA OUS metadata definitions
  • [SSA-6987] - legacy VLA indexer tweak, export -> visibility
  • [SSA-7054] - mods for gunicorn



  • No labels