Introduction

The AAT/PPI (Archive Access Tool/Pipeline Processing Interface) is designed to be a replacement for the NRAO's current archive tool (https://archive.nrao.edu) that gives astronomers the ability to utilize the NRAO's processing clusters to manipulate observation data using CASA. This release provides the additional capability of providing ALMA OUS restores.

Additional features include:

  • Responsive interface that works on mobile devices.
  • Natural language (text based) searches of the observation's title, abstract, source names, authors, and more; e.g., search on 'nova'.
  • Search results displayed as an Observations View (similar to the legacy archive) or a Projects View, which aggregates observations by project and also displays the title, abstract and other assorted details missing from the Observations View.
  • Display a scans report for a given execution block, including the scan intents and subscans.
  • The 'my data' feature: log in, press a button, and see a list of your observations.
  • If you link your ALMA and NRAO accounts, you can use either to access your observations.
  • For ALMA and VLA you can download your SDM as a measurement set and filter by scan intent.
  • For VLA you can download a calibrated measurement set restored from a previously run calibration pipeline.

New Features

  • refactoring of VLBA IDIFITS ingestion to support older format (multiple UV_DATA tables)
  • support for VLASS quicklook image and VLBA IDIFITs and UVFITs reingestion, in order to improve metadata quality
  • faster availability of restore functionality, as the search index updates are more complete upon calibration ingestion
  • support for customization of cluster job limits (both time and memory) to allow greater flexibility of CIPL processing
  • support for data quality notes

Known Issues


Release Notes - 3.8.0 - 2020-03-25

Research Request

  • [SSA-5488] - RabbitMQ connection issues in Amygdala
  • [SSA-6092] - Investigate monitoring systems

Engineering Task

  • [SSA-5660] - Adapt Workflows to Product System
  • [SSA-6069] - Unify python version handling
  • [SSA-6112] - Support ancillary products in the product service
  • [SSA-6113] - Apply SonarQube advice vis-a-vis resource handling in the workflow
  • [SSA-6234] - Tool for finding missing data / gap analysis

Feature

  • [SSA-5957] - Can Show Proprietary Status of Project in Both Archives
  • [SSA-5958] - Can Lock/Unlock Non-ALMA Project in AAT/PPI
  • [SSA-5959] - Can Lock/Unlock Non-ALMA Project in Legacy Archive
  • [SSA-5960] - Changes to AAT/PPI Status Trigger Re-Index
  • [SSA-5963] - Can Set Title/Abstract on S-Code Projects
  • [SSA-5964] - Can Set PI/Authors on S-Code Projects
  • [SSA-5965] - Changes to S-Code Project Trigger Re-Index
  • [SSA-6085] - Display release number in FE
  • [SSA-6115] - Reincorporate weblogs in Calibration Tar

Bug

  • [SSA-5887] - Faultchecker missed some ebs in mcaf/workspace but not in parallel-prod
  • [SSA-5956] - VLBA Reingestion Results in Open Connections on the Messaging Bug: Fix Before Continuing to Reingest
  • [SSA-6038] - CoI list in solr changed from comma separated string to true list.
  • [SSA-6054] - CIPL weblog.tgz not un-tarring in workflow
  • [SSA-6073] - Checking User-Specified Directories
  • [SSA-6140] - Unable to extract data from ALMA archive
  • [SSA-6163] - Rerun of AUDI imaging fails in 3.8.0
  • [SSA-6199] - VLA downloads of SDM binary data (and MS) fail for some datasets in both production 3.7.4 and test 3.8.0
  • [SSA-6243] - Multiple SDM downloads not functioning on AAT 3.8.0
  • [SSA-6269] - Make deployment prompt on the console rather than with X windows

Release Notes - 3.8.1 - 2020-06-03

Epic

  • [SSA-5966] - Retire Legacy Archive: Missing Feature Data Quality Notes

Research Request

  • [SSA-5967] - Research Current Implementation, Design & Document Compatible System for AAT/PPI
  • [SSA-6090] - Images for some ALMA projects are delivered as empty directories

Engineering Task

  • [SSA-6084] - Calibration migration didn't create science_products_product_groups
  • [SSA-6240] - Permission settings for ingestion manifests & artifacts tars
  • [SSA-6268] - Update the VLASS Quick Look index page

Feature

  • [SSA-5961] - Provide a Utility for Fixing Merged Accounts
  • [SSA-5978] - Must Index Data Quality Notes
  • [SSA-5979] - Must Display Data Quality Notes
  • [SSA-5980] - Must Persist Data Quality Notes in Metadata Database
  • [SSA-5981] - Must Migrate Existing Data Quality Notes
  • [SSA-6095] - MotD live on deployment
  • [SSA-6114] - AUDI Image interface should set the minimum number of channels to three
  • [SSA-6279] - s-code project updater should allow for adding multiple CoIs and deleting them

Bug

  • [SSA-6055] - CASA version selection should not appear on AUDI dialog
  • [SSA-6080] - Re-enable the ability to ingest over a previous calibration
  • [SSA-6100] - Error Messages Prevent Reprocessing (CIPL & AUDI)
  • [SSA-6109] - Key change for IDIFITS Source table
  • [SSA-6142] - Staging area overlap for AUDI ingestion
  • [SSA-6277] - The check of the profile name in pyat.__init__ fails when run off network (home)
  • [SSA-6337] - Ingestion of ALMA imaging products failing in 3.8.1 tests
  • [SSA-6365] - Modified flagtemplate files not being used in reruns of the VLA pipeline

Release Notes - 3.8.1 - 2020-06-18

Bug

Release Notes - 3.8.2 - 2020-07-28

Epic

  • [SSA-5966] - Retire Legacy Archive: Missing Feature Data Quality Notes
  • [SSA-6281] - Import Legacy Archive VLBA Metadata 1
  • [SSA-6285] - Refactor AUDI Reprocessing (ALMA Cycle 5 Data Issue)

Research Request

  • [SSA-6344] - The Kludge: Research and Design

Engineering Task

  • [SSA-6215] - Refactor test classes with failing tests
  • [SSA-6228] - VLBA ingestion punchlist
  • [SSA-6229] - missed database schema migration steps
  • [SSA-6238] - Move workflow-server tests package out of src/
  • [SSA-6291] - I/O Buffer Left Open in Data Fetcher
  • [SSA-6351] - Migrate links to error codes for data quality notes
  • [SSA-6490] - VLBA reingestion screening upgrade

Feature

  • [SSA-6241] - Angular resolution should be displayed for ALMA MOUSes
  • [SSA-6341] - New AUDI Reprocessing Task
  • [SSA-6342] - Expand almaReimageCube to specify a CASA for restoring
  • [SSA-6343] - AUDI PPR extension for 'imaging only' mode.
  • [SSA-6350] - Add error code / link to Notes dialog

Bug

  • [SSA-5135] - VLBA ingestion failing on memory allocation
  • [SSA-6213] - AAT MS request with incorrect PPR.xml
  • [SSA-6214] - AAT MS request PPR.xml file parsing error
  • [SSA-6216] - AAT MS online flagging incorrect
  • [SSA-6300] - AUDI interface should not allow certain characters
  • [SSA-6448] - RH should query the archive to determine project authorship
  • [SSA-6471] - Early Calibration Reingestion Issue - part 3
  • [SSA-6482] - Don't open AMQP channel for events until after cache update
  • No labels