Introduction
The AAT/PPI (Archive Access Tool/Pipeline Processing Interface) is designed to be a replacement for the NRAO's current archive tool (https://archive.nrao.edu) that gives astronomers the ability to utilize the NRAO's processing clusters to manipulate observation data using CASA. This release provides the additional capability of providing ALMA OUS restores.
Additional features include:
- Responsive interface that works on mobile devices.
- Natural language (text based) searches of the observation's title, abstract, source names, authors, and more; e.g., search on 'nova'.
- Search results displayed as an Observations View (similar to the legacy archive) or a Projects View, which aggregates observations by project and also displays the title, abstract and other assorted details missing from the Observations View.
- Display a scans report for a given execution block, including the scan intents and subscans.
- The 'my data' feature: log in, press a button, and see a list of your observations.
- If you link your ALMA and NRAO accounts, you can use either to access your observations.
- For ALMA and VLA you can download your SDM as a measurement set and filter by scan intent.
- For VLA you can download a calibrated measurement set restored from a previously run calibration pipeline.
New Features
- refactoring of VLBA IDIFITS ingestion to support older format (multiple UV_DATA tables)
- support for VLASS quicklook image and VLBA IDIFITs and UVFITs reingestion, in order to improve metadata quality
- faster availability of restore functionality, as the search index updates are more complete upon calibration ingestion
- support for customization of cluster job limits (both time and memory) to allow greater flexibility of CIPL processing
Known Issues
- With the expansion of metadata carried for ALMA observations (in particular the OUS structure information), there is a need to reingest all of the ALMA observations. This process will take several weeks, and newer data (Cycle 5+) will be prioritized. This will lead to some inconsistencies in how this data is presented as the reingestion occurs
Release Notes - 3.6.0 - 2019-03-12
Epic
- [SSA-5111] - implement OUS Restores
Engineering Task
- [SSA-4786] - Configurable Workflow Memory Requests
- [SSA-4905] - Replace log with LOG
- [SSA-5030] - Refactoriing of VLBA ingestion
- [SSA-5045] - Get rid of "unavailable entity" in dumplogs
- [SSA-5049] - Handle ALMA Database Problems Gracefully
- [SSA-5160] - Write developer documentation
- [SSA-5246] - Block tests of RH classes not in use
- [SSA-5278] - RH Third-Party Code Cleanup
- [SSA-5286] - Remove unused link-creation code from workflow
- [SSA-5294] - De-complicate unwieldy RequestResource.represent()
- [SSA-5300] - speed up VLBA reingestion
- [SSA-5367] - setup pipeline areas for the new profiles
- [SSA-5376] - test cluster job submission
- [SSA-5396] - acceptance testing for OUS restores
- [SSA-5445] - add a Builder to SimpleProjectData
- [SSA-5459] - Cascade delete needed for subscans_data_descriptions
- [SSA-5460] - Name long-lived rabbitmq queues
- [SSA-5484] - solr for multiple applications
Feature
- [SSA-4485] - VLASS workflow needs to update indexer in archive
- [SSA-4761] - Configurable Cluster Timeouts
- [SSA-4890] - per-workflow max time to live
- [SSA-5077] - Expand Session mapping algorithm to handle fail & semi-pass QA0 EBs.
- [SSA-5125] - modify ingestion to support OUS structure
- [SSA-5126] - modify Ingestion to support OUS calibrations
- [SSA-5128] - implement OUS restore workflow
- [SSA-5244] - Fix and restore direct-download link in result email
- [SSA-5304] - Update Execution Blocks Core Upon Calibration Ingestion
- [SSA-5346] - Add flux.csv files to the calibration tar files
- [SSA-5356] - Modification to accomodate 'Project View' date sorts that make sense
- [SSA-5358] - Modify Alma reingestion to include relevant calibrations.
- [SSA-5465] - track and display SB name associated with an MOUS
Bug
- [SSA-5216] - VLBA ingestion error mode
- [SSA-5241] - Unable to create file download links for local delivery
- [SSA-5247] - Link Generation Error for Multi-SDM tar local Use Case
- [SSA-5259] - Double MS Local Delivery Failure
- [SSA-5264] - Image Reingestion Process Bug
- [SSA-5270] - "Download Complete" email is sent thrice
- [SSA-5313] - Email delivery fails silently
- [SSA-5321] - Calibrations table entries for EVLA entries
- [SSA-5353] - Greater Granularity for Reindexing CAPO settings
- [SSA-5360] - Last Addition field on project not updating correctly on ingestion
- [SSA-5380] - Polarization session problem
- [SSA-5416] - calibration failure, DF error
- [SSA-5431] - fix 'verbose' flag
- [SSA-5438] - CMS request fails when datafetcher can't find CAL file
- [SSA-5468] - Apply Flags (and potentially other settings) mistranslated in RH
- [SSA-5470] - QA Fail on EBs from a STOP period
- [SSA-5481] - ngas_id error for Alma Calibrations
Release Notes - 3.6.1 - 2019-06-18
Epic
- [SSA-5599] - Scripted QA system, preliminary support
Engineering Task
- [SSA-5638] - Expanded 'Project' Handling for VLBA files
- [SSA-5662] - ALMA execution block ingestion fails due to project lookup
Feature
- [SSA-4535] - CIPL Reprocessing Workflow Overhaul
- [SSA-5562] - In project view for VLA and ALMA newest EBs should be shown first, oldest last
- [SSA-5592] - CASA Versions for Testing
- [SSA-5595] - Alter description of download options for VLA and ALMA data
- [SSA-5596] - Default option for data downloads should be 'create tar file'
- [SSA-5600] - Don't Overwrite Older PPRs
- [SSA-5634] - Add html rendering to MotD, so we can include links
Bug
- [SSA-5268] - Delivered products have wrong permissions, ownership
- [SSA-5560] - ALMA SDMs can be downloaded for projects that have not yet been delivered
- [SSA-5576] - Ingestion of cal tables has a wrongly-named variable
- [SSA-5649] - ALMA Total Power data should not be available to restore
- [SSA-5658] - No new VLASS1.2 SBs appearing
- [SSA-5659] - Temporarily disable sessionStorage for filter retention