View Source

Introduction

The AAT/PPI (Archive Access Tool/Pipeline Processing Interface) is designed to be a replacement for the NRAO's current archive tool (https://archive.nrao.edu) that gives astronomers the ability to utilize the NRAO's processing clusters to manipulate observation data using CASA. This release provides the additional capability of providing ALMA OUS restores.

Additional features include:

Responsive interface that works on mobile devices.
Natural language (text based) searches of the observation's title, abstract, source names, authors, and more; e.g., search on 'nova'.
Search results displayed as an Observations View (similar to the legacy archive) or a Projects View, which aggregates observations by project and also displays the title, abstract and other assorted details missing from the Observations View.
Display a scans report for a given execution block, including the scan intents and subscans.
The 'my data' feature: log in, press a button, and see a list of your observations.
If you link your ALMA and NRAO accounts, you can use either to access your observations.
For ALMA and VLA you can download your SDM as a measurement set and filter by scan intent.
For VLA you can download a calibrated measurement set restored from a previously run calibration pipeline.

New Features

refactoring of VLBA IDIFITS ingestion to support older format (multiple UV_DATA tables)
support for VLASS quicklook image and VLBA IDIFITs and UVFITs reingestion, in order to improve metadata quality
faster availability of restore functionality, as the search index updates are more complete upon calibration ingestion
support for customization of cluster job limits (both time and memory) to allow greater flexibility of CIPL processing

Known Issues

CASA is known to fail silently when restoring certain datasets, currently the AAT/PPI treats these silent failures as successes, which is not optimal. The following CASA log message indicates the restore had a problem and is invalid: "SEVERE flagmanager:::: An error occurred running task flagmanager."

Release Notes - 3.7.4 - 2020-01-31

Bug

[SSA-6097] - Incorrect EBs showing up in archive-new.nrao.edu vs ALMA archive
[SSA-6102] - AUDI Imaging Requests do not report pending QA to users via e-mail
[SSA-6103] - AUDI Imaging Requests do not report completion and availability for download to users via e-mail

Release Notes - 3.7.3 - 2020-01-15

Bug

[SSA-5846] - Workflows for Optimized imaging and ALMA/VLA restores need to parse CASA log for unreported errors
[SSA-6078] - Alma Reingestion retrieves BDFs by mistake.

Release Notes - 3.7.2 - 2019-12-20

Bug

[SSA-6070] - Skipped Calibration Tar Files

Release Notes - 3.7.1 - 2019-12-20

Bug

[SSA-6052] - Download from Legacy Archive return empty SDMs
[SSA-6056] - Calibration Ingestion doubles up on Vlass' additional directory
[SSA-6060] - PI Science qa2 data delivery perimssions wrong
[SSA-6064] - AAT SDM-BDF delivery to lustre does not get *.bin files
[SSA-6066] - Missing BIN files on SDM-BDF downloads

Release Notes - 3.7.0 - 2019-12-12

Epic

[SSA-4706] - Upgrade ingestion's weblog access.
[SSA-5113] - Implement ALMA user driven imaging cube capability
[SSA-5330] - Science Products, Ancillary Products and Associations
[SSA-5599] - Scripted QA system, preliminary support
[SSA-5869] - Thumbnail Generation and Storage Upgrade

Research Request

[SSA-5483] - SPW idenification must be passed to PPR for ALMA Optimized Imaging
[SSA-5856] - Migration changes

Engineering Task

[SSA-5016] - Remove special case filename changes from the DataFetcher
[SSA-5139] - refactor data fetcher
[SSA-5229] - Ingest Additional Quicklook Sets
[SSA-5332] - Design Product Locators
[SSA-5336] - Deploy Test Database for Products
[SSA-5337] - Data Migration to New Schema
[SSA-5343] - Update Ingestion Workflows for Product Groups
[SSA-5392] - Design/prototype parameters interface
[SSA-5393] - Implement QA Pass/image ingestion
[SSA-5507] - Add query for Spectral Window information to Optimized Imaging PPR generation.
[SSA-5524] - Harvest additional required data to facilitate Optimized Imaging
[SSA-5548] - Create views to simplify using the product system
[SSA-5581] - Split Reingestion 'unstick' utility
[SSA-5605] - Generalize the QA cleanup workflow
[SSA-5606] - QA CLIs for Alma Imaging QA
[SSA-5608] - commission NGAS server for ALMA cube ingestion
[SSA-5619] - Create an AOI Reprocessing Prep Task
[SSA-5625] - Remove hard coded capo-profile strings from the back end code
[SSA-5627] - Reingestion with Science Product Locators
[SSA-5660] - Adapt Workflows to Product System
[SSA-5661] - Rewrite the reprocessing utilties for the new design
[SSA-5666] - Further Issues with Schedblock Names
[SSA-5688] - Re-ingest VLASS images for new schema
[SSA-5722] - Switch NGAS Ingestion Hosts
[SSA-5770] - VLBA ingestion failing on new db schema for external names
[SSA-5771] - Update mr_books for product system
[SSA-5773] - Modify VLA ingestion for external name schema changes
[SSA-5787] - Migrate developer documentation from Confluence to sphinx
[SSA-5796] - Relative path from locator service determines extra subdirectories
[SSA-5841] - Update faultchecker to product schema
[SSA-5842] - revise Amygdala for product schema changes
[SSA-5878] - Segregate Testing Messages
[SSA-5921] - Data Population in Migration to Product Schema
[SSA-6000] - Reinstate Weblog Raiding as intermediate Thumbnail solution

Feature

[SSA-4171] - Persist QA info to metadata DB
[SSA-4172] - Persist workflow output filegroup id to metadata DB
[SSA-4253] - TBD: Provide CIPL products to observers?
[SSA-4625] - Image Ingestion/Reingestion CLIs
[SSA-4707] - Incorporate natural order searches of pipline stage directories
[SSA-4708] - Encapsulate weblog access in Weblog class
[SSA-4710] - RMS Noise via Imaging Weblog
[SSA-4729] - Add cluster job number CIPL JobID directory directory
[SSA-5175] - Implement CLI
[SSA-5202] - database schema additions for RealFast metadata
[SSA-5203] - ingestion modifications for RealFast metadata
[SSA-5312] - Add Restore-To-Cache functionality to Vlass Workflow Server
[SSA-5333] - Database Refactor for Products and Product Groups
[SSA-5335] - Modify ingestion for new product schema
[SSA-5338] - Schema Documentation
[SSA-5339] - Data Fetcher and Product Locator
[SSA-5340] - Request Handler Authorization
[SSA-5341] - Index and Product Locator
[SSA-5342] - Modify ingestion for new product groups
[SSA-5344] - Front End Modifications for Product Locators
[SSA-5345] - Products System Documentation
[SSA-5408] - "Can Calibrate" and "Public Data" search 'checkboxes'
[SSA-5414] - Session state storage and cueing user to active filters.
[SSA-5482] - SRDP Optimized imaging PPR should put field names with special characters inside double quotes
[SSA-5525] - Support download metrics
[SSA-5532] - Adding Continuum Images to ALMA Cube Generation
[SSA-5535] - Dialog box for 'Download Restored MS' should display a time estimate
[SSA-5536] - Declination field does not allow colons in coordinates
[SSA-5563] - Add restore time estimate to ALMA and VLA Restores
[SSA-5583] - Display unique Science Product Locators for VLBA (execution block - project + segment)
[SSA-5594] - Archive must distinguish between ALMA 12m array observations and ACA 7m observations
[SSA-5601] - Handle Image QA Directory Management
[SSA-5602] - Generalize Email Task
[SSA-5604] - ALMA image ingestion workflow
[SSA-5609] - modify ingestion to track NGAS cluster/location
[SSA-5610] - modify retrieval to work with remote NGAS server
[SSA-5632] - Update Ingestion to handle the Alma Cube case
[SSA-5633] - Create Alma reprocessing Workflow & CLI
[SSA-5645] - ALMA cube imaging jobs should be sent to the cluster with 8-way parallelization
[SSA-5687] - Archive GUI deployment cleanup
[SSA-5692] - Provide collected Field values for preparatory pipeline steps
[SSA-5698] - MotD to client X
[SSA-5702] - Add AOI data mining to ingestion of ALMA calibrations
[SSA-5713] - VLASS Collection Database Changes
[SSA-5721] - RH launch updates for new AOI metadata
[SSA-5742] - Add Alma Optimized Imaging Using Ingestion Manifest
[SSA-5753] - Users of the Optimized imaging interface should select from a fixed set of units
[SSA-5754] - Add rest frequency Field to optimized cube interface
[SSA-5755] - The ending frequency/ velocity should be dynamically calculated as part of the imaging interface
[SSA-5756] - The spw drop down box in the optimize imaging interface should display spectral resolution
[SSA-5758] - Optimized imaging input parameters should not reset if a spw is changed
[SSA-5759] - Optimized imaging default parameters
[SSA-5760] - Optimized imaging parameter validation
[SSA-5788] - Ability to select CASA version for optimized imaging needed on archive-test
[SSA-5837] - Optimized imaging framework needs to support SPW NAME instead of SPW ID
[SSA-5839] - Remove non-science fields from selection dialog in Optimized Imaging Interface
[SSA-5840] - Change the name of 'Optimized Imaging' button to 'Re-Imaging'
[SSA-5845] - Request Handler updates for ALMA User Defined Imaging
[SSA-5870] - Dual Storage for image_thumbnail Ancillary Products
[SSA-5871] - Thumbnail Recognition at Manifest Generation
[SSA-5925] - (not) displaying thumbnails
[SSA-6003] - Handle Multiple Types of Continuum Results

Bug

[SSA-5516] - Upgrade Pika to 1.0.0
[SSA-5553] - Opening archive-test retains search selection between sessions
[SSA-5566] - MS caching files for VLASS during calibration ingestion
[SSA-5589] - Fix Spinner hiccups
[SSA-5630] - PI CIPL workflow is storing PI data in vlass_auto
[SSA-5639] - Widgets for clearing search filters
[SSA-5642] - Fix ALMA observations not showing any cals.
[SSA-5647] - Reindex argument generation incomplete
[SSA-5695] - Handle image core changes for product locators
[SSA-5700] - Revisite cal mapping to ebs
[SSA-5735] - Revert change to stage_products handling
[SSA-5744] - mous view broken
[SSA-5750] - Fix broken service endpoint for getting an ebs full details.
[SSA-5786] - release/3.7 workflow issues
[SSA-5792] - Rest frequency not passed from GUI to Optimized imaging pipeline
[SSA-5817] - ALMA Optimized imaging requests say they fail but look fine
[SSA-5853] - ASDM Data for ALMA restores are missing ASDM Binary files
[SSA-5854] - Listing of MOUSes for a project with many MOUSes fails
[SSA-5880] - DataFetcherTask looks for incorrect CAL rawdata directory
[SSA-5883] - SDM ingestion failing on webtest
[SSA-5890] - ALMA re-imaging GUI still has “Request Description” in the Request Description box
[SSA-5901] - Project image tab is broken
[SSA-5909] - ALMA time estimates are too low
[SSA-5915] - VLBA Ingestion issues
[SSA-5919] - AUDI bounds checking incorrect
[SSA-5922] - AUDI form, Invalid-validation message on spw change
[SSA-5924] - Alma Calibration Ingestion Enhancements
[SSA-5929] - PI not recognized for VLBA data
[SSA-5992] - Data Fetcher: execution site as Capo property
[SSA-5994] - Deployment Permissions
[SSA-5996] - AUDI image RH failure
[SSA-5998] - DF attempts to get AUDI images fail
[SSA-6001] - ALMA SPW Name is listed with MOUS metadata screwing up formatting
[SSA-6027] - AUDI Ingestion issues from Testing
[SSA-6030] - Calibrated MS should not be an option for individual EBs
[SSA-6033] - CIPL fails at the DF stage
[SSA-6037] - 414 error when navigating to mydata

Sub-Task

[SSA-5584] - Ingestion manifests for product groups
[SSA-5651] - Create utility for handling imaging metadata
[SSA-5671] - Weblog Utility Update for ALMA Bands
[SSA-5736] - Modify EVLA SDM Ingestion for Ingestion Manifest
[SSA-5737] - Modify ALMA SDM Ingestion for Ingestion Manifest
[SSA-5738] - Modify EVLA Calibration Ingestion for Ingestion Manifest
[SSA-5739] - Modify ALMA Calibration Ingestion for Ingestion Manifest
[SSA-5741] - Modify VLASS Quicklook Image Ingestion for Ingestion Manifest
[SSA-5774] - Manifest migration for EVLA SDM Ingestion
[SSA-5775] - Manifest migration for ALMA SDM reingestion
[SSA-5776] - Manifest migration for EVLA calibration ingestion
[SSA-5777] - Manifest migration for ALMA Cal reingestion
[SSA-5778] - Manifest migration for VLASS Quicklook ingestion
[SSA-5847] - Database & Model Updates
[SSA-5848] - SPW Name Data Backfill
[SSA-5849] - Update Alma Cal Ingestion for data requirements change
[SSA-5850] - Audit handling of spectral window indentifier information
[SSA-5851] - Audit spw identifer data transfer
[SSA-5852] - Update the PPR template-handling utilities for spw name