You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Introduction

The AAT/PPI (Archive Access Tool/Pipeline Processing Interface) is designed to be a replacement for the NRAO's current archive tool (https://archive.nrao.edu) that gives astronomers the ability to utilize the NRAO's processing clusters to manipulate observation data using CASA. This preliminary release provides the capability to run EVLA and ALMA observations through their respective calibration pipelines; additional capabilities will be forthcoming.

Additional features include:

  • Responsive interface that works on mobile devices.
  • Natural language (text based) searches of the observation's title, abstract, source names, authors, and more; e.g., search on 'nova'.
  • Search results displayed as an Observations View (similar to the legacy archive) or a Projects View, which aggregates observations by project and also displays the title, abstract and other assorted details missing from the Observations View.
  • Display a scans report for a given execution block, including the scan intents and subscans.
  • The 'my data' feature: log in, press a button, and see a list of your observations.
  • If you link your ALMA and NRAO accounts, you can use either to access your observations.
  • For ALMA and VLA you can download your SDM as a measurement set and filter by scan intent.
  • For ALMA and VLA you can download a calibrated measurement set, which re-runs the calibration pipeline on your SDM.
  • For VLA you can download a calibrated measurement set restored from a previously run calibration pipeline.

New Features

  • support for images, starting with the VLASS 1.1 'quicklook' images; these have been loaded into the system and are available for download
  • support for VLBA IDIFITs and UVFITs data sets, which we are currently loading into the system
  • multi-product requests: users can now retrieve multiple products in a single request, with the following limitations:
    • requested products must be from the same project and must be of the same type, e.g. six execution blocks from 17B-197, or 12 images from VLASS1.1
    • users are limited by disk space and number of products. The current settings for EBs are 10 EBs or 10TB; for images 20 images or 10TB, but we will adjust these to fit the usage and our storage space

Known Issues

  • The search interface for this release of the AAT/PPI is built on a completely new database of metadata, redesigned around the needs of VLASS and SRDP (ingestion of calibration products, observations with thousands of sub-scans and so on). We have ingested metadata from ALMA and the Jansky VLA into it, but it lacks metadata for the GBT and legacy VLA. Work is underway to address these issues, but until it is complete users are advised to use the legacy archive instead.  VLBA ingestions  of UVFITs and IDIFITs have started and will be complete in several months time.

Release Notes - 3.6.0 (Testing) - 2019-03-12

Epic

Engineering Task

  • [SSA-4786] - Configurable Workflow Memory Requests
  • [SSA-4905] - Replace log with LOG
  • [SSA-5030] - Refactoriing of VLBA ingestion
  • [SSA-5045] - Get rid of "unavailable entity" in dumplogs
  • [SSA-5049] - Handle ALMA Database Problems Gracefully
  • [SSA-5160] - Write developer documentation
  • [SSA-5246] - Block tests of RH classes not in use
  • [SSA-5278] - RH Third-Party Code Cleanup
  • [SSA-5286] - Remove unused link-creation code from workflow
  • [SSA-5294] - De-complicate unwieldy RequestResource.represent()
  • [SSA-5300] - speed up VLBA reingestion
  • [SSA-5367] - setup pipeline areas for the new profiles
  • [SSA-5376] - test cluster job submission
  • [SSA-5396] - acceptance testing for OUS restores
  • [SSA-5445] - add a Builder to SimpleProjectData
  • [SSA-5459] - Cascade delete needed for subscans_data_descriptions
  • [SSA-5460] - Name long-lived rabbitmq queues

Feature

  • [SSA-4485] - VLASS workflow needs to update indexer in archive
  • [SSA-4761] - Configurable Cluster Timeouts
  • [SSA-4890] - per-workflow max time to live
  • [SSA-5077] - Expand Session mapping algorithm to handle fail & semi-pass QA0 EBs.
  • [SSA-5125] - modify ingestion to support OUS structure
  • [SSA-5126] - modify Ingestion to support OUS calibrations
  • [SSA-5128] - implement OUS restore workflow
  • [SSA-5244] - Fix and restore direct-download link in result email
  • [SSA-5304] - Update Execution Blocks Core Upon Calibration Ingestion
  • [SSA-5346] - Add flux.csv files to the calibration tar files
  • [SSA-5356] - Modification to accomodate 'Project View' date sorts that make sense
  • [SSA-5358] - Modify Alma reingestion to include relevant calibrations.
  • [SSA-5465] - track and display SB name associated with an MOUS

Bug

  • [SSA-5216] - VLBA ingestion error mode
  • [SSA-5241] - Unable to create file download links for local delivery
  • [SSA-5247] - Link Generation Error for Multi-SDM tar local Use Case
  • [SSA-5259] - Double MS Local Delivery Failure
  • [SSA-5264] - Image Reingestion Process Bug
  • [SSA-5270] - "Download Complete" email is sent thrice
  • [SSA-5313] - Email delivery fails silently
  • [SSA-5321] - Calibrations table entries for EVLA entries
  • [SSA-5353] - Greater Granularity for Reindexing CAPO settings
  • [SSA-5360] - Last Addition field on project not updating correctly on ingestion
  • [SSA-5380] - Polarization session problem
  • [SSA-5416] - calibration failure, DF error
  • [SSA-5431] - fix 'verbose' flag
  • [SSA-5438] - CMS request fails when datafetcher can't find CAL file
  • [SSA-5468] - Apply Flags (and potentially other settings) mistranslated in RH
  • [SSA-5470] - QA Fail on EBs from a STOP period
  • [SSA-5481] - ngas_id error for Alma Calibrations


  • No labels