View Source

Introduction

The AAT/PPI (Archive Access Tool/Pipeline Processing Interface) is designed to be a replacement for the NRAO's current archive tool (https://archive.nrao.edu) that gives astronomers the ability to utilize the NRAO's processing clusters to manipulate observation data using CASA. This preliminary release provides the capability to run EVLA and ALMA observations through their respective calibration pipelines; additional capabilities will be forthcoming.

Additional features include:

Responsive interface that works on mobile devices.
Natural language (text based) searches of the observation's title, abstract, source names, authors, and more; e.g., search on 'nova'.
Search results displayed as an Observations View (similar to the legacy archive) or a Projects View, which aggregates observations by project and also displays the title, abstract and other assorted details missing from the Observations View.
Display a scans report for a given execution block, including the scan intents and subscans.
The 'my data' feature: login, press a button, and see a list of your observations.
If you link your ALMA and NRAO accounts, you can use either to access your observations.
For ALMA and VLA you can download your SDM as a measurement set and filter by scan intent.
For ALMA and VLA you can download a calibrated measurement set, which re-runs the calibration pipeline on your SDM.
For VLA you can download a calibrated measurement set restored from a previously run calibration pipeline.

Known Issues

The search interface for this release of the AAT/PPI is built on a completely new database of metadata, redesigned around the needs of VLASS and SRDP (ingestion of calibration products, observations with thousands of sub-scans and so on). We have ingested metadata from ALMA and the Jansky VLA into it, but it lacks metadata for the GBT, legacy VLA, and VLBA. Work is underway to address these issues, but until it is complete users are advised to use the legacy archive instead.

Release Notes - 3.0.0 - 2017-12-18

Engineering Task

[SSA-3946] - tomcat, rabbitmq connections
[SSA-4216] - Add calibrations to project view execution blocks
[SSA-4266] - Implement calibration downloads in the front end
[SSA-4350] - front end tweaks
[SSA-4384] - Re-integrate ALMA and VLBA products

Feature

[SSA-4057] - Restore Workflow - front end changes
[SSA-4242] - front end, calibration status and download options
[SSA-4252] - Calibration Download Workflow
[SSA-4278] - revive 'restore' workflow
[SSA-4341] - scan intent filtering, custom options per EB
[SSA-4369] - Project view - exec-block table refinements
[SSA-4395] - Deliver Calibrated MS on a Reprocess
[SSA-4401] - offer 'sdm only' downloads for ALMA
[SSA-4408] - Casa Output Verification

Bug

[SSA-4077] - local delivery issues
[SSA-4210] - failure mode to catch
[SSA-4246] - Log messages in workflow.log that don't show up in dumplogs output
[SSA-4308] - observation stop sort
[SSA-4314] - DatabaseConnectionManager.getDataSource() throws RuntimeException
[SSA-4315] - tar 'local' delivery bug
[SSA-4359] - null subdirectory in download path
[SSA-4398] - ALMA reingestion issues
[SSA-4402] - early VLA eb's zero number of scans
[SSA-4412] - Specify a walltime option on qsub for some of our jobs
[SSA-4424] - 'reprocessing' bug
[SSA-4433] - more authors/ingestion foo
[SSA-4445] - queries by position busted

Release Notes - 3.1.0 - 2018-04-18

Engineering Task

[SSA-4274] - DataFetcher refactor
[SSA-4458] - VLBA Downloads
[SSA-4467] - Simplify and streamline connection pool
[SSA-4479] - Remove database queries from PprCreator
[SSA-4571] - Check for presence of files in source before delivery and in destination afterwards
[SSA-4577] - Include tail of CASA log in all result emails
[SSA-4587] - Streamline workflow request notification
[SSA-4631] - Improve workflow threading
[SSA-4639] - Hanging executor in SubmissionResource
[SSA-4643] - Schema changes for Image and image_products tables (testdb)
[SSA-4649] - Update CIPL reprocessing to handle new CIPL delivery
[SSA-4650] - Consolidate email code in workflows
[SSA-4652] - Port Request Handler DB to PostgreSQL
[SSA-4689] - Include failure details in download-failure email
[SSA-4700] - Keep track of workflow state

Feature

[SSA-4264] - /tmp/ Directory Naming
[SSA-4381] - Reinstate the ALMA workflows
[SSA-4426] - File Access Audit
[SSA-4495] - Create Image Ingestion Workflow
[SSA-4510] - move CIPL launch logic into amygdala
[SSA-4517] - CIPL Start Email
[SSA-4527] - Ingestion CLIs
[SSA-4544] - workflows use provided CASA
[SSA-4547] - add CASA option to CLI tools
[SSA-4669] - CLI for the RestoreToCache Workflow
[SSA-4691] - Workflows should specify qsub options for failure notifications
[SSA-4703] - Move CASA intents and procedures file names to CAPO
[SSA-4722] - Update BDF and SDM ingestion for realfast

Bug

[SSA-4461] - mr_clean: sym links
[SSA-4514] - Profile Deduction Fails for calFileSets (and probably other clis)
[SSA-4515] - Workflows use the wrong start date for their directory creation.
[SSA-4519] - Handling of Weblogs For Calibration Ingestion
[SSA-4558] - Local delivery fails if MD5SUMS already present in target directory
[SSA-4578] - Missing MD5 checksum file.
[SSA-4621] - nmprod MS cache area has world read/write permissions for some MSs
[SSA-4633] - Archive file size disagreements
[SSA-4641] - CIPL task failure email notification not sent
[SSA-4672] - Finished CIPL jobs have not moved to QA2
[SSA-4673] - calFileSets Script Not Starting Jobs, CIPL
[SSA-4701] - Capo-ify and Update Proceedures File Handling
[SSA-4719] - Local delivery of CAL download fails
[SSA-4727] - RH Authorization not handling files correctly

Release Notes - 3.2.0 - 2018-06-07

Epic

[SSA-4066] - Set up ingestion deployment and testing for parallel operation

Engineering Task

[SSA-4195] - mapping VLASS project codes onto the VLASS proposal
[SSA-4250] - ingest old calibration products
[SSA-4372] - Local delivery: check destination for permissions early
[SSA-4665] - Reinstate vmem limits for cluster jobs
[SSA-4724] - Implement freemarker templates for all request state emails
[SSA-4725] - Include workflow name and fileset ID in download notification emails
[SSA-4741] - mr_books upgrades
[SSA-4764] - Show current workflow task on request details page
[SSA-4783] - Archive value => enum
[SSA-4794] - Expanded Request ID System
[SSA-4795] - Include Request ID in Logging Framework
[SSA-4796] - Update Workflow Tasks to Handle System IDs
[SSA-4797] - Update Workflow Jobs to Handle New System IDs
[SSA-4803] - Upgrade dumplogs to handle IDs
[SSA-4815] - Request Attributes enum
[SSA-4823] - Incorporate ID values into workflow logging contexts
[SSA-4830] - Request Handler Tables Documentation

Feature

[SSA-4671] - Update Casa Error Handling
[SSA-4713] - Add the MS and Weblog Cache directories to mr_clean
[SSA-4748] - Make the RH match the FE in style
[SSA-4787] - Load image and image products tables with metadata from json file

Bug

[SSA-4589] - ALMA duplicates
[SSA-4755] - Permissions Issue with Local Delivery
[SSA-4756] - VLASS Download Fails
[SSA-4757] - Gson and Character sets
[SSA-4765] - mr_clean & long running calibrations
[SSA-4782] - Alma Reingestion Duplicates
[SSA-4801] - Workflow failure message doesn't make it to failure email, UI
[SSA-4807] - plus to minus
[SSA-4819] - CASA failure during CIPL on production but not test
[SSA-4821] - check-summer script fails due to truncated command
[SSA-4826] - JSON parsing issues during error message persistence
[SSA-4833] - Request Handler UI Issues

Sub-Task

[SSA-4069] - Add post-ingestion event

Release Notes - 3.2.1 - 2018-08-02

Engineering Task

[SSA-4664] - Download tar product should be one .tar.gz
[SSA-4824] - DataFetcher retrieval of ID from metadata.json
[SSA-4825] - Solr Indexer Logging Upgrade
[SSA-4836] - Restore Workflow: Implement more stringent criteria for CASA success
[SSA-4849] - Remove hard link to pyat version
[SSA-4856] - Restructure Amygdala Package
[SSA-4861] - GPL it
[SSA-4864] - "About" link in navigation bar: pull URL from Capo
[SSA-4874] - General log message pruning, decrufting, and refactoring
[SSA-4878] - Add user's email address to request parameters
[SSA-4881] - ArchiveWorkflowStartupTask convenience method to announce spool directory
[SSA-4901] - Create and Use a Proper Staging Directory
[SSA-4902] - Extend Calibration Ingestion to Use a New Staging Location

Feature

[SSA-4380] - Additional User Updates for Workflows
[SSA-4627] - Better Feedback From qa Scripts
[SSA-4850] - MotD update without full page reload
[SSA-4855] - Information Return for AAT/PPI command line interfaces.
[SSA-4863] - Capo-ify the splash screen and release notes message and link

Bug

[SSA-4660] - VLBA Files Missing From Search Interface
[SSA-4718] - nmprod storage dir tar file restoration
[SSA-4833] - Request Handler UI Issues
[SSA-4834] - Webtest Running Out of Database Connections
[SSA-4854] - Incorrect MD5SUM output
[SSA-4865] - User-edited request description is overwritten or disregarded
[SSA-4872] - Investigate Data Fetcher thread utilization
[SSA-4879] - Externalized Script Collisions
[SSA-4883] - DownloadNotificationBuilder: failure notification crashes if no error info is available
[SSA-4918] - Data fetcher fails under heavy load / add retry capability?
[SSA-4936] - Remove os.environment['USER'] from pyramid
[SSA-4940] - Restore Workflow: PPR missing a command