Introduction
The AAT/PPI (Archive Access Tool/Pipeline Processing Interface) is designed to be a replacement for the NRAO's current archive tool (https://archive.nrao.edu) that gives astronomers the ability to utilize the NRAO's processing clusters to manipulate observation data using CASA. This preliminary release provides the capability to run EVLA and ALMA observations through their respective calibration pipelines; additional capabilities will be forthcoming.
Additional features include:
- Responsive interface that works on mobile devices.
- Natural language (text based) searches of the observation's title, abstract, source names, authors, and more; e.g., search on 'nova'.
- Search results displayed as an Observations View (similar to the legacy archive) or a Projects View, which aggregates observations by project and also displays the title, abstract and other assorted details missing from the Observations View.
- Display a scans report for a given execution block, including the scan intents and subscans.
- The 'my data' feature: login, press a button, and see a list of your observations.
- If you link your ALMA and NRAO accounts, you can use either to access your observations.
- For ALMA and VLA you can download your SDM as a measurement set and filter by scan intent.
- For ALMA and VLA you can download a calibrated measurement set, which re-runs the calibration pipeline on your SDM.
- For VLA you can download a calibrated measurement set restored from a previously run calibration pipeline.
Known Issues
- The search interface for this release of the AAT/PPI is built on a completely new database of metadata, redesigned around the needs of VLASS and SRDP (ingestion of calibration products, observations with thousands of sub-scans and so on). We have ingested metadata from ALMA and the Jansky VLA into it, but it lacks metadata for the GBT, legacy VLA, and VLBA. Work is underway to address these issues, but until it is complete users are advised to use the legacy archive instead.
Release Notes - 3.0.0 - 2017-12-18
Engineering Task
- [SSA-3946] - tomcat, rabbitmq connections
- [SSA-4216] - Add calibrations to project view execution blocks
- [SSA-4266] - Implement calibration downloads in the front end
- [SSA-4350] - front end tweaks
- [SSA-4384] - Re-integrate ALMA and VLBA products
Feature
- [SSA-4057] - Restore Workflow - front end changes
- [SSA-4242] - front end, calibration status and download options
- [SSA-4252] - Calibration Download Workflow
- [SSA-4278] - revive 'restore' workflow
- [SSA-4341] - scan intent filtering, custom options per EB
- [SSA-4369] - Project view - exec-block table refinements
- [SSA-4395] - Deliver Calibrated MS on a Reprocess
- [SSA-4401] - offer 'sdm only' downloads for ALMA
- [SSA-4408] - Casa Output Verification
Bug
- [SSA-4077] - local delivery issues
- [SSA-4210] - failure mode to catch
- [SSA-4246] - Log messages in workflow.log that don't show up in dumplogs output
- [SSA-4308] - observation stop sort
- [SSA-4314] - DatabaseConnectionManager.getDataSource() throws RuntimeException
- [SSA-4315] - tar 'local' delivery bug
- [SSA-4359] - null subdirectory in download path
- [SSA-4398] - ALMA reingestion issues
- [SSA-4402] - early VLA eb's zero number of scans
- [SSA-4412] - Specify a walltime option on qsub for some of our jobs
- [SSA-4424] - 'reprocessing' bug
- [SSA-4433] - more authors/ingestion foo
- [SSA-4445] - queries by position busted
Release Notes - 3.1.0 - 2018-04-18
Engineering Task
- [SSA-4274] - DataFetcher refactor
- [SSA-4458] - VLBA Downloads
- [SSA-4467] - Simplify and streamline connection pool
- [SSA-4479] - Remove database queries from PprCreator
- [SSA-4571] - Check for presence of files in source before delivery and in destination afterwards
- [SSA-4577] - Include tail of CASA log in all result emails
- [SSA-4587] - Streamline workflow request notification
- [SSA-4631] - Improve workflow threading
- [SSA-4639] - Hanging executor in SubmissionResource
- [SSA-4643] - Schema changes for Image and image_products tables (testdb)
- [SSA-4649] - Update CIPL reprocessing to handle new CIPL delivery
- [SSA-4650] - Consolidate email code in workflows
- [SSA-4652] - Port Request Handler DB to PostgreSQL
- [SSA-4689] - Include failure details in download-failure email
- [SSA-4700] - Keep track of workflow state
Feature
- [SSA-4264] - /tmp/ Directory Naming
- [SSA-4381] - Reinstate the ALMA workflows
- [SSA-4426] - File Access Audit
- [SSA-4495] - Create Image Ingestion Workflow
- [SSA-4510] - move CIPL launch logic into amygdala
- [SSA-4517] - CIPL Start Email
- [SSA-4527] - Ingestion CLIs
- [SSA-4544] - workflows use provided CASA
- [SSA-4547] - add CASA option to CLI tools
- [SSA-4669] - CLI for the RestoreToCache Workflow
- [SSA-4691] - Workflows should specify qsub options for failure notifications
- [SSA-4703] - Move CASA intents and procedures file names to CAPO
- [SSA-4722] - Update BDF and SDM ingestion for realfast
Bug
- [SSA-4461] - mr_clean: sym links
- [SSA-4514] - Profile Deduction Fails for calFileSets (and probably other clis)
- [SSA-4515] - Workflows use the wrong start date for their directory creation.
- [SSA-4519] - Handling of Weblogs For Calibration Ingestion
- [SSA-4558] - Local delivery fails if MD5SUMS already present in target directory
- [SSA-4578] - Missing MD5 checksum file.
- [SSA-4621] - nmprod MS cache area has world read/write permissions for some MSs
- [SSA-4633] - Archive file size disagreements
- [SSA-4641] - CIPL task failure email notification not sent
- [SSA-4672] - Finished CIPL jobs have not moved to QA2
- [SSA-4673] - calFileSets Script Not Starting Jobs, CIPL
- [SSA-4701] - Capo-ify and Update Proceedures File Handling
- [SSA-4719] - Local delivery of CAL download fails
- [SSA-4727] - RH Authorization not handling files correctly