Page History

About the NRAO Archive Access Tool

Introduction

The AAT/PPI (Archive Access Tool/Pipeline Processing Interface) is designed to be a replacement for the NRAO's current archive tool (https://archive.nrao.edu) that gives astronomers the ability to utilize the NRAO's processing clusters to manipulate observation data using CASA: this preliminary release provides the capability to run EVLA and ALMA observations through their respective calibration pipelines, additional capabilities will be forthcoming.

...

• Two interfaces, basic and advanced.
• Responsive interface that works on mobile devices.
• Natural language (text based) searches of the observation's title, abstract, source names, authors and more, e.g. search on 'nova'.
• The 'my data' feature: login, press a button and see a list of your observations.
• If you link your ALMA and NRAO accounts, you can use either to access your observations.
• For ALMA and VLA you can download your SDM as a measurement set and filter by scan intent.

Testing and Feedback

We'd like testers to focus on authentication and authorization: before we allow outside access to this tool we'd like a high degree of confidence we won't be exposing proprietary data through it: we have tested it as well as we could but feel it could benefit from broader testing.

...

Please report your feedback to Mark Lacy (mlacy@nrao.edu), Claire Chandler (cchandle@nrao.edu) and Stephan Witz (switz@nrao.edu).

Known Issues

• Stale search indexes: the ALMA search index is re-built daily, so it should never be more than a day out of date: the VLA/VLBA/GBT search index is built less frequently and can be out of date by a week or more. We are working on it.
• Missing fields: the NRAO metadata database which we use to populate the search index spans four decades of observations, some of the observations have missing fields like array configuration or start and stop dates. We have no evidence of incorrect fields, just missing fields. There is no easy fix for the issue, we will have to extract each affected observation and update the database from the metadata in the files, we will start that after the software is released.
• Poor error feedback: there are many kinds of failures possible (observations that can't run through the calibration pipeline, observations that are missing some files, permissions errors writing files to user specified un-writable locations and so on), the AAT/PPI doesn't currently let the user know why something failed, and sometimes it won't let the user know it failed at all, requests never seem to complete. This is something we are actively working on.
• At this time only the calibration and flagging tables are downloadable after reprocessing, not the full calibrated measurement set.
• ALMA Cycle 0 and Cycle 1 observations are known to fail reprocessing, as are VLA observations before 2013, there is currently no feedback to the user that is the case. Users are advised to avoid reprocessing on these sets.
• GBT data delivery format: the SDFITS format for GBT data is not yet supported. GBT data can only be downloaded as a set of GBT-FITS files.
• The distributed applications that make up the AAT/PPI are fragile to database downtime, they don't recover well from it yet and have to be restarted to resume functioning: this is exacerbated by the fact that the system relies on several databases to work, not all of which are under the team's control. We are working to reduce the number of databases and to also make the system more robust.

Release Notes - 2.5.2 - 2017-05-04

<h4>        Feature
</h4>
<ul>
<li>[<a href='SSA-3988https://open-jira.nrao.edu/browse/SSA-3988'>SSA-3988</a>] -         Announcement banner / MOTD in the new archive
</li>
<li>[<a href='SSA-3991https://open-jira.nrao.edu/browse/SSA-3991'>SSA-3991</a>] -         pre-populate download/reprocessing email address
</li>
<li>[<a href='SSA-3994https://open-jira.nrao.edu/browse/SSA-3994'>SSA-3994</a>] -         filter by scan intent
</li>
<li>[<a href='SSA-3995https://open-jira.nrao.edu/browse/SSA-3995'>SSA-3995</a>] -         front end touches
</li>
<li>[<a href='SSA-3998https://open-jira.nrao.edu/browse/SSA-3998'>SSA-3998</a>] -         position input validation
</li>
</ul>
    
<h4>        Bug
</h4>
<ul>
<li>[<a href='SSA-3997https://open-jira.nrao.edu/browse/SSA-3997'>SSA-3997</a>] -         front end, floats and ints
</li>
<li>[<a href='SSA-3999https://open-jira.nrao.edu/browse/SSA-3999'>SSA-3999</a>] -         ArrayIndexOutOfBounds exception
</li>
</ul>

Release Notes - 2.5.3 - 2017-05-12

<h4>        Engineering Task
</h4>
<ul>
<li>[<a href='SSA-4010https://open-jira.nrao.edu/browse/SSA-4010'>SSA-4010</a>] -         Improve reliability of workflow-job signalling
</li>
</ul>
        
<h4>        Bug
</h4>
<ul>
<li>[<a href='SSA-4006https://open-jira.nrao.edu/browse/SSA-4006'>SSA-4006</a>] -         problems under load/maui kills jobs
</li>
</ul>
        

Release Notes - 2.5.4 - 2017-05-26

<h4>        Feature
</h4>
<ul>
<li>[<a href='SSA-3878https://open-jira.nrao.edu/browse/SSA-3878'>SSA-3878</a>] -         missing feature: NED/SIMBAD name resolution
</li>
<li>[<a href='SSA-3879https://open-jira.nrao.edu/browse/SSA-3879'>SSA-3879</a>] -         missing feature: different formats for source position input
</li>
<li>[<a href='SSA-4003https://open-jira.nrao.edu/browse/SSA-4003'>SSA-4003</a>] -         input position as HH:MM:SS.SSS
</li>
</ul>
    
<h4>        Bug
</h4>
<ul>
<li>[<a href='SSA-3983https://open-jira.nrao.edu/browse/SSA-3983'>SSA-3983</a>] -         RA/DEC notation defaults and range.
</li>
<li>[<a href='SSA-4033https://open-jira.nrao.edu/browse/SSA-4033'>SSA-4033</a>] -         local delivery permissions
</li>
</ul>

...