About the NRAO Archive Access Tool

Table of Contents

  1. About the NRAO Archive Access Tool

  2. Searching

  3. Authentication and Authorization

  4. Reprocessing

  5. Known Issues

    1. Search Index and Database

    2. Performance

    3. Reprocessing Errors

  6. Reporting Bugs (NRAO staff)

  7. Notes to the Users Committee

Introduction

This is the updated NRAO Science Data Archive. It is a tool to provide access to the vast repository of astronomical data we’ve collected over the years, both public data and data still in the proprietary period: data you have proposed for and observed or been given access to.

We are also offering the option to reprocess raw data and retrieve the results. Using the resources we have available, we can help you reduce your data.

This release is the culmination of the Phase II development effort. Everything present in Phase I is still present and operational. The focus in this release has been on three functional areas: search criteria, authentication and reprocessing, additionally the interface was re-designed to be responsive and cleaner.

Searching

The tool offers both a  basic interface and an  advanced interface form with a multitude of fields you can search on, including:

  • Telescope

    • ALMA array type (12m, 7m, etc)

    • VLA array configuration

    • ALMA maximum resolvable scale

    • GBT backends and receivers

  • Receiver band

  • Polarizations

  • Start/end date

  • Observation ID

  • Project code / title / abstract / PI / authors

  • Cone search source position with RA/Dec, coordinate system, and search radius

  • Full-width/half-max

  • Spatial/spectral resolution

  • Low/high frequency

  • Exposure time

In the  advanced interface each search field is  ANDed together, multiple selections within a field are  ORed , for example if you select Telescope: VLA, VLA Array Configurations: A, D, and Start Date: 2015-01-01, you are searching for any VLA observation that started on or after January 1, 2015 and was in A or D configurations.

The  basic interface offers a quick way to search the archive for specific topics, authors, source names and so on, and checks the text you provide against all of the fields above.

Authentication and Authorization

It has always been possible to submit download requests anonymously for non-proprietary data. You may now also log in and download or reprocess your proprietary data or data that you have been granted access to.

The archive also supports SSO level 2, so you may authenticate with either your ALMA or NRAO account. If the accounts are linked at ALMA, you will see the same stuff either way.<

Reprocessing

Reprocessing requests are now supported for raw data. ALMA requests and VLA requests for data taken after 2013 are functional (Jansky VLA sets from before 2013 had issues with intents  and can't be reprocessed without manual intervention.

Known Issues

Search Index and Database

The Archive seaches a combined index of ALMA and non-ALMA observations, this index is built from a database that has certain inconsistencies. We are in the middle of reimplementing the database and refining the procedure that builds it from the legacy database, meanwhile you can find results that are missing essential fields (start or stop dates, array configurations and so on). We have no evidence of observations with incorrect fields, just fields that are vacant.

Performance

Unlike the ALMA archive, the NRAO archive stages downloads before it allows access to them, meaning the files composing a file set are pulled from long term storage and assembled before presenting the option to download the files. This release of the NRAO Archive is hosted in NRAO's New Mexico facility, so staging ALMA files means streaming them from Virginia, which is not optimal. We plan to address this before the tool goes into production by hosting the download and reprocessing capabilities of ALMA data in Virginia and non-ALMA NRAO data in New Mexico.

Reprocessing Errors

The NRAO Archive offers primitive re-processing capabilities, meaning it can re-run raw data files through the ALMA or NRAO calibration pipeline and allow downloading the results. These features are very much a work in progress, one area that needs improvement is error reporting, the Archive currently allows users to reprocess data sets that are known to be bad, and there is no feedback to the user except that the request is complete and produced no files to download.

Reporting Bugs (NRAO Staff)

We will track bugs in Open JIRA. These should be reported to the 'SSA' Project and 'AAT/PPI' Component (not 'AAT, we use that for the current archive system). If it is easier you can submit a bug by emailing 'jira-ssa@nrao.edu'. We will also keep track of the more significant bugs in the Known Bugs section of this document, so new testers can be made aware of them.

Notes to the Users Committee

Please contact Mark Lacy (mlacy@nrao.edu) and Gustaaf van Moorsel (gvanmoor@nrao.edu) to report bugs, ask about issues and make feature suggestions. The areas we'd like the most help testing is authentication and authorization.

  • You shouldn't be able to reprocess an observation without logging in.
  • If the observation is within the proprietary period, you should have to be a proposer to get access to it.

Note that metadata is public, you are only challenged to authenticate when y

  • No labels