With the decommissioning of the legacy archive, there will no longer be a mechanism to query the NRAO archive without using a web browser. Having a scriptable interface to the NRAO archive is functionality that must be regarded as essential.
The scriptable interface must support all currently searchable metadata on the NRAO archive using the search interface. This search interface MUST be compatible with astroquery, but more generally it would be better to implement the interface using VO standards like TAP (Table Access Protocol) and SIAP (Simple Image Access Protocol) for access to FTIPS images. Moreover, the use of TAP should be able to query from the full array of metadata available for data in the archive and not only what is exposed in the web based search. Our goal should be to implement an interface that is comparable to ALMA's in functionality via astroquery and TAP/SIAP.
Ultimately, we also want users to be able to download data from this interface, EBs, FITS images, etc. However, we recommend that this implementation be staged. First, enable users to query the metadata and later we can add the capability to download through the scriptable interface. Thus, the interface should be designed with the goal of downloading in mind and not solely metadata queries. How to make the scriptable interface interact with requests for calibrated measurement sets and possibly AUDI images will have to be discussed further.
If possible, it would be useful to also enable access outside of astroquery using URL-specified search parameters, similar to what could be done with the legacy archive and provide the returns in machine readable format. For example:
curl -d 'SUBMIT=Submit+Query&QUERY_MODE=Prepare+Download&PROJECT_CODE=AS1037' https://archive.nrao.edu/archive/ArchiveQuery (the download directory would also be a parameter=value)
Example Use Cases:
- A user that has identified a transient event from other means (e.g., ZTF or Rubin), wants to see if calibrated data are available in the archive from VLASS. So the user would want to query on position with some search radius, only for VLASS data in the project title or FSID, and perhaps limited in time. In addition to the return of EBs and images, the user may also want to download the list of returned images and they should be able to do so without going to the web search interface. The user may also want to open those images in CARTA rather than downloading, which should also be a supported option. There should be an ability to request a CARTA session and the user is provided their URL for viewing.
- A user plans to propose for VLA time toward a particular object. They search what's available and want to see how good those data are. So the search return should provide EBs, images, and the image metadata which should include the RMS sensitivity, and beam size information. A user should also be able to directly download that image through the search interface or open the image in CARTA, in which case we would need a way to direct the user to the CARTA session by opening their browser or by returning a link within the archive interface.
- A user wants to see what data are available toward a science target. There are three EBs available with calibrations in the archive. The user wants to request those EBs as calibrated measurement sets.
(from A. Ginsburg) A user has a program being executed in separate SBs and wants to download the data as soon as they are available. They'll regularly run queries (cronjobs) to determine if new data are available and trigger the downloads. (this is a common case for large programs, e.g., the X-program)
(from A. Ginsburg) A user wants to monitor the variability of a quasar that is commonly used as a calibrator. They should be able to search for all programs that included that target as a calibrator or science target and the integration time and coverage from each observation.
- (from C. Casadio/User Helpdesk Ticket) A user wants to search database and download all data for 5000 sources. They should be able to do it all in a one query without human intervention for download (e.g. pop up window)
References:
VO Concepts: https://ivoa.net/deployers/intro_to_vo_concepts.html
Astroquery interface for ALMA data: https://astroquery.readthedocs.io/en/latest/alma/alma.html
pyVO: https://pyvo.readthedocs.io/en/latest/#