The goal of this page is to compile both issues with the New Archive (AAT) and features the Legacy Archive (LA) has but not the AAT. The issues posted will be reviewed by John Tobin and Anna Kapinska (archive sub-system scientists) for submission as Jira tickets or submission of a requested development project for SSA planning. See the example for an issue posting below, we ask that whomever records an issue put their name and date along side it such that we can reach out if further information is needed.
Example:
Real recorded issues start here:
- Changing the download default so that tar.gz is unselected by default (Emmanuel Momjian; 02/28/2022)
- Stephan made a ticket for it ( SSA-7344 - Getting issue details... STATUS ) so tar-ing is not selected by default.
- He will provide wget examples to be added into the science plone pages SSA-7421 - Getting issue details... STATUS
- He will also disable checksums by default ( SSA-7343 - Getting issue details... STATUS ).
- In AAT, currently you can download a calibration .tar file (if available) by clicking the icon in the Cals column. It would be nice to also access this from the same place as other download formats: Add to clipboard > Download > Choose download format > "Calibration files only". This is intuitive because the .tar file is already listed at the bottom of the download popup for use with requesting Calibrated MS. Additionally, this could allow for downloading several calibration tars from the same project at once (or from several projects if that capability is added for all download formats - separate ticket?). (Edward; 03/01/2022) SSA-7353 - Getting issue details... STATUS
- The AAT handles any data download request as 'pipeline processing' even if the request is for raw data (e.g., SDM-BDF). This might be a reason why it is significantly slower in the download process itself compared to the LA (reported by staff as well as users). Added to the frustration of external users is the lack of direct download to their lustre area ( SSA-7179 - Getting issue details... STATUS ). For the former, it will be highly worthwhile to enable more straight forward downloading when raw data are requested, i.e., removing the extra layer of treating these as 'pipeline process', so the speed is (hopefully) more comparable to the legacy archive's download (Emmanuel Momjian; 03/13/2022)
- The AAT does not support scripting. There are now several requests to have this supported in the new archive. This is a topic that has already been discussed with SW. There are now several examples on the use cases. Adding this here for completeness (Emmanuel Momjian; 03/14/22). Requirements have been submitted by JT for the May 6, 2022, planning meeting: Scriptable Query Interface to NRAO Archive.
- Download number limit max will be 100 data sets, but download volume limited to 10TB. Stephan made this change already OTF!
- Update (by Anna): there are now Select All requests, one for VLBA files within single segment (can be considered temporary until it gets superseded by the broader request) SSA-7185 - Getting issue details... STATUS , and a broader one accessible on many levels in the archive structure SSA-7352 - Getting issue details... STATUS
- VLA/VLBA Image archive contents should be migrated into AAT (https://archive.nrao.edu/archive/archiveimage.html) (04/28/2022: Frank Schinzel)
- (old) FITSAIPS files should be associated with actual project segments instead of being dumped into made-up segment (Anna Kapinska: 02-May-2022)
- This is what legacy archive used to do too (segment x), but in the new archive it should be fixed. See ticket SSA-7328 - Getting issue details... STATUS for details (that ticket is to be closed as it only requested that the FITSAIPS are ingested into new archive, the segment association is additional thing and needs a new ticket)
- There are 51 stray FITSAIPS projects that are left to sort out and ingest into new archive to AAT 4.2.0 SSA-7426 - Getting issue details... STATUS (Anna Kapinska: 03-May-2022)
- There are still UVFITS files missing in the new archive, e.g. project GP0024 (Anna Kapinska: 03-May-2022)
- For observing VLBA projects that incorporate other dishes (e.g. +Y1, +Y27 etc), the VLA files should be associated with the relevant VLBA segment and not considered as "mixed" projects. This has been descoped from ticket SSA-7185 - Getting issue details... STATUS but needs to be picked up in future AAT releases (Anna Kapinska: 03-May-2022)
Issues already in JIRA/Confluence:
| Done? | Title | JiRA ticket /Confluence page | Details |
|---|---|---|---|
| √ | create missing db entries for Flag.xml |
SSA-6273
-
Getting issue details...
STATUS
| Done |
| √ | fix ASDM.xml files missing FlagTable entities |
SSA-6274
-
Getting issue details...
STATUS
| Done |
| Gap Analysis Between Legacy and New Archive |
SSA-6222
-
Getting issue details...
STATUS
| can be done once RL finishes the AIPSFITS files | |
| Automated annotations for missing scans/BDF, Automated Error Recognition of Ingested Data | SSA-7190 - Getting issue details... STATUS https://open-confluence.nrao.edu/display/SRDP/Archive+Data+Annotation+Tools | ||
| archive should ask for confirmation to continue if the download or processing job is considered large | To be requested | The front end can show size estimates for various products. | |
| Tool for setting Proprietary Period of science observations on the VLA and the VLBA | https://open-confluence.nrao.edu/pages/viewpage.action?spaceKey=SPR&title=Tool+for+setting+Proprietary+Period+of+science+observations+on+the+VLA+and+the+VLBA | ||
| DA annotation tools/Archive data annotation tools | https://open-confluence.nrao.edu/display/SRDP/Archive+Data+Annotation+Tools | ||
| option to return UNTAR'd file does not work | SSA-6495 - Getting issue details... STATUS https://open-confluence.nrao.edu/display/SRDP/Product+lists+for+Basic+and+Calibrated+Measurement+Set+downloads | pipeline topic | |
| Indexing of VLA low band projects | SSA-7187 - Getting issue details... STATUS | ||
| Display of archive ingestion/observation issues |
SSA-7189
-
Getting issue details...
STATUS
| related to SSA-7190 | |
| Other tickets (so they are not forgotten) | |||
Proposal Contact Author not added to data.nrao.edu database | SSA-7465 - Getting issue details... STATUS | Bug when a project uses different PI and Contact Author | |
| A friendly error is needed when a users is not authorized to access proprietary data. | SSA-7474 - Getting issue details... STATUS |
For completeness, in the table below we give web pages that will not work once the legacy archive becomes internal only (we can move this to another confluence page if needed)
| Web page | Web link | JIRA ticket (if applicable) |
|---|---|---|
| my.nrao.edu dashboard (Under My Information - My Data) | https://my.nrao.edu/nrao-2.0/secure/Home.htm?index=1 | |
| VLA plone pages | https://science.nrao.edu/facilities/vla/archive/index | SW provided a wget example to be added to the science webpage. We need to add this example to the plone page, see the ticket SSA-7421 for the example. The email sent by the archive tool will include the wget command in the near future. SSA-7421 - Getting issue details... STATUS |
| VLBA plone pages | https://science.nrao.edu/facilities/vlba/facilities/data-archive/index | SW provided a wget example to be added to the science webpages. We should include the example (see SSA-7421) in the plone pages (VLA, and VLBA). The email sent by the archive tool will include the wget command in the near future. SSA-7421 - Getting issue details... STATUS |
| Computing guide | https://info.nrao.edu/computing/guide/cluster-processing/data-storage-and-retrieval | |
| Numerous CASA Guides referencing the archive | https://casaguides.nrao.edu/index.php?title=Main_Page |
| Data Delivery | |||
| Put warning that only one CMS at a time can be requested in AAT |
SSA-7106
-
Getting issue details...
STATUS
| could be a flyover text | |
| Data Delivery times out if tar file(s) are too large/time consuming | will not be an issue with taring not being the default. | ||
| attempted for 4.2.0 but is deferred | Don't restrict download number limit. | SSA-7352 - Getting issue details... STATUS | We would like no limit on the number of files, but keep the 10TB for the total downloaded volume. |
| 4.1.1 | open up local delivery restrictions (to /home and /lustre). | This should cover users with visitor accounts wanting to download data to lustre, and local staff wanting to download data to their local machines. We have a problem with the quota of the observer accounts (I downloaded 8 TB of data to my observer account!); CIS remains irresponsive! | |
| 4.1.1 | disable checksums by default | SSA-7343 - Getting issue details... STATUS | |
| 4.1.1 | make tar downloads default to 'off' | SSA-7344 - Getting issue details... STATUS | |
| archive-new delivery not chmod'd/chowned | |||
| Bulk download of Calibration tarballs | SSA-7353 - Getting issue details... STATUS | ||
| legacy VLA data deliveries are one level deeper in directories than an SDM |
SSA-7000
-
Getting issue details...
STATUS
| Will happen with workspaces later |
| Search Interface | |||
| Frequency-dependent FOV searches | SSA-4915 - Getting issue details... STATUS | ||
| Position searches provide distance from searched position | SSA-7387 - Getting issue details... STATUS | User request, but is a legacy AAT feature | |
| Allow search return of observation scans? | SSA-7388 - Getting issue details... STATUS | Helpdesk ticket for a search return we do not have in AAT, but legacy archive provides. | |
Allow search filtering based on integration time. | SSA-7424 - Getting issue details... STATUS | Based on legacy archive helpdisk question from user. |
| VLBA Stuff | |||
import FITSAIPS files without matching projects (51 files) | SSA-7426 - Getting issue details... STATUS | ||
Add FITSAIPS files to new archive | |||
| old VLBA project metadata issues | SSA-7379 - Getting issue details... STATUS | ||
| Wrong band designation in VLBA data (old data only?) | SSA-7420 - Getting issue details... STATUS | this is in both the legacy archive and the new archive tool. | |
| Add wget command in emails generated by the AAT upon downloading data | SSA-7421 - Getting issue details... STATUS | ||
| √ | Remove Legacy Archive label and link from new archive | SSA-7375 - Getting issue details... STATUS | The new archive has a link to the legacy archive at the top. This needs to be removed since the legacy archive will not be accessible from outside. |
| √ | Download number limit: 100 data sets, but download data volume limit to 10TB per request. | This change has already been made by Stephan, but 100 files is too limiting for various cases (take BM360 as one example). We would more likely want no limit on the number of files, but keep the 10TB for the total downloaded volume. | |
| 877 VLBA correlations with missing files entries | SSA-7471 - Getting issue details... STATUS | ||
| √ | VLBA data structure in the new archive | done | |
| VLBA data structure in the new archive - improved interface | SSA-7185 - Getting issue details... STATUS Confluence: GUI (segment hierarchy) for VLBA data in the new archive | ||
| Default Instrument of VLBA projects in AAT/PPI | SSA-7208 - Getting issue details... STATUS Confluence req: VLBA: Default Instrument of VLBA projects in AAT/PPI | ||
| VLBA: DA archive tools (note specifically the limitation of USNO data not having PI) | https://open-confluence.nrao.edu/display/SPR/VLBA%3A+DA+archive+tools proprietary period setter: SSA-7350 - Getting issue details... STATUS | needs follow up, somethings are inconsistent (especially the topic of the tool's readiness to change the proprietary period of VLBA data). | |
| proprietary clock for VLBA data | SSA-6552 - Getting issue details... STATUS | should have been done for 4.1.0. Stephan will follow up to confirm. | |
| VLBA Reingestion Refinements | SSA-5917 - Getting issue details... STATUS | ongoing | |
| Mark4 files with 0 length getting added to new archive metadata | Small number of files affected. | ||
| √ | VLBA Polarization Duplication | SSA-5640 - Getting issue details... STATUS | done |
| Incomplete VLBA data of observations crossing midnight (open in case more cases show up) | SSA-6916 - Getting issue details... STATUS | ||
| √ | String Terminators in VLBA fields | SSA-5641 - Getting issue details... STATUS | done |
| √ | BL473 unlock and missing segments | SSA-7148 - Getting issue details... STATUS | done |
| Create USNO download acknowledgement | SSA-6952 - Getting issue details... STATUS | ||
| √ | USNO Proprietary Download System/A way to deliver data from projects that are not archived | done |
| VLA Metadata Presentation | |||
| Metadata Issues in Legacy VLA data | SSA-7192 - Getting issue details... STATUS | Needs input from Bryan | |
| VLA Metadata display updates. | https://open-confluence.nrao.edu/display/SRDP/VLA+Metadata+Display+Updates | This has many items including 'telescope observing logs'. | |
Incorrect metadata sometimes displayed for projects with duplicate proposal IDs between VLA and VLBA | SSA-7318 - Getting issue details... STATUS | ||
| Legacy VLA data missing error codes in the new archive tool | SSA-7191 - Getting issue details... STATUS | Mapping of quality errors were lost. Stephan will continue following up with Bryan. Blocker for LA retirement. | |
| Make scanlist information useful | SSA-7188 - Getting issue details... STATUS https://open-confluence.nrao.edu/display/SRDP/VLA+Metadata+Display+Updates | This ticket has multiple points to make the new tool match the Legacy archive and to have the displayed data be much more useful. | |
| legacy VLA metadata error: EBs that claim to have run for 2+ days | SSA-7129 - Getting issue details... STATUS | requires re-ingestion | |
| Database has zeros for min/max frequencies for some archive files | SSA-7277 - Getting issue details... STATUS | ||
| √ | Subscans table holding both Radian and Degree positions | SSA-5275 - Getting issue details... STATUS | done |
| √ | archive search returning incorrect projects |
SSA-6192
-
Getting issue details...
STATUS
| done |
| VLA/VLBA proposal code degeneracy |
SSA-5538
-
Getting issue details...
STATUS
SSA-7318 - Getting issue details... STATUS | ||
| metadata errors in VLBA (BD114) and old VLA data in new archive |
SSA-7128
-
Getting issue details...
STATUS
SSA-7129 - Getting issue details... STATUS | This has been split into two tickets: one VLBA and one VLA |