Below is a list of items that are priorities for Archive implementation, some of these are Legacy AAT retirement tickets, others are new development. The order in the tables is in priority order.
Main topics (click link to jump to section in page):
VLA(VLBA) Metadata presentation
Data Delivery
| Priority/Done? | Notes | JiRA ticket /Confluence page | |
| Critical | Downloads via Scriptable query interface | ||
| critical | A friendly error is needed when a users is not authorized to access proprietary data. | SSA-7474 - Getting issue details... STATUS | confusing to staff as well -need to kill RH, WS 3.0+ |
| medium | Put warning that only one CMS at a time can be requested in AAT |
SSA-7106
-
Getting issue details...
STATUS
| could be a flyover text (currently has a flyover circle with slash) |
| high (WS 2.8/2.9) | Clean up the nested directory data delivery mess legacy VLA data deliveries are one level deeper in directories than an SDM | Delivery Directory Improvements SSA-7000 - Getting issue details... STATUS | WS 2.7 or 3.0? |
| attempted for 4.2.0 but is deferred | Don't restrict download number limit. New ticket to remove the '/100 string from webpage' | We would like no limit on the number of files, but keep the 10TB for the total downloaded volume. | |
| 4.2.1 | Fix select-all functionality
| SSA-7573 - Getting issue details... STATUS | |
| medium | Download weblogs directly (and/or serve them as well!) | Direct weblog download from archive search return interface | |
| medium | Bulk download of Calibration tarballs | SSA-7353 - Getting issue details... STATUS | |
| low | archive should ask for confirmation to continue if the download or processing job is considered large | To be requested | The front end can show size estimates for various products. |
| low | Data Delivery times out if tar file(s) are too large/time consuming | will not be an issue with taring not being the default. | |
| low | archive-new delivery not chmod'd/chowned | ||
| √ 2022.2 pipeline | option to return UNTAR'd file does not work | SSA-6495 - Getting issue details... STATUS https://open-confluence.nrao.edu/display/SRDP/Product+lists+for+Basic+and+Calibrated+Measurement+Set+downloads | pipeline topic |
| √ (4.2.0) | The email sent by the archive tool will include the wget command in the near future. | SSA-7421 - Getting issue details... STATUS | |
| √ | open up local delivery restrictions (to /home and /lustre). | This should cover users with visitor accounts wanting to download data to lustre, and local staff wanting to download data to their local machines. We have a problem with the quota of the observer accounts (I downloaded 8 TB of data to my observer account!); CIS remains irresponsive! | |
| √ | disable checksums by default | SSA-7343 - Getting issue details... STATUS | |
| √ | make tar downloads default to 'off' | SSA-7344 - Getting issue details... STATUS |
Search interface/Data Selection
| Priority/Done? | Notes | ||
| high | Fix Frequency-dependent FOV and position searches
| ||
| high (4.2.2) | Position searches provide distance from searched position and sort by distance
| SSA-7387 - Getting issue details... STATUS | User request, but is a legacy AAT feature |
| medium | Treat wild cards consistently in AAT search interface
| SSA-7496 - Getting issue details... STATUS | |
| low | Allow search return of observation scans? | SSA-7388 - Getting issue details... STATUS | Helpdesk ticket for a search return we do not have in AAT, but legacy archive provides. |
| high | Allow search filtering based on total time on source. | SSA-7424 - Getting issue details... STATUS | Based on legacy archive helpdisk question from user. UC feedback as well. |
| medium | Fix mosaic position searches for VLASS and ALMA mosaics (subscan vs. scan) | ||
| Rearrange boxes in 'Search Inputs' | Ticket to be created. | ||
| √ (4.2.0) | Scriptable query interface | Scriptable Query Interface to NRAO Archive SSA-7395 - Getting issue details... STATUS | |
| √ | Download number limit: 100 data sets, but download data volume limit to 10TB per request. Doesn't fully solve problem see also SSA-7352 - Getting issue details... STATUS | This change has already been made by Stephan, but 100 files is too limiting for various cases (take BM360 as one example). We would more likely want no limit on the number of files, but keep the 10TB for the total downloaded volume. |
VLBA Stuff
| Priority/Done? | Notes | JiRA ticket /Confluence page | |
| High | VLBA: DA archive tools (note specifically the limitation of USNO data not having PI)
| https://open-confluence.nrao.edu/display/SPR/VLBA%3A+DA+archive+tools proprietary period setter: SSA-7350 - Getting issue details... STATUS | needs follow up, some things are inconsistent (especially the topic of the tool's readiness to change the proprietary period of VLBA data). |
| High | VLBA+EVLA projects without metadata and sometimes also mismatched PI | SSA-7423 - Getting issue details... STATUS | |
| High (4.2.1) | Proxy issue for huge VLBA projects | SSA-7436 - Getting issue details... STATUS | |
| Med | Wrong band designation in VLBA data (old data only?) | SSA-7420 - Getting issue details... STATUS | This is in both the legacy archive and the new archive tool. Issue needs more research on how prevalent it is; unclear. |
| Med |
| SSA-7609 - Getting issue details... STATUS | Ticket created Anna Kapinska, Emmanuel Momjian, and Anthony Sowinski should check it to make sure it's correct and specifies all requirements. |
| Med | Default Instrument of VLBA projects in AAT/PPI | SSA-7208 - Getting issue details... STATUS Confluence req: VLBA: Default Instrument of VLBA projects in AAT/PPI | |
| Med | Add PFT (proposal finder tool) link in Project View for VLBA projects | SSA-7461 - Getting issue details... STATUS | |
| Low | Import remaining 8 GMVA files | SSA-7438 - Getting issue details... STATUS | |
| Low | Mark4 files with 0 length getting added to new archive metadata | Small number of files affected. | |
| Low | Incomplete VLBA data of observations crossing midnight (open in case more cases show up) | SSA-6916 - Getting issue details... STATUS | |
| High | Create USNO download acknowledgement | SSA-6952 - Getting issue details... STATUS | |
| √ 4.2.0 | Add wget command in emails generated by the AAT upon downloading data | SSA-7421 - Getting issue details... STATUS | |
| √ 4.1.1 | import FITSAIPS files without matching projects (51 files) and project GP0024 | SSA-7426 - Getting issue details... STATUS | |
| √ 4.1.1 | Add FITSAIPS files to new archive and segment association | ||
| √ 4.1.1 | old VLBA project metadata issues | SSA-7379 - Getting issue details... STATUS | |
| √ 4.1.1 | VLBA data structure in the new archive - improved interface | SSA-7185 - Getting issue details... STATUS Confluence: GUI (segment hierarchy) for VLBA data in the new archive | |
| √ (4.1.1) | Database has zeros for min/max frequencies for some archive files | SSA-7277 - Getting issue details... STATUS | |
| √ | 877 VLBA correlations with missing files entries | SSA-7471 - Getting issue details... STATUS | |
| √ | proprietary clock for VLBA data | SSA-6552 - Getting issue details... STATUS | done |
| √ | VLBA Reingestion Refinements | SSA-5917 - Getting issue details... STATUS | ongoing → done |
| √ | VLBA Polarization Duplication | SSA-5640 - Getting issue details... STATUS | done |
| √ | String Terminators in VLBA fields | SSA-5641 - Getting issue details... STATUS | done |
| √ | BL473 unlock and missing segments | SSA-7148 - Getting issue details... STATUS | done |
| √ | VLBA data structure in the new archive | done | |
| √ | USNO Proprietary Download System/A way to deliver data from projects that are not archived | done | |
| √ | Remove Legacy Archive label and link from new archive | SSA-7375 - Getting issue details... STATUS | The new archive has a link to the legacy archive at the top. This needs to be removed since the legacy archive will not be accessible from outside. |
VLA Metadata Presentation
| Priority/Done? | Notes | JiRA ticket /Confluence page | |
| Critical (4.3.0) | Make scanlist information useful | SSA-7188 - Getting issue details... STATUS https://open-confluence.nrao.edu/display/SRDP/VLA+Metadata+Display+Updates JT: revise to capture what we want about the frequency/setup information | This ticket has multiple points to make the new tool match the Legacy archive and to have the displayed data be much more useful. |
| High (4.3/4.4) | VLA Metadata display updates. | https://open-confluence.nrao.edu/display/SRDP/VLA+Metadata+Display+Updates | This has many items including 'telescope observing logs'. |
| Medium | Incorrect metadata sometimes displayed for projects with duplicate proposal IDs between VLA and VLBA | SSA-7318 - Getting issue details... STATUS | |
| Medium | VLA/VLBA proposal code degeneracy |
SSA-5538
-
Getting issue details...
STATUS
SSA-7318 - Getting issue details... STATUS | |
| Medium | Indexing of VLA low band projects | SSA-7187 - Getting issue details... STATUS | |
| √ | Subscans table holding both Radian and Degree positions | SSA-5275 - Getting issue details... STATUS | done |
| √ | archive search returning incorrect projects |
SSA-6192
-
Getting issue details...
STATUS
| done |
Legacy VLA Data
| Priority/Done? | Notes | JiRA ticket /Confluence page | |
| High | Metadata Issues in Legacy VLA data | SSA-7192 - Getting issue details... STATUS | Needs input from Bryan |
| High | Legacy VLA data missing error codes in the new archive tool | SSA-7191 - Getting issue details... STATUS | Mapping of quality errors were lost. Stephan will continue following up with Bryan. Blocker for LA retirement. |
| High | metadata errors in VLBA (BD114) and old VLA data in new archive |
SSA-7128
-
Getting issue details...
STATUS
SSA-7129 - Getting issue details... STATUS | This has been split into two tickets: one VLBA and one VLA |
| Medium | legacy VLA metadata error: EBs that claim to have run for 2+ days | SSA-7129 - Getting issue details... STATUS | requires re-ingestion |
Data Integrity
| Priority/Done? | Notes | JiRA ticket /Confluence page | |
| Blocker | Gap Analysis Between Legacy and New Archive |
SSA-6222
-
Getting issue details...
STATUS
| can be done once |
| Critical | Automated annotations for missing scans/BDF, Automated Error Recognition of Ingested Data | SSA-7190 - Getting issue details... STATUS https://open-confluence.nrao.edu/display/SRDP/Archive+Data+Annotation+Tools | |
| Critical | Display of archive ingestion/observation issues |
SSA-7189
-
Getting issue details...
STATUS
| related to SSA-7190 |
| Critical | Identification of ingestion problems and automaticallyfix | Example ticket of a silent failure: SSA-7566 - Getting issue details... STATUS | |
| √ | create missing db entries for Flag.xml |
SSA-6273
-
Getting issue details...
STATUS
| Done |
| √ | fix ASDM.xml files missing FlagTable entities |
SSA-6274
-
Getting issue details...
STATUS
| Done |
DA Tools for Data Editing
| Priority/Done? | Notes | JiRA ticket /Confluence page | |
| √ | Tool for setting Proprietary Period of science observations on the VLA and the VLBA | https://open-confluence.nrao.edu/pages/viewpage.action?spaceKey=SPR&title=Tool+for+setting+Proprietary+Period+of+science+observations+on+the+VLA+and+the+VLBA | |
| Critical | DA annotation tools/Archive data annotation tools | https://open-confluence.nrao.edu/display/SRDP/Archive+Data+Annotation+Tools |
Missing Data
| Priority/Done? | Notes | JiRA ticket /Confluence page | |
| Critical |
| Tix to be created May require discussion about how new archive will display images and the uvfits datasets. | |
| High | ALMA Standard images into AAT | SSA-7278 - Getting issue details... STATUS | |
| High | Large project support - ALFALFA + Other collection ingest | SSA-7437 - Getting issue details... STATUS SSA-5712 - Getting issue details... STATUS ALFALFA: SRAO and Collection Support |
Other
| Priority/Done? | Notes | JiRA ticket /Confluence page | Details |
|---|---|---|---|
| Other tickets (so they are not forgotten) | Tickets remaining are listed throughout their respective categories. | ||
| Critical | Proposal Contact Author not added to data.nrao.edu database | SSA-7465 - Getting issue details... STATUS | Bug when a project uses different PI and Contact Author |
| Medium | Preserve login between AAT and my.nrao.edu | SSA-7322 - Getting issue details... STATUS | |
| √ (4.2.0) | Remove pop-up page when accessing data.nrao.edu | SSA-7502 - Getting issue details... STATUS |
ALMA
| Priority/Done? | Notes | JiRA ticket /Confluence page | Details |
|---|---|---|---|
| Critical | ALMA MOUS/calibration ingest unreliable; ALMA butler implementation into production | SSA-7482 - Getting issue details... STATUS |
Product Versioning
| Priority/Done? | Notes | JiRA ticket /Confluence page | Details |
|---|---|---|---|
| Critical | Need to allow multiple versions of products (images/calibrations) in the archive | Product Versioning and Replacement Design SSA-6476 - Getting issue details... STATUS |