Below is a list of items that are priorities for Archive implementation, some of these are Legacy AAT retirement tickets, others are new development. The order in the tables is in priority order (aside from VLBA at the moment).
Data Delivery
.
Main topics (click link to jump to section in page):
VLA(VLBA) Metadata presentation
Data Delivery
| Anchor | ||||
|---|---|---|---|---|
|
| Priority/Done? | Notes | JiRA ticket /Confluence page | |
| Critical | Downloads via Scriptable query interface |
|
|
|
| ||||||||||||||
| critical | A friendly error is needed when a users is not authorized to access proprietary data. |
|
Clean up the nested directory data delivery mess
legacy VLA data deliveries are one level deeper in directories than an SDM
| confusing to staff as well -need to kill RH, WS 3.0+ | ||
| critical | Restore data using same version of CASA as calibrated |
|
|
| |||
| medium | Put warning that only one CMS at a time can be requested in AAT |
|
| could be a flyover text (currently has a flyover circle with slash) | ||
| high (WS 2.8/2.9) | Clean up the nested directory data delivery mess legacy VLA data deliveries are one level deeper in directories than an SDM |
|
| WS 2.7 or 3.0? | ||
| planned 4.2.2 | Don't restrict download number limit. New ticket to remove the '/100 string from webpage' |
|
|
|
| We would like no limit on the number of files, but keep the 10TB for the total downloaded volume. | ||||
| 4.2.1 | Fix select-all functionality
|
|
|
| |||
| medium | Download weblogs directly (and/or serve them as well!) |
|
|
|
|
|
|
| |||
| medium | Bulk download of Calibration tarballs |
|
Search interface/Data Selection
| |||
| low | archive should ask for confirmation to continue if the download or processing job is considered large | To be requested | The front end can show size estimates for various products. |
| low | Data Delivery times out if tar file(s) are too large/time consuming |
|
| will not be an issue with taring not being the default. | |||||||||||||
| low | archive-new delivery not chmod'd/chowned |
|
| |||
| √ 2022.2 pipeline | option to return UNTAR'd file does not work |
|
|
| pipeline topic | ||||||
| √ (4.2.0) | The email sent by the archive tool will include the wget command in the near future. |
|
|
| ||||||||||||||
| √ | open up local delivery restrictions (to /home and /lustre). |
|
Helpdesk ticket for a search return we do not have in AAT, but legacy archive provides.
The part related to opening the download destination to nm-### accounts on lustre is
|
| This should cover users with visitor accounts wanting to download data to lustre, and local staff wanting to download data to their local machines. We have a problem with the quota of the observer accounts (I downloaded 8 TB of data to my observer account!); CIS remains irresponsive! | |||||||
| √ | disable checksums by default |
|
|
|
Download number limit: 100 data sets, but download data volume limit to 10TB per request.
VLBA Stuff
| ||||||||
| √ | make tar downloads default to 'off' |
|
import FITSAIPS files without matching projects (51 files) and project GP0024
|
Search interface/Data SelectionAnchor Search Search
| Priority/Done? | Notes | ||||||||||||||||||||||||||||||||||||||
| high | Fix Frequency-dependent FOV and position searches
|
| old VLBA project metadata issues |
| |||||||||||||||||||||||||||||||||||
| high (4.2.2) | Position searches provide distance from searched position and sort by distance
|
| Wrong band designation in VLBA data (old data only?) |
| this is in both the legacy archive and the new archive tool. | Add wget command in emails generated by the AAT upon downloading data
| User request, but is a legacy AAT feature | ||||||||||||||||||||||||||||||||
| medium | Treat wild cards consistently in AAT search interface
|
| 7421
| ||||||||||||||||||||||||||||||||||||
| low | Allow search return of observation scans? | For observing VLBA projects that incorporate other dishes (e.g. +Y1, +Y27 etc), the VLA files should be associated with the relevant VLBA segment and not considered as "mixed" projects. This has been descoped from ticket
| 7185 but needs to be picked up in future AAT releases
| Helpdesk ticket for a search return we do not have in AAT, but legacy archive provides. | |||||||||||||||||||||||||||||||||||
| high | Allow search filtering based on total time on source. | 877 VLBA correlations with missing files entries |
| VLBA data structure in the new archive - improved interface |
Confluence: GUI (segment hierarchy) for VLBA data in the new archive | Default Instrument of VLBA projects in AAT/PPI |
| 7208
| Based on legacy archive helpdisk question from user. UC feedback as well. | ||||||||||||||||||||||||||||||
| medium | Fix mosaic position searches for VLASS and ALMA mosaics (subscan vs. scan) |
| VLBA: DA archive tools (note specifically the limitation of USNO data not having PI) | ||||||||||||||||||||||||||||||||||||
| Jira | server | DMS
| SSA
| 7350needs follow up, somethings are inconsistent (especially the topic of the tool's readiness to change the proprietary period of VLBA data). | proprietary clock for VLBA data
| ||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Rearrange boxes in 'Search Inputs' | Ticket to be created. | ||||||||||||||||||||||||||||||||||||||
| √ (4.2.0) | Scriptable query interface | Scriptable Query Interface to NRAO Archive
| |||||||||||||||||||||||||||||||||||||
| √ | Download number limit: 100 data sets, but download data volume limit to 10TB per request. Doesn't fully solve problem see also
| columnIds | issuekey,summary,issuetype
| 6552should have been done for 4.1.0. Stephan will follow up to confirm. |
| This change has already been made by Stephan, but 100 files is too limiting for various cases (take BM360 as one example). We would more likely want no limit on the number of files, but keep the 10TB for the total downloaded volume. |
VLBA StuffAnchor VLBA VLBA
| Priority/Done? | Notes | JiRA ticket /Confluence page | ||||||||||||||||||||||||||||||||||||
| High | VLBA: DA archive tools (note specifically the limitation of USNO data not having PI)
| https://open-confluence.nrao.edu/display/SPR/VLBA%3A+DA+archive+tools proprietary period setter: | VLBA Reingestion Refinements | |||||||||||||||||||||||||||||||||||
| Jira | ||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| server | DMS JIRA | |||||||||||||||||||||||||||||||||||||
| columnIds | issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution | columns | key,summary,type,created,updated,due
| |||||||||||||||||||||||||||||||||||
| serverId | eb2e750b-a83a-387e-8345-36eee8a98f01 | |||||||||||||||||||||||||||||||||||||
| key | SSA-5917 | ongoing | Mark4 files with 0 length getting added to new archive metadata |
server | DMS JIRA | columnIds | issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution | columns | key,
| Small number of files affected. | Incomplete VLBA data of observations crossing midnight (open in case more cases show up)
| The VLBA DA tools for changing PI and setting proprietary period have been complete for some time. The confluence page also mentions a tool for annotations; maybe that part should be moved to the Data Integrity section? | ||||||||||||||||||||||||||
| High | VLBA+EVLA projects without metadata and sometimes also mismatched PI |
| 6916
| |||||||||||||||||||||||||||||||||||
| High (4.2.1) | Proxy issue for huge VLBA projectsCreate USNO download acknowledgement |
| √ | VLBA Polarization Duplication
| ||||||||||||||||||||||||||||||||||
| High (4.2.1) | Slow VLBA indexing |
| 5640done | √ | String Terminators in VLBA fields
| 5641
| done | √ | BL473 unlock and missing segments | |||||||||||||||||||||||||||||
| Med | Wrong band designation in VLBA data (old data only?) |
| This is in both the legacy archive and the new archive tool. Issue needs more research on how prevalent it is; unclear. | |||||||||||||||||||||||||||||||||||
| Med |
| |||||||||||||||||||||||||||||||||||||
| columnIds | issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution | |||||||||||||||||||||||||||||||||||||
| columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution | |||||||||||||||||||||||||||||||||||||
| serverId | eb2e750b-a83a-387e-8345-36eee8a98f01 | |||||||||||||||||||||||||||||||||||||
| key | SSA-7148 | done | √ | VLBA data structure in the new archive | ||||||||||||||||||||||||||||||||||
| Jira | ||||||||||||||||||||||||||||||||||||||
| server | DMS JIRA | columnIds | issuekey,summary
| 7076
|
| Ticket created Anna Kapinska, Emmanuel Momjian, and Anthony Sowinski should check it to make sure it's correct and specifies all requirements. | ||||||||||||||||||||||||||||||||
| Med | Default Instrument of VLBA projects in AAT/PPI |
| 7182||||||||||||||||||||||||||||||||||||
| Jira | ||||||||||||||||||||||||||||||||||||||
| server | DMS JIRA | columnIds | issuekey,summary,issuetype,created,updated,duedate,assignee,
| 7183
And a new ticket created:
| 7184done | |||||||||||||||||||||||||||||||||
| √ | USNO Proprietary Download System/A way to deliver data from projects that are not archived | done | ||||||||||||||||||||||||||||||||||||
| Med | Add PFT (proposal finder tool) link in Project View for VLBA projects |
| √ | Remove Legacy Archive label and link from new archive | ||||||||||||||||||||||||||||||||||
| Jira | server | DMS JIRA
| 7375The new archive has a link to the legacy archive at the top. This needs to be removed since the legacy archive will not be accessible from outside. |
VLA Metadata Presentation
| ||||||||||||||||||||||||||
| Low | Import remaining 8 GMVA files | Priority/Done? | Notes | JiRA ticket /Confluence page | Make scanlist information useful |
| This ticket has multiple points to make the new tool match the Legacy archive and to have the displayed data be much more useful. | VLA Metadata display updates. | https://open-confluence.nrao.edu/display/SRDP/VLA+Metadata+Display+Updates | This has many items including 'telescope observing logs'. |
| |||||||||||||||
| Low | Mark4 files with 0 length getting added to new archive metadataIncorrect metadata sometimes displayed for projects with duplicate proposal IDs between VLA and VLBA | VLA/VLBA proposal code degeneracy | Small number of files affected. | |||||||||||||||||||||||
| Low | Incomplete VLBA data of observations crossing midnight (open in case more cases show up) |
| ||||||||||||||||||||||||
| High | Create USNO download acknowledgement | 5538
| 7318
| |||||||||||||||||||||||
| Remove Legacy Archive label and link from new archive | Indexing of VLA low band projects
| |||||||||||||||||||||||||
| columnIds | issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution | |||||||||||||||||||||||||
| serverId | eb2e750b-a83a-387e-8345-36eee8a98f01 | |||||||||||||||||||||||||
| key | SSA-7187 |
| √ | Subscans table holding both Radian and Degree positions |
| Update (26-Sep-2022, Anna): the link is still present on some of the AAT subpages, especially the download status page when request for data is submitted. The new archive has a link to the legacy archive at the top. This needs to be removed since the legacy archive will not be accessible from outside. | ||||||||||||||||||||
| √ 4.2.0 | Add wget command in emails generated by the AAT upon downloading data |
| ||||||||||||||||||||||||
| √ 4.1.1 | import FITSAIPS files without matching projects (51 files) and project GP0024 |
| ||||||||||||||||||||||||
| done | √ 4.1.1 | archive search returning incorrect projects Add FITSAIPS files to new archive and segment association |
| done |
Legacy VLA Data
| √ 4.1.1 | old VLBA project metadata issues | Priority/Done? | Notes | JiRA ticket /Confluence page | Metadata Issues in Legacy VLA data | ||||||||||||||||||||||||||||||||||||||||||||||
| Jira |
|
| |||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| √ 4.1.1 | VLBA data structure | 7192Needs input from Bryan | Legacy VLA data missing error codes in the new archive tool- improved interface |
Confluence: GUI (segment hierarchy) for VLBA data in the new archive | |||||||||||||||||||||||||||||||||||||||||||||||
| √ (4.1.1) | Database has zeros for min/max frequencies for some archive files |
| |||||||||||||||||||||||||||||||||||||||||||||||||
| √ | 877 VLBA correlations with missing files entries |
| |||||||||||||||||||||||||||||||||||||||||||||||||
| √ | proprietary clock for VLBA data | Mapping of quality errors were lost. Stephan will continue following up with Bryan. Blocker for LA retirement. | metadata errors in VLBA (BD114) and old VLA data in new archive |
| done | ||||||||||||||||||||||||||||||||||||||||||||||
| √ | VLBA Reingestion Refinements |
| This has been split into two tickets: one VLBA and one VLA |
| ongoing → done | ||||||||||||||||||||||||||||||||||||||||||||||
| √ | VLBA Polarization Duplication |
| done | ||||||||||||||||||||||||||||||||||||||||||||||||
| √ | String Terminators in VLBA fields |
| done | ||||||||||||||||||||||||||||||||||||||||||||||||
| √ | BL473 unlock and missing segments |
| done | ||||||||||||||||||||||||||||||||||||||||||||||||
| √ | VLBA data structure in the new archive |
| done | ||||||||||||||||||||||||||||||||||||||||||||||||
| √ | USNO Proprietary Download System/A way to deliver data from projects that are not archived | done | |||||||||||||||||||||||||||||||||||||||||||||||||
VLA Metadata Presentation Anchor VLAMetadata VLAMetadata
| Priority/Done? | Notes | JiRA ticket /Confluence page | |||||||||||||||||||||||||
| critical | Scan list does not show coordinates between dec= 00 and dec = -1 correctly |
| |||||||||||||||||||||||||
| Critical (4.3.0) | Make scanlist information useful |
JT: revise to capture what we want about the frequency/setup information | This ticket has multiple points to make the new tool match the Legacy archive and to have the displayed data be much more useful. | ||||||||||||||||||||||||
| High (4.3/4.4) | VLA Metadata display updates. | https://open-confluence.nrao.edu/display/SRDP/VLA+Metadata+Display+Updates
| This has many items including 'telescope observing logs'. | ||||||||||||||||||||||||
| Medium | Incorrect metadata sometimes displayed for projects with duplicate proposal IDs between VLA and VLBA |
| |||||||||||||||||||||||||
| Medium | VLA/VLBA proposal code degeneracy |
| |||||||||||||||||||||||||
| Medium | Indexing of VLA low band projects |
| |||||||||||||||||||||||||
| √ | Subscans table holding both Radian and Degree positions |
| done | ||||||||||||||||||||||||
| √ | archive search returning incorrect projects |
| done | ||||||||||||||||||||||||
Legacy VLA DataAnchor Legacy Legacy
| Priority/Done? | Notes | JiRA ticket /Confluence page | |||||||||||||||||||||||||
| Critical | corrupt legacy VLA data in the AAT |
| |||||||||||||||||||||||||
| High | Metadata Issues in Legacy VLA data |
| Needs input from Bryan | ||||||||||||||||||||||||
| High | Legacy VLA data missing error codes in the new archive tool |
| Mapping of quality errors were lost. Stephan will continue following up with Bryan. Blocker for LA retirement. | ||||||||||||||||||||||||
| High | metadata errors in VLBA (BD114) and old VLA data in new archive |
| This has been split into two tickets: one VLBA and one VLA | ||||||||||||||||||||||||
| Medium | legacy VLA metadata error: EBs that claim to have run for 2+ days |
| requires re-ingestion |
Data IntegrityAnchor Integrity Integrity
| Priority/Done? | Notes | JiRA ticket /Confluence page | |||||||||||||||||||||||||
| Blocker | Gap Analysis Between Legacy and New Archive |
| can be done once | ||||||||||||||||||||||||
| Critical | Automated annotations for missing scans/BDF, Automated Error Recognition of Ingested Data |
| |||||||||||||||||||||||||
| Critical | Display of archive ingestion/observation issues |
| related to SSA-7190 | ||||||||||||||||||||||||
| Critical | Identification of ingestion problems and automatically fix | Example ticket of a silent failure:
| |||||||||||||||||||||||||
| √ | create missing db entries for Flag.xml |
| Done | ||||||||||||||||||||||||
| √ | fix ASDM.xml files missing FlagTable entities |
| Done |
DA Tools for Data EditingAnchor DATools DATools
| Priority/Done? | Notes | JiRA ticket /Confluence page | ||||||||||||||
| √ | Tool for setting Proprietary Period of science observations on the VLA and the VLBA | https://open-confluence.nrao.edu/pages/viewpage.action?spaceKey=SPR&title=Tool+for+setting+Proprietary+Period+of+science+observations+on+the+VLA+and+the+VLBA | ||||||||||||||
| Critical | DA annotation tools/Archive data annotation tools | legacy VLA metadata error: EBs that claim to have run for 2+ days
| SSA-7129 | requires re-ingestion |
Data Integrity
| Priority/Done? | Notes | JiRA ticket /Confluence page | Gap Analysis Between Legacy and New Archive
| SSA-6222 | can be done once RL finishes the AIPSFITS files | Automated annotations for missing scans/BDF, Automated Error Recognition of Ingested Data
|
Missing DataAnchor Missing Missing
| Priority/Done? | Notes | JiRA ticket /Confluence page | |||||||||||||||||||||||||||||
| Critical |
| Tix to be created
| SSA
| 7190https://open-confluence.nrao.edu/display/SRDP/Archive+Data+Annotation+ToolsDisplay of archive ingestion/observation issues | |||||||||||||||||||||||||||
| Jira | |||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| server | DMS JIRA | ||||||||||||||||||||||||||||||
| columnIds | issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution | columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
May require discussion about how new archive will display images and the uvfits datasets. | ||||||||||||||||||||||||||||
| High | ALMA Standard images into AAT |
| 7189related to SSA-7190 | √ | create missing db entries for Flag.xml
| ||||||||||||||||||||||||||
| High | Large project support - ALFALFA + Other collection ingest | Done | √ | fix ASDM.xml files missing FlagTable entities |
| 6273
|
| 36eee8a98f01||||||||||||||||||||||||
| key | SSA-6274 | Done |
|
Other
DA Tools for Data EditingAnchor Other Other
| Priority/Done? | Notes | JiRA ticket /Confluence page | Tool for setting Proprietary Period of science observations on the VLA and the VLBADetails | ||||
|---|---|---|---|---|---|---|---|
| Other tickets (so they are not forgotten) | https://open- | confluencepages | viewpage.action | spaceKey=SPR&title=Tool+for+setting+Proprietary+Period+of+science+observations+on+the+VLA+and+the+VLBA | DA annotation tools/Archive data annotation tools | https://open-confluence.nrao.edu/display/SRDP/Archive+Data+Annotation+Tools |
Missing Data
...
- VLA/VLBA Image archive contents should be migrated into AAT (https://archive.nrao.edu/archive/archiveimage.html)
| Tickets remaining are listed throughout their respective categories. | |||||||||||||||
| Critical | Proposal Contact Author not added to data.nrao.edu database |
| Bug when a project uses different PI and Contact Author | ||||||||||||
| Medium | Preserve login between AAT and my.nrao.edu |
| |||||||||||||
| √ (4.2.0) | Remove pop-up page when accessing data.nrao.edu |
|
Other
| Priority/Done? | Notes | JiRA ticket /Confluence page | Details | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Other tickets (so they are not forgotten) | Proposal Contact Author not added to data.nrao.edu database |
| Bug when a project uses different PI and Contact Author |
|
ALMAAnchor ALMA ALMA
| Priority/Done? | Notes | JiRA ticket /Confluence page | Details | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Critical | ALMA MOUS/calibration ingest unreliable; ALMA butler implementation into productionPreserve login between AAT and my.nrao.edu |
|
Product VersioningAnchor Versioning Versioning
| Priority/Done? | Notes | JiRA ticket /Confluence page | Details | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Critical | Need to allow multiple versions of products (images/calibrations) in the archive | Product Versioning and Replacement Design
|