New Features

2.8.5.1

  • Fixes for restore user email propagation to workspaces system
  • Fixes for sending restore failure notifications to the analyst lists
  • Fixes to the DA spelunker utility

2.8.5

  • Full implementation of Calibrated Measurement Set Restores for EVLA and ALMA data
  • Delivery improvements for CMS restores
  • Improvements to the CASA Matrix to enable dynamic building of PPRs based on specific CASA version recipes
  • Improvements to file ingestion staging to allow for multiple attempts when necessary
  • bug fixes for the operations metrics reports
  • Improvements to directory tracking and clean up
  • Improvements to request cancellation

2.8.4.1

  • Catch and respond to condor_submit and condor_submit_dag error codes when cluster submission fails

2.8.4

  • CARTA4.1 support
  • Implementation of the CASA Version Matrix for CASA version selection based on initial product and processing type
  • Image download delivery improvements, including separation of image products and weblog delivery
  • Calibration download delivery improvements
  • Infrastructure preparation for improved CMS restores

2.8.3.1

  • Fixes to workflow log monitoring
  • Fixes for automatic lustre cleanup of system aborted processing directories
  • Fixes for workflow reacquire on initial deployment

2.8.3

  • Workflow support to full image curation
  • Upgraded WS for HTCondor10
  • Misc. bug fixes
  • Optimization of lustre storage usage.
    • Data will be removed from lustre within 14 days of processing completion for calibration and imaging (SRDP status dependent)
    • QA Fail, Cancellation, or Closure of Workspaces requests now immediately removes the processing directory from lustre
    • Capability requests now share a single copy of all rawdata between multiple versions

2.8.2.2

  • Workflow support for partial curation
  • Misc. bug fixes

2.8.2.1

  • Misc. bug fixes

2.8.2

  • Only science bands will be shown on the Active Requests page for newly ingested data processing

  • The background colors of the execution status shown on the Active Requests page now reflects the status

  • Preliminary support of RADIAL

  • NGAS Download Status is now visible and filterable on the Active Requests page

  • QA Pass - Custom action now available. Customize Emails to users when necessary

  • The Email dialog has been removed from QA Fail

  • Setting the SRDP flag on the Active Requests page now reflects changes to all versions

  • VLBI Ingestion Support:
    - VLBA/GMVA file ingestion has been standardized to use the manifest driven system in use for other instruments
    - VLBA/GMVA file ingestion will be removed from the Legacy Archive and current AAT/PPI services
    - VLBA/GMVA ingestion is now controlled by AAT/WS via StackStorm
    - this includes:
    - directory monitoring for the difx area
    - directory renaming to reflect ingestion status
    - HTCondor based ingestion workflow implementation for VLBA, GMVA and Mark4 data types.

2.8.1.3

  • Permanent fixes for UI update bugs

2.8.1.2

  • Mitigation of UI update bug causing increased server load

2.8.1.1

  • Bug fix for user notifications
  • improved robustness of notification service against invalid email addresses

2.8.1

  • Workflow Progress Tracking
  • Ability to write tailored emails to users
  • Miscellaneous UI improvement
  • Ancillary image products are now delivered for image downloads
  • DA tool improvements

2.8.0

  • All basic downloads from the AAT now go through WS
  • Delivery improvements
  • VLASS cube image QA and caching
  • CARTA experience improvements

Known Issues

2.8.1.2

There is likely an N+1 query issue in the backend endpoint that supports the UI active request page updates. Investigations are ongoing

Release 2.8.5.1 - 2024.11.07

Story

  • [WS-2841] - As an operations manager, I would like the e-mail addresses for restores to be propagated into workspaces

Bug

  • [WS-2840] - Limit spelunker query to workflow_requests to relevant data
  • [WS-2894] - Restore failure notifications are not being sent to the proper mailing lists

Release 2.8.5 - 2024.10.30

Story

  • [WS-380] - As an operations manager I want to be notified if a VLA restore job fails
  • [WS-381] - As an operations manager, I want to be notified if an ALMA restore job fails
  • [WS-382] - As a user, I want to be notified when my ALMA restore job is complete
  • [WS-383] - As a user, I want to be notified when my VLA restore job is complete
  • [WS-1275] - As an operations manager, I would like to be sure that the PPR recipes are updated correctly
  • [WS-1936] - Please remove duplicate rows from the ws_metric operations report
  • [WS-2097] - As a user, I want the delivery improvements for EVLA CMS downloads implemented
  • [WS-2149] - As an operations manager, I would like calibration restores to have only one copy of the calibrated ms.
  • [WS-2241] - SSA OPS Request: Name staging directories more uniquely
  • [WS-2533] - As a user, I would like to run a calibration ALMA restore through WS.
  • [WS-2538] - As a user, I want to be able to download a restored ALMA CMS
  • [WS-2581] - As a project scientist, I want the CASA Matrix to disallow versions/pipelines older than the original processing version
  • [WS-2639] - As a project scientist, I require that ALMA restore products must staged for download from NAASC filesystems

Bug

  • [WS-2710] - Restore workflows should fail if CASA doesn't produce at least one MS
  • [WS-2711] - Restore workflows should use the PPR over piperestore script if the calibration was produced with a different major CASA version
  • [WS-2791] - As a user, I would like to run ALMA restores with the runpipeline.py script
  • [WS-2792] - As a user, I would like the piperestore script to work across major CASA versions
  • [WS-2834] - As a user, I would like to use the piperestore script for calibrations generated with CASA < 5
  • [WS-2839] - Acknowledgements for USNO included with some VLA projects

Epic

  • [WS-2184] - Conversion and improvements for EVLA Restores
  • [WS-2185] - Conversion and improvements for ALMA Restores
  • [WS-2193] - Implementation of CASA Version Matrix

Release 2.8.4.1 - 2024.08.08

Bug

  • [WS-2580] - Catch the condor submit exit code and fail workflow if cluster submission fails

Release 2.8.4 - 2024.07.08

Story

  • [WS-274] - As a user I want to restore an EVLA CMS through the WS UI (but not have it delivered to me)
  • [WS-1405] - As a product owner, I would like the AAT to automatically select the appropriate version of CASA to restore my data
  • [WS-1729] - As a user I would like weblogs with my image products
  • [WS-1851] - As a user, I’d like for the archive to track the completion date of imaging and calibration pipelines so the delivery directory structure for downloads is more functional.
  • [WS-1967] - As a project scientist, I would like CARTA in the AAT/WS upgraded to version 4.1
  • [WS-2209] - As a archive maintainer, I want to know which versions of CASA will run on the current processing cluster OS
  • [WS-2351] - As a archive user, I want to know which CASA versions are valid for my request
  • [WS-2352] - As an archive user, I want to be able to process data with my chosen CASA version
  • [WS-2353] - As an operations manager, I want to track the PPR procedure name on the relevant capability definition
  • [WS-2355] - As an operations manager, I want to run calibration curation automatically
  • [WS-2425] - As a developer, I want casa_envoy to support calibrated-measurement-set restores


Release 2.8.3.1 - 2024.06.18

Bug

  • [WS-2434] - Fix HTC restart workflow state setting
  • [WS-2435] - Fix HTC restart limited workflow abort cleanup
  • [WS-2436] - Fix reacquire system skipping reacquisitions
  • [WS-2488] - prevent wf_monitor from closing connections early
  • [WS-2492] - some DAGs fail to track condor.log despite running successfully

Release 2.8.3 - 2024.05.02

Task

  • [WS-2213] - Update Workflow container to use HTCondor10
  • [WS-2310] - Investigate StackStorm networking issues

Story

  • [WS-1353] - As an operations manager, I would like data deleted from spool in a more timely fashion
  • [WS-1409] - As VLASS Operations Manager, whenever a tile is calibrated for QuickLook and Accepted and Archived, it should automatically be split into per-image measurement sets
  • [WS-1429] - As operations manager, I need the SECC workflow to retrieve the SECI component list
  • [WS-1574] - As VLASS Operations Manager, I would like a setting to turn on/off auto deletion of PIMS after SECI/SECC imaging
  • [WS-1770] - As a product owner, I need images to be reingested such that the tt0 image is the primary data product
  • [WS-1879] - As a VLASS DA, I would like the email subject line for pimscache to inform of success/failure
  • [WS-1975] - As a DA working on VLASS, I would like to be able to create a flagging template for imaging pipeline jobs in a similar way to calibration jobs
  • [WS-2147] - As an operations manager, I would like failed versions removed from disk upon QA pass
  • [WS-2148] - As an operations manager, I would like an email if the cleanup script is unable to delete a directory on spool
  • [WS-2150] - As an operations manager, I would like the data from failed requests removed from spool.
  • [WS-2227] - As an operations manager, I would like one copy of the raw data to be used by all versions of a capability request

Bug

  • [WS-2116] - pimscache: integer-proof the <RelativePath> data with quotes
  • [WS-2165] - wrong syntax in vlass_seci jobs
  • [WS-2210] - Close Request's email modal needs to send the correct destination_email
  • [WS-2214] - internal_notes' Version 1's timestamp is frozen
  • [WS-2313] - Targeted curation not working on production

Epic

  • [WS-2181] - Optimize use of Lustre resources

Release 2.8.2.2 - 2023.12.21

Bug

  • [WS-1893] - As a product owner, I would like existing data to have their metadata updated to support refactored scanlists
  • [WS-2161] - Fix execution status filtering for SRDP products
  • [WS-2167] - Fix PPR format and encoding after updating for HTCondor

Release 2.8.2.1 - 2023.12.08

Bug

  • [WS-2156] - As a DA, I would like to see the QA notes file persist after QA pass
  • [WS-2164] - VLBA ingestion is missing difx log file

Release 2.8.2 - 2023.11.30

Story

  • [WS-1491] - As an operations manager, I only want to see the science bands in an observation listed
  • [WS-1657] - As a developer, I want the pexes to be built in the pipeline and installable via pip using the gitlab repo
  • [WS-1658] - As a developer, I want to get rid of the base and cache Docker images
  • [WS-1659] - As a developer, I want the CI pipeline to only rebuild things that changed
  • [WS-1715] - As an operations manager, I would like the background color of execution status to reflect the state
  • [WS-1812] - As a DA I would like an indication on the active requests page when a request has completed downloading data from NGAS.
  • [WS-1813] - As a DA or AOD I would like the dialog boxes for the QA notes and internal notes not to close when "save" is pressed, but have a separate "save & close" button in addition to "save".
  • [WS-1814] - As a DA, I do not want the failure email dialog to come up when hitting QAFail (to fail the version), only for "fail entire request" and "close request".
  • [WS-1815] - As a DA, I would like a "QA Pass - custom" button that brings up an email form for cases when an observation passes QA, but I have comments to add outside of the DA notes and/or want to cc schedsoc.
  • [WS-1816] - As a DA, I would like to revert to the previous version when the "Fail QA" button is pressed, and be able to set or unset its SRDP status.
  • [WS-1944] - As a user I would like to be able to filter the active requests page based on the NGAS download status

Bug

  • [WS-1912] - Stray Whitespace Causing Errors
  • [WS-1977] - UI User notifications lack access to PI/CoI email addresses

Research Request

  • [WS-1618] - As a product owner, I want to know what the right solution is to handling the NGAS thread limits across all the SSA systems
  • [WS-1798] - Prototype submission of VLASS jobs to RADIAL nodes


Release 2.8.1.3 - 2023.07.26

Bug

  • [WS-1895] - Identify and correct active request N+1 query issue
  • [WS-1896] - Investigate and correct active request page polling behavior

Task

  • [WS-1897] - Return active request update timing to 3 second

Release 2.8.1.2 - 2023.07.12

Bug

  • [WS-1886] - WS UI updating is causing high load on production and impacting other services

Release 2.8.1.1 - 2023.06.22

Task

  • [WS-1866] - Revert and version lock dsnparse library for contact wresting
  • [WS-1867] - Fix notification service to be robust against invalid email addresses

Release 2.8.1 - 2023.06.07

Story

  • [WS-1168] - As a DA, I would like the option to write a tailored email to a user when I fail a request, and cc schedsoc
  • [WS-1372] - As an operations manager, I want to retain workflow progress visibility for converted AAT and VLASS workflows
  • [WS-1463] - As production manger, I would like the QA workflow terminology to reflect the process rather than job titles
  • [WS-1490] - As a data analyst or operations manager, I would like the request number listed in the ws_metrics report
  • [WS-1492] - As operations manager, when creating a new version of a calibration, the interface should automatically show the latest version
  • [WS-1493] - As an operations manager, I would like to be able to sort Active Requests by Execution status
  • [WS-1514] - As an operations manager I would like the option to display more active requests
  • [WS-1515] - As an operations manager, I would like the interface to show a different color for the DA/AOD assignment boxes
  • [WS-1568] - As an operations manager, I would like to be able to filter displayed requests by Execution Status
  • [WS-1576] - As VLASS Operations Manager, I would like a e-mail notification when PIMSCache has finished for a tile
  • [WS-1577] - As VLASS Operations Manager, I would like the Quicklook imaging to start after PIMSCache has completed for a Tile
  • [WS-1578] - As a workspaces users (DA or ops manager), I do not want my current page to be retained when switching queues
  • [WS-1594] - As a WS user, I would like to be able to open a WS request in a new tab from the active requests page
  • [WS-1614] - As a user, I want all the ancillary images to be downloaded when downloading the primary image product
  • [WS-1615] - As a DA, I would like to remove a failed request from the active requests list
  • [WS-1620] - As a VLASS operations manager, I want improved reporting for PIMSCache, including when completion emails are sent
  • [WS-1711] - As an operations manager, I need a way to associate the request ID with its working directory

Release 2.8.0 - 2023.03.30

Research Request

  • [WS-1500] - As a product owner, I want to ensure that the developers have all the information they need to implement GBT support

Bug

  • [WS-1513] - Sorting by request appears to be broken in Workspaces interface
  • [WS-1538] - Active requests page jumps on refresh
  • [WS-1652] - increase HTCondor running job limits

Story

  • [WS-50] - As an anonymous scientist I want to download a VLA data set in the native format (or tar file) to my home computer.
  • [WS-52] - As an anonymous scientist I want to download a VLBA data set in the native format (or tar file) to my home computer
  • [WS-53] - As an anonymous scientist I want to download a GBT data set in the native format (or tar file) to my home computer
  • [WS-110] - As a user, I want to able to download a script that will use wget/curl to retrieve files from my request
  • [WS-121] - As an anonymous scientist I want to download an ALMA data set in the native format (or tar file) to my home computer
  • [WS-122] - As an authenticated user I want to download a proprietary ALMA data set in the native format (or tar file) to my home computer
  • [WS-1110] - As a DA, I would like to be able to QA reject bad planes from a cube but pass the good ones
  • [WS-1496] - As an anonymous user, I want to download images in their native format (or tar file) to my home computer
  • [WS-1497] - As an authenticated user, I want to download my proprietary images in native format (or tarfiles) to my home computer
  • [WS-1498] - As an authenticated user, I want to download proprietary or public images in native format (or tarfile) to a directory on lustre
  • [WS-1512] - As a user, I do not want to be prompted to send user data for CARTA when the session loads
  • No labels