Page History

NOTE: This an updated version of the original page for the upcoming WS 2.5 version. The new features mentioned here are currently on WS-Dev and WS-Test but not production.

This guide is intended for Stage 1 Reviewers (DAs and AODs ) and Stage 2 Reviewers (previously "AoD") using the workspaces capabilities for VLA calibration and imaging, initially for the pilot. For the pilot, we will be selecting EBs from specified projects (TBD) to be run through (and warn the PIs!). Auto-calibration is not yet running .

Step-by-step guide for

...

WS 2.8 (see also slides from Mark's presentation linked here, for WS 2.

...

7 but not much has changed)

1. Go to https://ws.nrao.edu/workspaces/active-requests (do NOT log in, this is not required and will break your session) and enter the EB to be processed in the "SDM ID" box at the top, hit the "Calibrate" button. Make sure the Auto-calibration button at the top right is set to run (this has nothing yet to do with the automatic start of jobs, which is controlled by the CAPO profile*).
2. The job should appear in the job list with a status of "executing". Select the job by clicking the request on the left-hand column, this should open a request page, with the option to submit the job at the lower right. Go ahead and submit the job.
3. The job should appear in the job list with a status of "executing".
4. You should be able to see the working directory in lustre for the request in the Version Parameters section of the request page. You can click on the text to copy it.Note the details in the pilot google spreadsheet https://docs.google.com/spreadsheets/d/10Jn6TyEsthP0KQgbuhpIPn4fqWlLvhEO7tAkFMPny0o/edit?usp=sharing
5. At any point prior to QA you should set the SRDP status to T/F with the checkbox and assign a DA Stage 1 Reviewer (yourself) and an AOD a Stage 2 Reviewer to the job. 
6. When complete, the status will change to "Awaiting QA". Click the link to the request page in the "Request" column, or the link in the email to WorkapsacesWorkspaces-Analysts, which will take you to the request page.
7. Follow the link to the weblog and perform QA (if you need to you can go to the working directory on spool using the directory name from step 43). The QA Notes can be viewed and edited by clicking the Edit QA Notes button on the request page. You will need them for the email to the user later
8. On the bottom right is a button to edit the Internal Notes; these notes persist between versions and are not seen by the user.
9. If you need to add flags, click "Create new version" on the bottom right of the page. A flagtemplate is pre-loaded, so you just need to edit it.
10. If you need to edit the PPR, this is also accessible to edit at this stage.
11. , and upload a flagtemplate.txt file. Select the latest version on the drop-down menu under "Versions", then click "Submit new version". The old version remains and you can toggle between them using the dropdown menu. Note that once uploaded, the flag file (or any other uploaded file) cannot be overwritten in subsequent versions, to add flags to later versions you will need to go to the relevant directory on spool soon after the job is started and copy or edit the flagtemplate file directly.drop-down menu. 
12. When you are satisfied, inform the AOD (always Mark Lacy for the pilot) that a job is ready for QA. The AOD will double-check the calibration and click "QAPass", or send it back to the DA for a rerun. QAPass edit the QA notes and hit "Send to Stage 2". Make sure the Cal Status in Drew's spreadsheet has been set to "QA Ready".
13. Stage 2 Reviewer role: check the calibration, the QA notes and that the SRDP status is set correctly. If not satisfied, you can Return to Stage 1 after adding comments in the Internal Notes. If satisfied, hit the QA Pass button. QA Pass will pass the current selected version and fail all the others., sending email to workspaces-analysts and the PI (cc'd to workspaces-analysts). The execution state should change to "Ingesting", then "Complete", at which point you should get an email saying the request was ingested.  A . An email will be sent to Workspaces-Analysts to say that the calibration data were ingested, and to the user (cc'd to Workspaces-Analysts) with the QA notes and instructions for retrieval. In addition, Drew's spreadsheet should be updated to set the Cal Status to "Archived"; if SRDP set the QA2 SRDP column entry to "Pass".
14. Stage 2 Reviewer role: if the data are SRDP wait for the calibration ingest to complete, then hit the "Create Image request" button appear at the bottom right of the request page. 
15. Create the image request, this will create the request and (should) take you to a new image request page (note that with some browsers/brower settings you might need to go back to the active-requests page, select tstd_cms_imaging from the top-left menu and click on the request to get to the request page).
16. . Once the imaging request has been created (its imaging request will open in a new browser tab), enter the workspaces ID number in the "Imaging WS ID" column. In Workspaces, the Stage 1 and Stage 2 reviewers will be set to the same people as the calibration request, and status to SRDP, but you may want to check that.
17. Stage 2 Reviewer role: Submit the image request Submit the image request   (bottom right button on the image request page).
18. Once the imaging is complete, the state will change to "Awaiting QA". The Stage 1 reviewer should watch the image queue for their datasets to check for new job completions.
19. Weblog review can be carried out in the same way as for callibration, once the state has changed to "Waiting for QA".
20. You can upload a new PPR for imaging, but currently no flagging can be applied at the imaging stage, if you need more flagging you will have to go back to step 1 and recalibrate.
21. calibration. Note the "Launch CARTA session" capability that can be used to inspect the image data. Flagging can be added by uploading a <FSID>.flagtargetstemplate.txt.
22. Once satisfied, edit the QA notes and hit "Send to Stage 2".
23. Stage 2 Reviewer role:  If satisfied, check the QA notes and Once satisfied, Let the AOD (Mark Lacy) know that the image is ready for review via Mattermost; @mlacy). The AOD will click "QA Pass" and an email will be sent to Workspaces-Analysts to say that the data were ingested.The archive takes a while (~24hr?) to reindex and have the data appear, so wait one day, then send an email to the PI and co-Is with text as in the bottom info box below:, and to the user with the QA notes and instructions for retrieval.

Command line tools 

There are several command line utilities that SSA have written to help with e.g. diagnosing issues. They are described on https://ssa.gitlab-pages.nrao.edu/workspaces/source/tools.html

They are located in /lustre/aoc/cluster/pipeline/dsoc-prod/workspaces/sbin and need to be run as vlapipe on mcilroy in the dsoc-prod environment (activate_profile dsoc-prod) or on hamilton on dsoc-test (activate_profile dsoc-test).

Checking on jobs:

./wf_inspector <req number> -v <version number> will log into the running job.

Metrics: 

Write out the operations report as a csv:

./ws_metrics -o <date1> <date2> --report  

Number of executions of a capability between date1 and date2:

./ws_metrics -b <capability> <date1> <date2>

Dates in format YYYY-MM-DD. Note that the report is not actually filtered on the dates yet. 

Changing Stage 1 and 2 reviewers in the menus:

./mod_analyst e.g. to add Stage 2 reviewer:

./mod_analyst -name "Nathan Bockisch" -group "Stage 2" -email "nbockisc@nrao.edu"

or to remove a Stage 1 reviewer:

./mod_analyst -name "Nathan Bockisch" -rm

Linking AAT requests to workspaces directories

./spelunker -h
Usage of ./spelunker:
  -id int
    	[Required] The Workspaces or Archive ID for the workflow (default -1)
  -profile string
    	[Optional] The CAPO profile to use, defaults to CAPO_PROFILE env variable (default "dsoc-dev")
  -v	Display the version of this utility

Digging deeper

Instructions for diagnosing issues with jobs - please check with SSA before doing more than just checking on jobs.

Instructions for running arbitrary versions of CASA (from Charlotte):

Email templates:

The operations manager should check the email templates that are sent to the users after every deployment and edit if needed. For example, the filesizes are included as a keyword, but not actually looked up, so those should be removed.

Pipeline recipes

Note that, currently, the pipeline recipes used by workspaces are not automatically updated when the pipeline is updated, they need to be edited separately (easy to forget this step)!

Related articles

...