You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

NOTE: This an updated version of the original page for the upcoming WS 2.5 version. The new features mentioned here are currently on WS-Dev and WS-Test but not production.

This guide is intended for DAs and AODs using the workspaces capabilities for VLA calibration and imaging, initially for the pilot. For the pilot, we will be selecting EBs from specified projects (TBD) to be run through (and warn the PIs!). Auto-calibration is not yet running (question).

Step-by-step guide for pilot (WS 2.0)

  1. Go to https://ws.nrao.edu/workspaces/active-requests (do NOT log in, this is not required and will break your session) and enter the EB to be processed in the "SDM ID" box at the top, hit the "Calibrate" button. Make sure Auto-calibration button at the top right is set to run (this has nothing yet to do with the automatic start of jobs, which is controlled by the CAPO profile*).
  2. Select the job by clicking the request on the left-hand column, this should open a request page, with the option to submit the job at the lower right. Go ahead and submit the job.
  3. The job should appear in the job list with a status of "executing".
  4. You should be able to see the working directory in lustre for the request in the Version Parameters section of the request page. You can click on the text to copy it.
  5. Note the details in the pilot google spreadsheet https://docs.google.com/spreadsheets/d/10Jn6TyEsthP0KQgbuhpIPn4fqWlLvhEO7tAkFMPny0o/edit?usp=sharing
  6. At any point prior to QA you should set the SRDP status to T/F with the checkbox and assign a DA (yourself) and an AOD to the job. 
  7. When complete, the status will change to "Awaiting QA". Click the link to the request page in the "Request" column, or the link the email to Workapsaces-Analysts, which will take you to the request page.
  8. Follow the link to the weblog and perform QA (if you need to you can go to the working directory on spool using the directory name from step 4). The QA Notes can be viewed and edited by clicking the Edit QA Notes button on the request page. You will need them for the email to the user later.
  9. If you need to add flags, click "Create new version" on the bottom right of the page, and upload a flagtemplate.txt file. Select the latest version on the drop-down menu under "Versions", then click "Submit new version". The old version remains and you can toggle between them using the dropdown menu. Note that once uploaded, the flag file (or any other uploaded file) cannot be overwritten in subsequent versions, to add flags to later versions you will need to go to the relevant directory on spool soon after the job is started and copy or edit the flagtemplate file directly.
  10. When you are satisfied, inform the AOD (always Mark Lacy for the pilot) that a job is ready for QA. The AOD will double-check the calibration and click "QAPass", or send it back to the DA for a rerun. QAPass will pass the current selected version and fail all the others.
  11. The execution state should change to "Ingesting", then "Complete", at which point you should get an email saying the request was ingested.  A "Create Image request" button appear at the bottom right of the request page. 
  12. Create the image request, this will create the request and (should) take you to a new image request page (note that with some browsers/brower settings you might need to go back to the active-requests page, select tstd_cms_imaging from the top-left menu and click on the request to get to the request page).
  13. Submit the image request  (bottom right button on the image request page).
  14. Weblog review can be carried out in the same way as for callibration, once the state has changed to "Waiting for QA".
  15. You can upload a new PPR for imaging, but currently no flagging can be applied at the imaging stage, if you need more flagging you will have to go back to step 1 and recalibrate.
  16. Once satisfied, Let the AOD (Mark Lacy) know that the image is ready for review via Mattermost; @mlacy). The AOD will click "QA Pass" and an email will be sent to Workspaces-Analysts to say that the data were ingested.
  17. The archive takes a while (~24hr?) to reindex and have the data appear, so wait one day, then send an email to the PI and co-Is with text as in the bottom info box below:


*need to set edu.nrao.workspaces.StandardCalibrationSettings.runState = PAUSE, not STOP in /home/casa/capo/dsoc-prod.properties))


Hello,
One of your Scheduling Blocks,

<EB ID>, observed on <date>, <band>?[e.g. BM506.sb39359007.eb41147858.59600.36297047454, observed on 01/21/2022, Q-band

has been processed through the VLA CASA Imaging Pipeline using CASA 6.2.1, which is designed to handle Stokes I continuum data. Continuum images from this execution block are now available from the archive.

? These results have been checked by NRAO staff and notes about the quality assurance are below (these notes are also in the weblog).

<insert qa_notes.html>

Accessing Pipeline Products:

The images may be downloaded from the NRAO archive access tool https://data.nrao.edu. To access the pipeline products, please follow these steps:

? Click the "Log in" link at the top right of this page and select the NRAO login, which should take you to a page where you will enter your my.nrao.edu login information.
? Once logged in, you should be able to access your projects, both public and proprietary by selecting <yourusername>'s data: navigate to the desired project and click the "+" symbol at the left to expand the list of available SBs.
? The available images can be listed and selected for download by clicking the “Images” tab underneath the project abstract.

A measurement set with or without. the calibration applied can also be obtained from the archive by selecting them for download under the “Observations” tab should you wish to reimage and/or recalibrate the data yourself

____________
For more information about the pipeline, including instructions for rerunning the pipeline, applying pipeline calibration to raw data, or modification to suit your particular science goals, or for access to the scripted pipeline please visit our pipeline web page: https://science.nrao.edu/facilities/vla/data-processing/pipeline
For more information about the SRDP project, please see https://science.nrao.edu/srdp
Please let us know if you have any questions or concerns through the NRAO Helpdesk (https://help.nrao.edu/) , using the VLA Pipeline department for questions about the pipeline processing, the VLA/VLBA Archive and Data Retrieval department for questions about data retrieval, and the VLA Data Products department for questions about quality assurance and the use of of Science-Ready (SRDP) products.

Command line tools 

There are several command line utilities that SSA have written to help with e.g. diagnosing issues. They are described on https://ssa.gitlab-pages.nrao.edu/workspaces/source/tools.html

  • No labels