A restore of ALMA data only makes sense at the MOUS level. Individual EBs are not calibrated, so the calibration products are all linked to the MOUS. There is currently no automated processing performed at the GOUS or SGOUS, and the infrastructure for sub-MOUS restores is incomplete.
ALMA products are provided with a *.scriptForPI.py which attempts to cover all the possibilities of how one might go about performing a restore, and relies heavily on assumptions about directory layouts and the possible different naming schemes for files within the directory structure. The aim of this document is to reduce an MOUS level restore to it's basics: a standard CASA products/rawdata/working layout with the critical data placed appropriately and a PPR created to guide the pipeline.
NOTE: Still a work in progress with temporary tools until I refine the methodology.
Can We Restore?
I'm assuming I start with an MOUS uid (for instance: uid://A001/X1284/X265f) and access to the ALMA metadata database. We first need to know if it's is calibrated in the first place. If: SELECT COUNT(*) FROM ASA_PRODUCT_FILES WHERE ASA_MOUS_UID = 'uid://A001/X1284/X265f' AND FILE_CLASS = 'calibration'; is greater than zero, we're in luck.
Setting up:
On a machine which can see CV's Lustre (Hopper and aatweb-dev are ones we already use for workflow basics), go to an working area (I'm using /lustre/naasc/web/almapipe/pipeline/vatest/testing) and create yourself a new directory (and the standard subdirectories for CASA processing).
Next we need to constituent ASDMs (or EBs): SELECT DISTINCT ASDM_UID FROM ASA_SCIENCE WHERE MEMBER_OUSS_ID='uid://A001/X1284/X265f'; Which gives us four EBs: uid://A002/Xc7111c/X86a7, uid://A002/Xc7111c/X409e, uid://A002/Xc72427/X2dfe, uid://A002/Xc7111c/X3979 . Using those ASDM_UIDs you can download the raw data using asdmExportLight -d rawdata {uid_here}.
To download the calibration products, I have modifed a script used at JAO in Santiago to query and run wget to obtain the files directly from the local NGAS system (naasc_listfiles.py). In the products subdirectory, run that script with the MOUS uid from above (~almapipe/workflows/nmtest/bin/python3.6 ~almapipe/pipeline/vatest/testing/naasc_listfiles.py uid://A001/X1284/X265f).
Two Roads Diverge:
Easy Way:
If we obtain the scripts (very similar to obtaining the calibration information above, just change the FILE_CLASS to 'scripts') which accompany the ALMA products, we could simply run their .hifa_cal.casa_piperestorescript.py ( /home/casa/packages/RHEL6/release/casa-release-5.1.1-5/bin/casa --pipeline -c member.uid___A001_X 1284_X265f.hifa_calimage.casa_piperestorescript.py). This fires right up and gets to work, but failed on the 3rd EB (of 4!) due to a typo in my download of the data.
For Greater Flexibility:
The restore script is very useful, but for the AAT-PPI, we're trying to stick to a single method of interacting with the casa pipeline: Creating a PPR. We'll need more information for the PPR:
- Project Code
- SELECT DISTINCT PROJECT_CODE FROM ASA_PROJECT JOIN ASA_SCIENCE ON ASA_PROJECT.PROJECT_UID = ASA_SCIENCE.PROJECT_UID WHERE MEMBER_OUSS_ID='';
- ObsProject uid
- ObsUnitSet partId TODO: can't find someone who knows what this is
- ProjectStatusRef
- OUSStatusRef
- EB → Session Mappings: TODO: can't get a good answer on this yet.
- SchedBlockRef
- ASA_SCIENCE + MOUS uid
- If there's more than one, then we have an issue, but it should be rare
- SBStatusRef
- SCHED_BLOCK_STATUS + SB uid
- SCHED_BLOCK_STATUS + SB uid
NOTE: It may be possible to skip some of that information (2-5). We have been ignoring the ProjectStructure section of the PPR for years with no difficulties. However, the session mapping is a blocker (I suspect it effects how the calibrations are applied).
I've created a first-pass PPR for our test data set, but it has some flaw I'm not seeing yet (the pipeline rejects the ProcessingRequests). PPR_jls.xml