A restore of ALMA data only makes sense at the MOUS level. Individual EBs are not calibrated, so the calibration products are all linked to the MOUS. There is currently no automated processing performed at the GOUS or SGOUS, and the infrastructure for sub-MOUS restores is incomplete.

ALMA products are provided with a *.scriptForPI.py which attempts to cover all the possibilities of how one might go about performing a restore, and relies heavily on assumptions about directory layouts and the possible different naming schemes for files within the directory structure. The aim of this document is to reduce an MOUS level restore to it's basics: a standard CASA products/rawdata/working layout with the critical data placed appropriately and a PPR created to guide the pipeline.

NOTE: Still a work in progress with temporary tools until I refine the methodology.

Can We Restore?

I'm assuming I start with an MOUS uid (for instance: uid://A001/X1284/X265f) and access to the ALMA metadata database. We first need to know if it's is calibrated in the first place. If: SELECT COUNT(*) FROM ASA_PRODUCT_FILES WHERE ASA_MOUS_UID = 'uid://A001/X1284/X265f' AND FILE_CLASS = 'calibration'; is greater than zero, that's a decent start. Validation of the setup after we perform it will still be required.

Setting up:

On a machine which can see CV's Lustre (Hopper and aatweb-dev are ones we already use for workflow basics), go to an working area (I'm using /lustre/naasc/web/almapipe/pipeline/vatest/testing) and create yourself a new directory (and the standard subdirectories for CASA processing).

Next we need to constituent ASDMs (or EBs): SELECT DISTINCT ASDM_UID FROM ASA_SCIENCE WHERE MEMBER_OUSS_ID='uid://A001/X1284/X265f'; Which gives us four EBs: uid://A002/Xc7111c/X86a7, uid://A002/Xc7111c/X409e, uid://A002/Xc72427/X2dfe, uid://A002/Xc7111c/X3979 . Using those ASDM_UIDs you can download the raw data using asdmExportLight -d rawdata {uid_here}.

To download the calibration products, I have modifed a script used at JAO in Santiago to query and run wget to obtain the files directly from the local NGAS system (naasc_listfiles.py). The script needs access to cx_Oracle, so activate the vatest profile before you run it. In the products subdirectory, run that script with the MOUS uid from above (~almapipe/workflows/nmtest/bin/python3.6 ~almapipe/pipeline/vatest/testing/resources/naasc_listfiles.py --get uid://A001/X1284/X265f). This will also obtain all the script files (including the initial PPR and pipeline manifest (the latter is required for a proper restore).

NOTE: Copy the pipeline manifest over to the rawdata directory

Can we restore usefully?

There is quite a bit of manual work in calibrating some ALMA data. In addition to a standard pipeline restore, they also go through 3 other channels (recalibration via casa_pipescript.py, reapplying a custom calibration via scriptForCalibration.py and a separate single-dish calibration scheme via NEED_NAME_HERE). Need to examine scriptForPI.py more carefully in order to determine how to separate ASDMs which can be restored vs those that need special treatment.

There is also a chance that the piperestorescript.py has been hand-edited in Chile due to operational issue placing bad data into the ASDM. However, in all but a handful of cases, those fixes are incorporated into the calibration tables, and therefore can be safely ignored in a restore PPR.

Two Roads Diverge:

Easy Way:

If we obtain the scripts (very similar to obtaining the calibration information above, just change the FILE_CLASS to 'scripts') which accompany the ALMA products, we could simply run their .hifa_cal.casa_piperestorescript.py ( /home/casa/packages/RHEL6/release/casa-release-5.1.1-5/bin/casa --pipeline -c member.uid___A001_X 1284_X265f.hifa_calimage.casa_piperestorescript.py). This fires right up and gets to work, but failed on the 3rd EB (of 4!) due to a typo in my download of the data. Further refinements have shown that there is an error with observatory position which the piperestorescript.py doesn't handle properly, although my PPR gets further....

For Greater Flexibility:

The restore script is very useful, but for the AAT-PPI, we're trying to stick to a single method of interacting with the casa pipeline: Creating a PPR. We'll need more information for the PPR:

Project Code
1. SELECT DISTINCT PROJECT_CODE FROM ASA_PROJECT JOIN ASA_SCIENCE ON ASA_PROJECT.PROJECT_UID = ASA_SCIENCE.PROJECT_UID WHERE MEMBER_OUSS_ID='';
ObsProject uid
ObsUnitSet partId
1. 2) and 3) can be gotten together: SELECT OBSUNITSETUID, OBSUNITSETPARTID FROM AQUA_OUS WHERE OUS_STATUS_ENTITY_ID = '';
ProjectStatusRef
1. SELECT OBS_PROJECT_STATUS_ID FROM OBS_UNIT_SET_STATUS WHERE STATUS_ENTITY_ID='';
OUSStatusRef
1. the entity ID for this is just the MOUS id we're given
EB → Session Mappings:
1. This one is nontrivial, first we want a list of EBs:
  1. SELECT DISTINCT ASDM_UID FROM ASA_SCIENCE WHERE MEMBER_OUSS_UID='';
2. Then, we extract the MOUS Status XML file:
  1. SELECT XML FROM OBS_UNIT_SET_STATUS WHERE STATUS_ENTITY_ID='';
3. Then we need to walk the XML for each SESSION segment create a list of EBs for that session.
4. Check for Fail or SemiPass EBs in AQUA_EXECBLOCKS, and exclude their sessions
  1. WAIT: Might be easier to check the SESSIONID (and QA1 status?)
5. Then arrange the data into the PPR matching sessions (to the EBs (multiple separated by '|') to EBs in the Intents section.
SchedBlockRef
1. SELECT DISTINCT SCHEDBLOCK_UID FROM ALMA.ASA_SCIENCE WHERE MEMBER_OUSS_ID = '';
2. If there's more than one, then we have an issue, but the DOMAIN_ENTITY_STATE of the SCHED_BLOCK_STATUS table should be able to tell us which we want.
SBStatusRef
1. SELECT STATUS_ENTITY_ID FROM SCHED_BLOCK_STATUS WHERE DOMAIN_ENTITY_ ID='';

NOTE: Running it this way requires at minimum the 3 SCIPIPE evironment variables (ROOT, LOG, SCRIPT) which I have notes about elsewhere.

NOTE: It may be possible to skip some of that information (2-5). We have been ignoring the ProjectStructure section of the PPR for years with no difficulties, but not currently. The way I have found that works is to incorporate the ProjectStructure section, and use the runpipeline.py (instead of runvlapipeline.py) in conjuction with the full Project/SOUS/GOUS/MOUS directory structure. More experimentation (and perhaps understanding of the pipeline infrastructure system) is necessary before we can characterize this completely.

I've created a first-pass PPR for our test data set, but it has some flaw I'm not seeing yet (the pipeline rejects the ProcessingRequests). PPR_jls.xml

I have one that works as completely as the piperestorescript.py : failing on an observatory position. PPR_jls_worked.xml

NOTE: Copy the pipeline manifest XML files over to rawdata.

Now we need some further experimentation, and some idea of how to handle any variation in parameters to the hifa_restoredata call.

A New Test With uid://A001/X12a3/X80e

EBs:

Product Files (calibration & script FILE_CLASS):

member.uid___A001_X12a3_X80e.calimage.pipeline_manifest.xml
member.uid___A001_X12a3_X80e.hifa_calimage.auxproducts.tgz → → → → → WARNING: need to unzip this one so the pipeline can access the files within.
member.uid___A001_X12a3_X80e.hifa_calimage.casa_piperestorescript.py
member.uid___A001_X12a3_X80e.hifa_calimage.casa_pipescript.py
member.uid___A001_X12a3_X80e.session_2.auxcaltables.tgz
member.uid___A001_X12a3_X80e.session_2.caltables.tgz
uid___A002_Xcf92df_X3453.ms.calapply.txt
uid___A002_Xcf92df_X3453.ms.flagversions.tgz
uid___A002_Xcf92df_X3453_target.ms.auxcalapply.txt
member.uid___A001_X12a3_X80e.scriptForPI.py

From the looks of this, we just have a single session (2), with 2 ebs.

Directory Structure:

2017.1.01347.S
- SOUS_uid___A001_X12a3_X80c
  - GOUS_uid___A001_X12a3_X80d
    - MOUS_uid___A001_X12a3_X80e

Info for the PPR (using Alma_Restore_PPR_Queries.sql.html):

Project Code: 2017.1.01347.S
ObsProject UID: uid://A001/X1221/Xdf8
ObsProject PartID: X659301709
ObsProjectStatusUID: uid://A001/X1221/Xdfc
OUSStatusRef: same as our base information
Scheduling Block UID: uid://A001/X12a3/X804
Scheduling Block Status UID: uid://A001/X12a3/X80f

Status XML:

<?xml version="1.0" encoding="UTF-8"?>
<ouss:OUSStatus xmlns:val="Alma/ValueTypes" xmlns:orv="Alma/ObsPrep/ObsReview" xmlns:ouss="Alma/Scheduling/OUSStatus" xmlns:oat="Alma/ObsPrep/ObsAttachment" xmlns:ps="Alma/Scheduling/ProjectStatus" xmlns:sbs="Alma/Scheduling/SBStatus" xmlns:prj="Alma/ObsPrep/ObsProject" xmlns:ent="Alma/CommonEntity" xmlns:sbl="Alma/ObsPrep/SchedBlock" xmlns:prp="Alma/ObsPrep/ObsProposal" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" schemaVersion="13" revision="23" almatype="APDM::OUSStatus" xsi:type="ouss:OUSStatus">
 <ps:Status State="Delivered"/>
 <ps:TimeOfUpdate>2018-07-10 11:00:46.672</ps:TimeOfUpdate>
 <ps:executionsRemaining>-1</ps:executionsRemaining>
 <ps:successfulExecutions>2</ps:successfulExecutions>
 <ps:failedExecutions>0</ps:failedExecutions>
 <ps:secondsRemaining>7200</ps:secondsRemaining>
 <ps:successfulSeconds>0</ps:successfulSeconds>
 <ps:failedSeconds>0</ps:failedSeconds>
 <ps:hasExecutionCount>true</ps:hasExecutionCount>
 <ps:hasTimeLimit>true</ps:hasTimeLimit>
 <ps:bookkeepingInitialized>false</ps:bookkeepingInitialized>
 <ps:TotalUsedTimeInSec>0</ps:TotalUsedTimeInSec>
 <ps:ContainingObsUnitSetRef entityId="uid://A001/X12a3/X80d" entityTypeName="OUSStatus" documentVersion="1"/>
 <ps:ProjectStatusRef entityId="uid://A001/X1221/Xdfc" entityTypeName="ProjectStatus" documentVersion="1"/>
 <ouss:OUSStatusEntity entityId="uid://A001/X12a3/X80e" entityIdEncrypted="-- id encryption not implemented --" entityTypeName="OUSStatus" schemaVersion="13"/>
 <ouss:ObsUnitSetRef entityId="uid://A001/X1221/Xdf8" partId="X659301709" entityTypeName="ObsProject"/>
 <ouss:NumberSBsCompleted>2</ouss:NumberSBsCompleted>
 <ouss:SBStatusRef entityId="uid://A001/X12a3/X80f" entityTypeName="SBStatus" documentVersion="1"/>
 <ouss:Session entityPartId="X00000000" almatype="APDM::Session">
 <ouss:StartTime>2018-06-06 12:41:10.826</ouss:StartTime>
 <ouss:EndTime>2018-06-06 13:07:22.036</ouss:EndTime>
 <ouss:ExecBlockRef>
 <val:ExecBlockId>uid://A002/Xce574d/Xa33e</val:ExecBlockId>
 </ouss:ExecBlockRef>
 <ouss:SBStatusRef entityId="uid://A001/X12a3/X80f" entityTypeName="SBStatus"/>
 </ouss:Session>
 <ouss:Session entityPartId="X00000001" almatype="APDM::Session">
 <ouss:StartTime>2018-07-10 10:34:32.962</ouss:StartTime>
 <ouss:EndTime>2018-07-10 11:00:46.858</ouss:EndTime>
 <ouss:ExecBlockRef>
 <val:ExecBlockId>uid://A002/Xcf92df/X3453</val:ExecBlockId>
 </ouss:ExecBlockRef>
 <ouss:SBStatusRef entityId="uid://A001/X12a3/X80f" entityTypeName="SBStatus"/>
 </ouss:Session>
</ouss:OUSStatus>

Ok, so I was wrong. Looks like there's two sessions, but that would indicate that we might be missing products in our list.

But notice, the SBStatusRef and ExecBlockIds match what we found before, which might be a good consistency check if we can parse this XML reasonably.

And here's the PPR I created:

<?xml version="1.0" encoding="UTF-8"?>
<SciPipeRequest xmlns:ent="Alma/CommonEntity"
    xmlns:val="Alma/ValueTypes" xmlns:prp="Alma/ObsPrep/ObsProposal"
    xmlns:orv="Alma/ObsPrep/ObsReview"
    xmlns:ps="Alma/ObsPrep/ProjectStatus"
    xmlns:oat="Alma/ObsPrep/ObsAttachment"
    xmlns:prj="Alma/ObsPrep/ObsProject"
    xmlns:sbl="Alma/ObsPrep/SchedBlock"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="SciPipeRequest">
    <SciPipeRequestEntity entityId="UID_UNASSIGNED"
        entityTypeName="SciPipeRequest" datamodelVersion="0.1"/>
    <ProjectSummary>
        <ProposalCode>2017.1.01347.S</ProposalCode>
        <ProposalTitle>This is our test data set</ProposalTitle>
        <Observatory>ALMA Joint Observatory</Observatory>
        <Telescope>ALMA</Telescope>
        <ProcessingSite>NAASC</ProcessingSite>
        <Operator>Jim Sheckard</Operator>
        <Mode>CSV</Mode>
        <Version>Undefined</Version>
        <CreationTime>2018-08-07T13:18:00.000</CreationTime>
    </ProjectSummary>
    <ProjectStructure>
       <ObsUnitSetRef entityId="uid://A001/X1221/Xdf8"
            partId="X659301709" entityTypeName="ObsProject"/>
        <ObsUnitSetTitle>Undefined</ObsUnitSetTitle>
        <ObsUnitSetType>Member</ObsUnitSetType>
        <ProjectStatusRef entityId="uid://A001/X1221/Xdfc"
            entityTypeName="ProjectStatus" documentVersion="1"/>
        <OUSStatusRef entityId="uid://A001/X12a3/X80e" entityTypeName="OUSStatus"/>
    </ProjectStructure>
    <ProcessingRequests>
      <ProcessingRequest>
        <RootDirectory>/lustre/naasc/web/almapipe/pipeline/vatest/testing</RootDirectory>
        <ProcessingIntents>
          <Intents>
            <Keyword>PROCESS</Keyword>
            <Value>true</Value>
          </Intents>
          <Intents>
            <Keyword>SESSION_1</Keyword>
            <Value>uid://A002/Xce574d/Xa33e</Value>
          </Intents>
          <Intents>
            <Keyword>SESSION_2</Keyword>
            <Value>uid://A002/Xcf92df/X3453</Value>
          </Intents>
          <Intents>
            <Keyword>INTERFEROMETRY_STANDARD_OBSERVING_MODE</Keyword>
            <Value>Undefined</Value>
          </Intents>
        </ProcessingIntents>
        <ProcessingProcedure>
          <ProcedureTitle>hifa_restore_jls</ProcedureTitle>
          <ProcessingCommand>
            <Command>hifa_restoredata</Command>
            <ParameterSet/>
          </ProcessingCommand>
        </ProcessingProcedure>
        <DataSet>
          <SchedBlockSet>
            <SchedBlockIdentifier>
              <RelativePath>jls_ous_restore_v1_myppr</RelativePath>
              <SchedBlockRef entityId="uid://A001/X12a3/X804"
                             entityTypeName="SchedBlock" documentVersion="1"/>
              <SBStatusRef entityId="uid://A001/X12a3/X80f" entityTypeName="SBStatus"/>
              <SBTitle>Undefined</SBTitle>
              <AsdmIdentifier>
                <AsdmRef>
                  <ExecBlockId>uid://A002/Xce574d/Xa33e</ExecBlockId>
                </AsdmRef>
                <AsdmDiskName>uid___A002_Xce574d_Xa33e</AsdmDiskName>
              </AsdmIdentifier>
              <AsdmIdentifier>
                <AsdmRef>
                  <ExecBlockId>uid://A002/Xcf92df/X3453</ExecBlockId>
                </AsdmRef>
                <AsdmDiskName>uid___A002_Xcf92df_X3453</AsdmDiskName>
              </AsdmIdentifier>
            </SchedBlockIdentifier>
          </SchedBlockSet>
        </DataSet>
      </ProcessingRequest>
    </ProcessingRequests>
    <ResultsProcessing>
        <ArchiveResults>false</ArchiveResults>
        <CleanUpDisk>false</CleanUpDisk>
        <UpdateProjectLifeCycle>false</UpdateProjectLifeCycle>
        <NotifyOperatorWhenDone>false</NotifyOperatorWhenDone>
        <PipelineOperatorAdress>Unknown</PipelineOperatorAdress>
    </ResultsProcessing>
</SciPipeRequest>

But I'm still a bit nervous about the 1session/2session thing. Doing a bit more looking, the manifest only mentions one of the two exec blocks, not both. I quick peek at the AQUA_EXECBLOCK table shows one as SemiPass in QA0, so that may have something to do with it.

Even Better: the pipeline-generated restore script:

from recipes.almahelpers import fixsyscaltimes # SACM/JAO - Fixes
__rethrow_casa_exceptions = True
h_init()
try:
    hifa_importdata (dbservice=False, bdfflags=False, vis=['../rawdata/uid___A002_Xcf92df_X3453'], session=['session_2'], ocorr_mode='ca')
    fixsyscaltimes(vis = 'uid___A002_Xcf92df_X3453.ms')# SACM/JAO - Fixes
    h_save() # SACM/JAO - Finish weblog after fixes
    h_init() # SACM/JAO - Restart weblog after fixes
    hifa_restoredata (vis=['uid___A002_Xcf92df_X3453'], session=['session_2'], ocorr_mode='ca')
finally:
    h_save()

So we would need to need to add two additional procedures. How can we know when this is necessary?

How common is this?

Catarina did a good job with this curveball. Was it intentional?

Even better: We can skip the XML portion.

Using the AQUA_EXECBLOCKS & AQUA_SESSIONS tables, we can eliminate the need for parsing the MOUS Status XML blob.

SELECT DISTINCT ASA_SCIENCE.ASDM_UID, AQUA_EXECBLOCK.SESSIONID, S.ENDTIME FROM ALMA.AQUA_EXECBLOCK
JOIN ALMA.ASA_SCIENCE ON ASA_SCIENCE.ASDM_UID = AQUA_EXECBLOCK.EXECBLOCKUID
JOIN AQUA_SESSION S on AQUA_EXECBLOCK.SESSIONID = S.SESSIONID
WHERE ASA_SCIENCE.MEMBER_OUSS_ID='uid://A001/X1284/X265f'
ORDER BY S.ENDTIME ASC;

This provides a time-ordered list of EBs and their matching session ID. So, make a session_1 out of the ebs that match the first ID listed, etc. Then prune EBs by the AQUA_EXECBLOCKS.QA0STATUS: remove EBs that are not PASS (this needs TBD, but I've seen an SemiPass skipped). Even if a sessions is emptied by this, leave it in place, you need to match the correct session-labeled calibration files to their relevant EB.

I've used this to pick out proper session:eb mapping & session ordering for 5 different MOUS so far: uid://A001/X1284/X265f, uid://A001/X12a3/X80e, uid://A001/X12cc/X4a, uid://A001/X1284/X266, ~~uid://A001/X1284/Xf3b~~.

The XML Strikes Back:

uid://A001/X1284/Xf3b is infact NOT quite handled. There was an attempted, but FAILED execution of the SB. The EB does not show as tied to the MOUS in ASA_SCIENCE, and the only connection I see thus far to anything is via the SB & MOUS Status XML.

Page tree

The Do-It-Yourself Guide to Alma Restores