What are Sessions?
Typically, when considering the the structure of ALMA data, it is only at the Observational Unit Set (OUS) level. Unfortunately, when it comes to processing ALMA data, there are finer levels of structure which matter. This is due to how ALMA handles the concept of 'Array Configuration'.
For the EVLA, an array configuration is a static mapping of antenna layout (and for each individual instance of that configuration, you know which antenna is sitting on which pad), and typically that information stays roughly the same for the entire ~3 months of that configuration. Between each configuration is a somewhat hectic period of shuffling antennae into their new positions for the next configuration.
For ALMA, configurations are not nearly so defined. First off, ALMA configurations are not specifically defined. They're characterized by 'typical' baseline minima and maxima, but those values are not hard limit. The change between C-2 and C-3 configurations is not the result of a large-effort short-time process. Instead, configuration changes are less defined, as ALMA antennae are shifted on a near-daily basis. An ALMA configuration change is more a matter of the new configuration's parameters better matching the current array layout.
Because of the frequent movement of antennas, data taken even a few days apart require different definitions of antenna locations and other fundamental reference information. For the purposes of data processing, ALMA uses the concept of SESSIONS. A session is a set of execution blocks which were observed nearly back-to-back, and thus can use the same reference information. If there are changes to the references between two execution blocks, the new execution block is part of a new session.
This process mostly happens under the hood for ALMA users, but the AAT/PPI needs to be aware of this procedure because the mapping of an EB into the correct session is required to generate a restored Calibrated Measurement Set. The calibration table information is labeled by session number, requiring the accurate recreation of the initial session:EB mapping.
Further complicating the process is that in creating this mapping the ALMA pipeline counts failed observations in the creation of sessions, so while only the EBs with QA0 status 'Pass' are used in calibration or imaging, those with 'Fail' and 'SemiPass' must be accounted for in the creation of the session information within the PPR.
Creating Session Mappings for a Restore PPR:
In order to create the appropriate mapping, you need a listing of ALL execution blocks associated with a particular MOUS. The following query (modified from one provided by Kana Sugimoto) provides the fundamental information we require:
select AQUA_EXECBLOCK.EXECBLOCKUID, AQUA_EXECBLOCK.SESSIONID, AQUA_SESSION.STARTTIME from AQUA_EXECBLOCK
join AQUA_SESSION on AQUA_EXECBLOCK.SESSIONID = AQUA_SESSION.SESSIONID
join SHIFTLOG_ENTRIES on SHIFTLOG_ENTRIES.SE_EB_UID = AQUA_EXECBLOCK.EXECBLOCKUID
join BMMV_SCHEDBLOCK on BMMV_SCHEDBLOCK.ARCHIVE_UID = SHIFTLOG_ENTRIES.SE_SB_ID
where BMMV_SCHEDBLOCK.MOUS_STATUS_UID = ?
order by AQUA_SESSION.STARTTIME ASC;
This gives you a means of grouping EBs which are in the same session together, ordered by their time of observation. For each separate session (labeled as SESSION_1, SESSION_2, etc not by their database ID), remove from the list any QA0 SemiPass and Fail:
select QA0STATUS from AQUA_EXECBLOCK where EXECBLOCKUID='ASDM_UID';
This may leave some session lables with NO associated EBs. This is not an unreasonable result, and those empty sessions are required to provide the propper mapping. The list of remaining EBs should then be flattened out into a string, with the UIDs separated with ' | ' between multiple EBs.
That group of paired strings can be passed to the AlmaPprCreator.generateProcessingIntents() method and it will generate the appropriate XML, as in the example PPR below:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns2:SciPipeRequest xmlns:ns2="Alma/pipelinescience/AlmaSciPipeRequest">
<SciPipeRequestEntity entityId="uid://NOPE/X0000/X0000" entityIdEncrypted="none" entityTypeName="SciPipeRequest" schemaVersion="3.14" datamodelVersion="0.1"/>
<ProjectSummary>
<ProposalCode>2017.1.00895.S</ProposalCode>
<ProposalTitle>Unknown</ProposalTitle>
<PIName>Elisabeth Mills</PIName>
<Observatory>JAO</Observatory>
<Telescope>ALMA</Telescope>
<ProcessingSite>NAASC</ProcessingSite>
<Operator>almapipe</Operator>
<Mode>CSV</Mode>
<Version>Undefined</Version>
<CreationTime>2019-02-09T04:55:07</CreationTime>
</ProjectSummary>
<ProjectStructure>
<ObsUnitSetRef entityId="uid://A001/X1220/Xa74" partId="X1541554216" entityTypeName="ObsProject" documentVersion="0.1"/>
<ObsUnitSetTitle>Undefined</ObsUnitSetTitle>
<ObsUnitSetType>Member</ObsUnitSetType>
<ProjectStatusRef entityId="uid://A001/X1220/Xa78" entityTypeName="ProjectStatus" documentVersion="0.1"/>
<OUSStatusRef entityId="uid://A001/X129e/Xb6" entityTypeName="OUSStatus" documentVersion="0.1"/>
</ProjectStructure>
<ProcessingRequests>
<RootDirectory>/lustre/naasc/web/almapipe/pipeline/vatest/spool</RootDirectory>
<ProcessingRequest>
<ProcessingIntents>
<Intents>
<Keyword>PROCESS</Keyword>
<Value>true</Value>
</Intents>
<Intents>
<Keyword>SESSION_9</Keyword>
<Value>uid://A002/Xc77d4f/X522</Value>
</Intents>
<Intents>
<Keyword>SESSION_10</Keyword>
<Value>uid://A002/Xc77d4f/X3d06</Value>
</Intents>
<Intents>
<Keyword>SESSION_11</Keyword>
<Value>uid://A002/Xc77d4f/X46ca</Value>
</Intents>
<Intents>
<Keyword>SESSION_12</Keyword>
<Value>uid://A002/Xc77d4f/X4d4d</Value>
</Intents>
<Intents>
<Keyword>SESSION_13</Keyword>
<Value>uid://A002/Xc790bf/X1201 | uid://A002/Xc790bf/X3e2</Value>
</Intents>
<Intents>
<Keyword>SESSION_14</Keyword>
<Value>uid://A002/Xc790bf/X5bf1</Value>
</Intents>
<Intents>
<Keyword>SESSION_15</Keyword>
<Value>uid://A002/Xc790bf/X608a</Value>
</Intents>
<Intents>
<Keyword>SESSION_3</Keyword>
<Value>uid://A002/Xc74b5b/X644d</Value>
</Intents>
<Intents>
<Keyword>SESSION_4</Keyword>
<Value>uid://A002/Xc75eba/X9aa</Value>
</Intents>
<Intents>
<Keyword>SESSION_1</Keyword>
<Value>uid://A002/Xc733ea/X179e</Value>
</Intents>
<Intents>
<Keyword>SESSION_2</Keyword>
<Value>uid://A002/Xc73dee/X798</Value>
</Intents>
<Intents>
<Keyword>SESSION_7</Keyword>
<Value>uid://A002/Xc772ca/X1f5f</Value>
</Intents>
<Intents>
<Keyword>SESSION_8</Keyword>
<Value>uid://A002/Xc77d4f/X109</Value>
</Intents>
<Intents>
<Keyword>SESSION_5</Keyword>
<Value>uid://A002/Xc75eba/X54db</Value>
</Intents>
<Intents>
<Keyword>SESSION_6</Keyword>
<Value>uid://A002/Xc772ca/Xaaf</Value>
</Intents>
<Intents>
<Keyword>INTERFEROMETRY_STANDARD_OBSERVING_MODE</Keyword>
<Value>Undefined</Value>
</Intents>
</ProcessingIntents>
<ProcessingProcedure>
<ProcedureTitle>AAT/PPI Alma Restore</ProcedureTitle>
<ProcessingCommand>
<Command>hifa_restoredata</Command>
<ParameterSet>
<Parameter>
<Keyword>ocorr_mode</Keyword>
<Value>ca</Value>
</Parameter>
</ParameterSet>
</ProcessingCommand>
</ProcessingProcedure>
<DataSet>
<SchedBlockSet>
<SchedBlockIdentifier>
<RelativePath>uid___A002_Xc790bf_X608a_2017_12_06_T15_44_53.474</RelativePath>
<SchedBlockRef entityId="uid://A001/X129e/Xb2" entityTypeName="SchedBlock" documentVersion="0.1"/>
<SBStatusRef entityId="uid://A001/X129e/Xb7" entityTypeName="SBStatus" documentVersion="0.1"/>
<SBTitle>unknown</SBTitle>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc77d4f/X522</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc77d4f_X522</AsdmDiskName>
</AsdmIdentifier>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc77d4f/X3d06</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc77d4f_X3d06</AsdmDiskName>
</AsdmIdentifier>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc77d4f/X46ca</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc77d4f_X46ca</AsdmDiskName>
</AsdmIdentifier>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc77d4f/X4d4d</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc77d4f_X4d4d</AsdmDiskName>
</AsdmIdentifier>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc790bf/X1201</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc790bf_X1201</AsdmDiskName>
</AsdmIdentifier>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc790bf/X3e2</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc790bf_X3e2</AsdmDiskName>
</AsdmIdentifier>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc790bf/X5bf1</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc790bf_X5bf1</AsdmDiskName>
</AsdmIdentifier>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc790bf/X608a</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc790bf_X608a</AsdmDiskName>
</AsdmIdentifier>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc74b5b/X644d</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc74b5b_X644d</AsdmDiskName>
</AsdmIdentifier>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc75eba/X9aa</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc75eba_X9aa</AsdmDiskName>
</AsdmIdentifier>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc733ea/X179e</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc733ea_X179e</AsdmDiskName>
</AsdmIdentifier>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc73dee/X798</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc73dee_X798</AsdmDiskName>
</AsdmIdentifier>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc772ca/X1f5f</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc772ca_X1f5f</AsdmDiskName>
</AsdmIdentifier>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc77d4f/X109</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc77d4f_X109</AsdmDiskName>
</AsdmIdentifier>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc75eba/X54db</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc75eba_X54db</AsdmDiskName>
</AsdmIdentifier>
<AsdmIdentifier>
<AsdmRef>
<ExecBlockId>uid://A002/Xc772ca/Xaaf</ExecBlockId>
</AsdmRef>
<AsdmDiskName>uid___A002_Xc772ca_Xaaf</AsdmDiskName>
</AsdmIdentifier>
</SchedBlockIdentifier>
</SchedBlockSet>
</DataSet>
</ProcessingRequest>
</ProcessingRequests>
<ResultsProcessing>
<ArchiveResults>false</ArchiveResults>
<CleanUpDisk>false</CleanUpDisk>
<UpdateProjectLifeCycle>false</UpdateProjectLifeCycle>
<NotifyOperatorWhenDone>false</NotifyOperatorWhenDone>
<SDMall>false</SDMall>
<SDMonly>false</SDMonly>
<PipelineOperatorAddress>unknown</PipelineOperatorAddress>
</ResultsProcessing>
</ns2:SciPipeRequest>
Implementation Lessons Learned:
Even if it might be tempting, DO NOT use the ASA_SCIENCE table for this. It only lists QA0 Pass and SemiPass ASDMs, and in addition has been found to be fundamentally wrong on several occasions (ASDMs listed under two separate MOUSes, ASDMs simply missing from their MOUS listing, etc).
The current implementation of ALMA restore PPR generation separates all the database calls from the XML handling. Since the database calls are all structured based upon the PPR's needs, it may be a better design to simplify the outward PPR generation interface and encapsulate much of the structure-specific data retrieval.
Comparing the session structure created to the initial PPR (which is archived alongside the calibration tables) is a good test of whether a restore will proceed correctly.
2 Comments
Daniel Lyons
What are "43-2 and 43-3 configurations"?
Do session numbers start at 1 within the OUS for a project, or is the first session in an OUS liable to start with whatever the last observation finished with? Is this function monotonically increasing?
James Sheckard
I was going on memory for the configuration names. Good catch! They have configurations C-1 to C-10 (I'll edit that) but there are also references to C43-1 to C43-10 as their 'representative configurations'. Basically, the properties of the C43-N configurations are used for planning purposes under the assumption that 43/50 antennae active at a time is their baseline.
Current Processing: For any MOUS, session numbers always start a 1, and count up until they're finished observing.
Future speculation: Information about sessions are also stored in their database (so there's a auto-gen increasing number in the shadows). I gather that the 'true' session identifiers may get incorporated sometime in the distant future, mostly related to better polarization calibrations.