Notes on OUS-palooza

This page will hold our notes from implementing first fringes on an OUS level ALMA restore, as tracked in JIRA:

SSA-4811 - Getting issue details... STATUS .

In addition, for familiarization purposes, we should upgrade our current ACS installation:

SSA-4916 - Getting issue details... STATUS .

Test Data Set Details:

    Project 2017.1.00886.L

    MOUS ID: uid://A001/X1284/X265f

    made up of four ASDM EBs: uid://A002/Xc7111c/X3979,
    uid://A002/Xc7111c/X409e, uid://A002/X7111c/X86a7, uid://A002/Xc72427/X2dfe

        

        All these were processed recently, so have the calibration
        products stored separately (note though the data are still
        proprietary, so should not be shown too widely).

Adding the almapipe group to our accounts might make life easier (particularly for look-but-don't-touch access into the working areas).

PPR Examples:

Kana was kind enough to provide a few example PPRs that perform multi-eb processing:

A combined calibration/imaging run for an OUS for 2 EBs from a single Session: calimage_2eb_1session_PPR_uid___A001_X12cc_X4b.xml
A calibration-only run for an OUS containing 2 EBs which spanned 2 Sessions: calonly_2eb_2session_PPR_uid___A001_X1284_X267.xml
An imaging-only run for an OUS containing 2 EBs which spanned 2 Sessions: imageonly_2eb_2session_PPR_uid___A001_X1284_X267.xml

I'm just starting in on understanding these, and comparing them with the PPRs for CIPL and Vlass quicklook imaging.

Software Details:

For our test data set, we want to be using CASA/pipeline version 5.1.1-5r40896. This is the ALMA-approved version of CASA for Cycle 5. See: https://almascience.nrao.edu/processing/science-pipeline

The new functionality for the ALMA Common Software (ACS) is built into the asdmExportLight command through a new option. This hopefully will make it fairly easy (once we understand the Project & Database structures) to retrieve the desired calibration information. One wrinkle, however, is that ACS is an overloaded acronym. There is the 'base package' of ACS, and extensions to it. Further investigation has revealed that we will need the Archive extension at minimum, and likely will want the Pipeline & ICD components as well.

The pipelineMakeRequest tool will be useful for the creation of comparison scripts for our more extended PPR generation software. It is ALMA focused, and builds the entire directory structure (including data download, if requested), complete with OUS Status files for each level.

The NAASC keeps a set of ACS installations for their purposes in /home/acs. Unfortunately, this area doesn't include the ACS-2018JUN release of the software with a fixed version of calibration retrieval. I'll perform a test installation of an updated Enhanced-ACS in /home/alderaan_2/ for initial verification & prototyping.

SSA may also need to update the version of ACSSW in /home/ssa/ACS, as we are currently using a version tagged for Cycle 3. After the initial verification, the /home/ssa/ACS area can be updated with an updated version to support the restore functionality.

ALMA Archive Questions:

We need to generally understand the layout of their archive, and how things interlink. But if we come up with specific questions as well, record them.

Each OUS has a status associated with it.
- How are they used (observation? reduction? both?)
- What are the possible status values?
- What triggers a transition in status?
- Are statuses at all levels equally important?
Update Timescales.
- periodic checks reasonable?
- proprietary or calibration status clawback?
Full Structure: Plan vs Observation vs Reduction
- Potential:
  - Proposal
    - Science Goal(s)
      - Each Science Goal is tied to a single ALMA receiver.
      - Each Science Goal has an associated SGOUS once the proposal becomes a project.
- Observation:
  - Project:
    - SGOUS
      - GOUS
        MOUS
        Each MOUS contains data from a single array configuration.
        Typically 1:1 with Scheduling Blocks, unless there's a problem with the SB.
        Scheduling Block(s)
        Each Scheduling Block contains:
        Once/session calibration scans (bandpass, flux calibrator, etc)
        Structure for switching between science target(s) and more variable calibrators (phase, polarization, etc)
        Session(s)
        Due to system & atmospheric stability issues, each SB is limited to about 2 hours. If conditions permit, the SB can be run multiple times in succession. Those contiguous runs are called sessions.
        They seem to be important for CASA, and so should be tracked.
        Execution Blocks
        SDM+BDFs for a single execution of an SB. The basic data units (stored as tar files in ngas)
        Failed observations handled how?
- Reduction:
  - Project
    - Science Goal OUS(s)
      - Group OUS (1:1 with science goals for the foreseeable future).
        Member OUS(s)
        Calibration & Imaging performed at this level via the pipeline.
        Proprietary Period begins upon the release of Imaging products to the PI.
        Session(s)
        Execution Blocks(s) or ASDM(s)
How will we deal with extracting calibration tables that were ingested before the recent changes?
- Are we gonna do that, or is handling it an ALMA infrastructure thing?
  - There currently is no streamlined method for extracting the calibration information within the ALMA infrastructure software.
  - ToDo: We'll need a homegrown methodology of extracting the calibration tables directly from ngas with wget (prototype to be provided by Mark Lacy)
- Are they gonna go back and split the calibrations out of the older observations?
  - The older calibrations will be split out at some later time, allowing restores of the older data.
  - How will we know when this is accomplished for an MOUS?
Others?

Relevant Links:

CIPL Restore Instructions
Comparing Measurement Sets (Rough Draft)
Base ACS homepage.
1. Suggested Version: 2018Jun
2. Download Location: here (requires login)
Other ACS component homes?
1. ARCHIVE
2. PIPELINE
3. ICD
4. Others unlikely to be relevant

Page tree