The Observation Unit Set (OUS) used by ALMA servers a dual purpose. It provides an organizational structure for separating Scheduling Blocks (SBs) for different configurations or arrays when generating those SBs from the higher level Science Goals defined in the proposal. An OUS also provides an organizational structure for data processing which involves more than one EB, which is the standard case for the ALMA pipeline (unlike the EVLA, where each SB is calibrated separately).
ALMA creates a 3-layer structure of OUSs by default, but only uses the lowest level (also called a Member OUS or MOUS) for any automated processing. The upper layers were envisioned to be used in more complicated imaging or analysis tasks, but they are not currently used, and are not likely to be used in the near future (~10 years). For the time being, we can restrict ourselves to a single OUS layer for any processing we might need.
Fundamental change: The OUS not only links the inputs to the products, but it contains information about how they were gotten.
An OUS is a container at the metadata level, largely analogous to our filegroups for organizing collections of files which make up the contents of NGAS. An OUS links the data used in processing, to the results of that processing. For instance: every science observation by the VLA results in a single Execution Block (EB) which is then calibrated. This can be represented as an OUS with a link to the row in the execution_blocks table and the (yet to be defined or populated) calibration_tables table. A separate run of the calibration pipeline of the same data is represented with a new OUS pointing to the same EB, but to the new calibration information.
Since the VLA does not use the OUS concept on the observation planning side (and VLA SBs can be much longer than ALMA SBs), we may wish to have our own name for this structure that makes sense to us. Any suggestions? The next-gen PST planning committee seems to be leaning toward the name 'Program', but that doesn't feel quite right either.
So, an OUS/Program/Fizz has a set of 'requirements' (i.e. things it needs to do its work) and a type of processing it wants to do. Some, however, might want to react to individual components of its requirements being fulfilled. So it might perform intermediate actions as requirements are filled, and then a final processing that happens once all requirements are filled. (Not yet reflected below, design needs to change) (Will intermediate actions be internal to the organizational structure (monitoring projects) or should it be more general to allow some sort of more intensive processing?)
Database Considerations
As the OUS structure would have a One-to-Many Relationship to various tables, I propose the following set of tables to handle the relationships.
observation_set Table
| Column Name | Column Type | Comments |
|---|---|---|
| ous_id | integer | auto-generated ID value for this OUS |
| Structural Columns | ||
| project_id | integer | parent Project for the OUS if it is not the child of another OUS |
| parent_ous_id | integer | parent OUS for this OUS (null for top-level OUSs) |
| has_data | boolean | Indicates whether a join with the data linking table will be useful |
| has_products | boolean | Indicates whether a join with the products linking table will be useful |
| Metadata Columns | ||
| ous_name | varchar | Human-readable name for this processing structure |
| purpose | integer | Combined values of the types of products this structure will produce (planned, to allow multi-stage processing) |
| time_of_creation | datetime | When this was created (may play into proprietary period) |
| last_update | datetime | Last addition to this structure (typically last ingested products) |
ous_to_execution_blocks
| Column Name | Column Type | Comments |
|---|---|---|
| ous_id | integer | Foreign Key to the observation_set table |
| execution_block_id | integer | Foreign Key to the execution_blocks table |
ous_to_products
| Column Name | Column Type | Comments |
|---|---|---|
| ous_id | integer | Foreign Key to the observation_set table |
| product_type_id | integer | Foreign Key to the products_type table |
| execution_block_id | integer | id value for the relevant product table |
ous_to_workflows
Column Name
| Column Name | Column Type | Comments |
|---|---|---|
| ous_id | integer | Foreign Key to the observation_set table |
| workflow_id | integer | Foreign Key to the workflows table |
product_type
This table would be a slowly expanding list of our processing products. Currently we have calibration_tables (as yet undefined) and image_sets. Coming shortly are catalogs (at least an initial version for VLASS processing) followed by cubes.
| Column Name | Column Type | Comments |
|---|---|---|
| product_type_id | integer | Chosen (factor of 2, or 10, etc) for combination purposes, as done for polarizations |
| product_name | varchar | Name of the type (Calibrations, Images, Catalogs, etc) |
| product_table | varchar | Table name for these types of products. |