Overview

Ingestion of RealFast SDMs is an extension of the standard SDM ingestion process.  In short, the process involves: 

1. Ingest the SDM for the EB as normal
2. Ingest the associated PNG candidate file(s)
3. Link the new EB to its donor EB 

The details are covered in the following pages: 

Collection Support: realfast

Using the System

Assumptions: 

• RealFast team has access to the AAT/PPI command line installation areas (/users/vlapipe/workflows/)
• Vlapipe user (and group) have access (Read, Write, Execute) access.  
• Everyone has (Read) access (to facilitate the AAT/PPI services access) 
• The defined staging area (see below) is on the same filesystem as the data to be ingested

The process is contained within a special-purpose workflow, which can be initiated with the 'realfastIngest' command, installed under the vlapipe account.  

NOTE: There are separate installations for the Test and Production systems, both living in the vlapipe account's area.  Be careful to use the one you want. command & options

mcilroy$ /users/vlapipe/workflows/nmtest/bin/realfastIngest -h
usage: realfastIngest [-h] [-P PROFILE] [-s SDM_PATH] [-p PNG_PATH]
                      sdmName [sdmName ...]

RealFast SDM Ingestion, version 3.9.0b2: Initiates an ingestion workflow for the SDM and ancillary 
    files for each execution block listed.

positional arguments:
  sdmName               FileSet Identifiers(s) to ingest

optional arguments:
  -h, --help            show this help message and exit
  -P PROFILE, --profile PROFILE
                        profile name to use, e.g. nmtest, mnprod
  -s SDM_PATH, --sdm_path SDM_PATH
                        Path to the RealFast SDM (overrides CAPO setting)
  -p PNG_PATH, --png_path PNG_PATH
                        Path to the candidate PNG files (overrides CAPO setting)
mcilroy$ 

The two path arguments are provided for flexibility, but it is assumed that the default values in the CAPO profiles (dsoc-test, dsoc-prod/nmprod) are the typical location.  The command can then be invoked with (for example): 

realfastIngest realfast_18B-320.sb38241161.eb38244520.59002.47251115741_1591099113820

This will initiate the process, and the SDM will shortly be available in the UI (a matter of ~10 minutes).  

The workflow will gather all the materials (SDM, PNG, files required for full ingestion) in once place (realfastStagePath), and initiate ingestion upon those files.   After successful ingestion of the metadata, and (if requested) of the files into NGAS, the workflow will trigger a reindex of the project.  

There are a set of values in the CAPO profiles for use with this workflow: defaults, and potential issues with changing them

edu.nrao.archive.workflow.config.collection.RealfastSettings.serviceUrl = https://webtest.aoc.nrao.edu/archiveServices/
#
edu.nrao.archive.workflow.config.collection.RealfastSettings.pngNameArgument = realfast_ancillaries?path=
edu.nrao.archive.workflow.config.collection.RealfastSettings.donorLocatorArgument = realfast_associate?path=
edu.nrao.archive.workflow.config.collection.RealfastSettings.collectionMetadataArgument = realfast_collection?path=
#
edu.nrao.archive.workflow.config.collection.RealfastSettings.ingestNGAS = false
edu.nrao.archive.workflow.config.collection.RealfastSettings.realfastStagePath = /lustre/aoc/cluster/pipeline/nmtest/stage_products
#
edu.nrao.archive.workflow.config.collection.RealfastSettings.realfastSdmPath = /lustre/aoc/sciops/pdemores/realfast_sdms
edu.nrao.archive.workflow.config.collection.RealfastSettings.realfastPngPath = /lustre/aoc/sciops/pdemores/realfast_sdms

Under The Hood

What the workflow does in more detail: 

1. Obtain the PNG file's partial name 
   1. (via service which reads the JSON under Annotation.xml)
2. Find and link the required PNG file into a subdirectory of the staging area
3. Obtain the donor SDM's SPL
   1. (via service which reads the JSON under Annotation.xml & queries the AAT)
4. Obtain the collection metadata 
   1. (via service which reads the JSON under Annotation.xml)
5. Write the collection metadata to a file in the staging area
6. Link the SDM & BDFs into the staging area
7. Write the Ingestion Manifest 
   1. SDM Science Product, with PNG ancillary product
   2. Associate Group with the donor SDM 
8. Prepare ingestion artifacts
9. Trigger ingestion 
   1. Ingestion sends a 'complete' signal upon success