Overview

Ingestion of the IDIFTS files from eLWA observations is a new functionality for the AAT/PPI.   The process is, in a nutshell: 

...

Collection Support: ELWA

Using the System

Assumptions: 

• eLWA team has access to the AAT/PPI command line installation areas (/users/vlapipe/workflows/)
• Vlapipe user (and group) have access (Read, Write, Execute) access.  
• Everyone has (Read) access (to facilitate the AAT/PPI services access) 
• The defined staging area (see below) is on the same filesystem as the data to be ingested

...

usage: elwaIngest [-h] [-P PROFILE] [-p PATH] [-E EMAIL] filename

ELWA IDIFITS Ingestion, version 4.0.1b1: Initiates an ingestion workflow to attach the  provided
    IDIFITS file to its corresponding EVLA EB

positional arguments:
  filename              Filename(s) to ingest

optional arguments:
  -h, --help            show this help message and exit
  -P PROFILE, --profile PROFILE
                        profile name to use, e.g. nmtestdsoc-test, mnproddsoc-prod
  -p PATH, --path PATH  Path to the IDIFITS file to ingeste (overrides CAPO setting)
  -E EMAIL, --email EMAIL
                        Optional email address for a completion message

The two path arguments are provided for flexibility, but it is assumed that the default values in the CAPO profiles (dsoc-test, dsoc-prod) are the typical location.  If those paths are correct, the command can then be invoked with:

...

activate_profile dsoc-prod

realfastIngest elwaIngest buildIDI_TSUB0001_40203068.FITS_1


...

The workflow will stage the file for ingestion and perform some preparatory work.  Then it will call ingestion to set up the metadata and place the files in NGAS (if desired).  There is not currently an external sign of the ingestion (we don't yet display any collection information).  It would be possible to hook the command into a simple feedback utility I've written for other CLIs, so you could get an email about basic success or failure, if that is desired, so I've hooked the utility into a simplistic feedback system I created for another purpose.  You should be provided with the working directory for the ingestion workflow (where the logs for some pieces of the process will show up), and if you provide an email address (-E), you'll get an email about whether the process completed with an error or not.

There are a set of values in the CAPO profiles for use with this workflow: 

#
# ELWA Collection Settings
#
edu.nrao.archive.workflow.config.collection.ElwaSettings.ingestNgas = false
edu.nrao.archive.workflow.config.collection.ElwaSettings.elwaSourcePath = /lustre/aoc/cluster/pipeline/jls_test/elwa
edu.nrao.archive.workflow.config.collection.ElwaSettings.elwaServiceEndpoint = elwa_science_source?path=

Under The Hood

It should be noted that the elwaIngest command isn't doing any processing itself.  It only prepares the basic metadata and initiates the workflow.  It is possible to provide some limited feedback (a working directory name where some log files are kept, and a success/fail email) with a bit of additional work.  

...