Overview

Ingestion of the IDIFTS files from eLWA observations is a new functionality for the AAT/PPI.   The process is, in a nutshell: 

  1. Add the IDIFITS file and the new ingestion artifacts as ancillary products for the linked EB and its Product Group
    1. Create the full ancillary product metadata structure
  2. Update the linked EB to be part of the ELWA collection
  3. Place the new files into NGAS

The details are covered in the following pages: 

SSA-6528 - Getting issue details... STATUS

Collection Support: ELWA

Using the System

Assumptions: 

  • eLWA team has access to the AAT/PPI command line installation areas (/users/vlapipe/workflows/)
  • Vlapipe user (and group) have access (Read, Write, Execute) access.  
  • Everyone has (Read) access (to facilitate the AAT/PPI services access) 
  • The defined staging area (see below) is on the same filesystem as the data to be ingested

The process is contained within a special-purpose workflow, which can be initiated with the 'elwaIngest' command, installed under the vlapipe account.  

NOTE: There are separate installations for the Test and Production systems, both living in the vlapipe account's area.  Be careful to use the one you want. 

CLI Arguments 
usage: elwaIngest [-h] [-P PROFILE] [-p PATH] [-E EMAIL] filename

ELWA IDIFITS Ingestion, version 4.0.1b1: Initiates an ingestion workflow to attach the  provided
    IDIFITS file to its corresponding EVLA EB

positional arguments:
  filename              Filename(s) to ingest

optional arguments:
  -h, --help            show this help message and exit
  -P PROFILE, --profile PROFILE
                        profile name to use, e.g. dsoc-test, dsoc-prod
  -p PATH, --path PATH  Path to the IDIFITS file to ingeste (overrides CAPO setting)
  -E EMAIL, --email EMAIL
                        Optional email address for a completion message

The two path arguments are provided for flexibility, but it is assumed that the default values in the CAPO profiles (dsoc-test, dsoc-prod) are the typical location.  If those paths are correct, the command can then be invoked with:

Testing:

As vlapipe:

activate_profile dsoc-test

elwaIngest buildIDI_TSUB0001_40203068.FITS_1

elwaIngest -p /lustre/aoc/scipos/etc/etc/etc buildIDI_TSUB0001_40203068.FITS_1

Not as vlapipe: 

/users/vlapipe/workflows/dsoc-test/bin/elwaIngest buildIDI_TSUB0001_40203068.FITS_1


Production: 

As vlapipe:

activate_profile dsoc-prod

elwaIngest buildIDI_TSUB0001_40203068.FITS_1

Not as vlapipe: 

/users/vlapipe/workflows/dsoc-prod/bin/elwaIngest buildIDI_TSUB0001_40203068.FITS_1


The workflow will stage the file for ingestion and perform some preparatory work.  Then it will call ingestion to set up the metadata and place the files in NGAS (if desired).  There is not currently an external sign of the ingestion, so I've hooked the utility into a simplistic feedback system I created for another purpose.  You should be provided with the working directory for the ingestion workflow (where the logs for some pieces of the process will show up), and if you provide an email address (-E), you'll get an email about whether the process completed with an error or not.


There are a set of values in the CAPO profiles for use with this workflow: 

ELWA CAPO Settings 
#
# ELWA Collection Settings
#
edu.nrao.archive.workflow.config.collection.ElwaSettings.ingestNgas = false
edu.nrao.archive.workflow.config.collection.ElwaSettings.elwaSourcePath = /lustre/aoc/cluster/pipeline/jls_test/elwa
edu.nrao.archive.workflow.config.collection.ElwaSettings.elwaServiceEndpoint = elwa_science_source?path=


Under The Hood

It should be noted that the elwaIngest command isn't doing any processing itself.  It only prepares the basic metadata and initiates the workflow.  It is possible to provide some limited feedback (a working directory name where some log files are kept, and a success/fail email) with a bit of additional work.  

What the workflow does in more detail: 

  1. Find and link the IDIFTIS file to the staging area (under stage_products directory, named after the file)
  2. Obtain the associated SDM's SPL
    1. (via service which reads the FITS header & queries the AAT)
  3. Write the Ingestion Manifest  & collection metadata file
    1. The IDIFITS file with associated SPL for linking
    2. Collection metadata only consists of the collection's name
  4. Prepare ingestion artifacts
  5. Trigger ingestion 



  • No labels