SSA-6931 - Getting issue details... STATUS

(Link to PIMS/SECI Testing Plan Confluence page)

Introduction

PIMSCache is a tool for managing a cache of measurement sets, it includes commands for adding a PIMS to the cache, listing whether a PIMS is in the cache, removing a PIMS and so on. This document covers setting it up and executing it.

Words and Their Meanings

  • CAPO Profile: this is an environment variable or command line parameter that specifies the configuration profile to use. The CAPO Profiles we have right now include dsoc-prod, dsoc-test, dsoc-dev, naasc-prod, naasc-test, naasc-dev, vlass.prod, vlass.test and vlass.dev.
  • PIMS: a per-image measurement set, this is a calibrated measurement set that has been split into a smaller size to speed up imaging. 
  • PIMSCache: a system for managing a cache of PIMS, also the name of the command line tool for doing the same (pimscache).
  • workflows: each CAPO Profile has a corresponding software installation area, collectively these are called the workflows and live in the vlapipe account, in the workflows subdirectory. By 'setting up the workflows' you prepare your environment to activate and use a specific CAPO Profile's workflow.
  • virtualenv (virtual environment): each workflow's installation area is a virtualenv, or virtual environment. This is a way to install Python software so that its required modules are independent of any system installed Python modules.

Requirements for Using PIMSCache

  • The user must be logged into an NRAO workstation, server or a cluster node at the DSOC (any of which can see the /lustre/aoc and /users/vlapipe directories). Laptops are generally not suitable, nor are macs, though you can use these to log into a workstation, server or cluster node and run the commands there.
  • The user must be in the vlapipe user group.
  • To see if you are in the vlapipe group, type groups; this should list all of the user groups your account is in: if "vlapipe" isn't in that list, contact the helpdesk. Below, I've logged into a cluster node that I reserved (nmpost023) and run the groups command, which shows I'm in the "vlapipe" user's group.
  • As of WS 2.7, you must run pimscache as user "vlapipe".

PIMSCache Commands

PIMScache Usage

Running pimscache -h produces a message that describes its overall usage:


(vlass.d3) vlapipe@hamilton$ pimscache --help
usage: pimscache [-h] {split,ls,rm,lspc,cp,ln} ...

The Per-Image Measurement Set (PIMS) Cache system.

This tool is one-stop shopping for all your PIMS cache maintenance needs.

Actions for data analysts:

- split: use this to create PIMS from a restore
- ls: use this to investigate what PIMS are in the cache
- rm: use this to delete PIMS from the cache
- lspc: use this to see what phase centers exist under a tile

Each of these actions has additional arguments you may need to specify.
To get the help on a specific action, specify -h after the action.

The remaining actions are used by the workflow system and can be safely ignored:

- cp: workflows use this to insert a PIMS into the cache
- ln: workflows use this to retrieve a PIMS from the cache

positional arguments:
{split,ls,rm,lspc,cp,ln}
split split to per-image measurement sets (for DAs)
ls list the contents of the cache (for DAs)
rm remove PIMS from cache (for workflows and DAs)
lspc list phase centers under a given tile (for DAs)
cp copy PIMS to cache (for workflows)
ln link phase center from the cache into the target
directory (for workflows)

optional arguments:
-h, --help show this help message and exit


You can get further help on the commands by typing pimscache $COMMAND -h, where $COMMAND is one of split, ls, rm, lspc. For example, pimscache split -h yields:


(vlass.d3) vlapipe@hamilton$ pimscache split -h
usage: pimscache split [-h] [-v VERSION-ID | -c CAL] [-p DEC+RA] [-t TILE]
                       [-r PATH-TO-RESTORE] [--casa-home CASA_HOME]

optional arguments:
  -h, --help            show this help message and exit
  -v VERSION-ID, --version_id VERSION-ID
                        The version ID (jobspec ID) whose EB we are checking
                        for (use this OR execblock and calibration)
  -c CAL, --calibration CAL
                        The calibration we are checking for (use this OR
                        version ID), something like
                        VLASS2.1_T09t34.T09t01.T09t04.T09t07_P56724v1
  -p DEC+RA, --phase-center DEC+RA
                        Right ascension/declination coordinate of a phase
                        center
  -t TILE, --tile TILE  Name of a tile (e.g. T01t01)
  -r PATH-TO-RESTORE, --existing_restore PATH-TO-RESTORE
                        Path to an existing restore to use instead of
                        restoring from scratch
  --casa-home CASA_HOME
                        CASA installation directory; defaults to the
                        reprocessing home



Setup for BASH Users

Step One: Log into an NRAO Workstation, Server or Reserved Cluster Node

Step Two: Setup the Workflows

source ~vlapipe/workflows/setup_workflows.sh


Step Three: Activate Your Chosen Workflow's Profile

activate_profile vlass.test


Notice that your prompt has changed, indicating the active profile, but vlass.t3 wasn't what you typed: each of the VLASS CAPO Profiles (vlass.prod, vlass.test or vlass.dev) has one of three different CAPO Profiles associated with it: for example, vlass.test has vlass.t1, vlass.t2 and vlass.t3. vlass.test is a shortcut that points at the currently executing profile, vlass.t3.

At this point the pimscache command should be available to your account, test that with:

which pimscache

If it says no such command or produces an error, something is wrong, seek help. If not, when you are done, either deactivate_profile or just log out.

Setup for TCSH/CSH Users

Step One: Log Into an NRAO Workstation, Server or Reserved Cluster Node

Here I also switched over to tcsh, but a user whose chosen shell is tcsh would not have to do this step:

Step Two: Set the CAPO_PROFILE Environment Variable

Here I set the CAPO_PROFILE environment variable to the profile I wish to use, vlass.test for now, and vlass.prod once pimscache is in production:

setenv CAPO_PROFILE vlass.test

Step Three: Activate Your Chosen Workflow's Virtual Environment

Here I activate the virtual environment by 'source'ing a file in ~vlapipe/workflows/$CAPO_PROFILE/bin:

source ~vlapipe/workflows/vlass.test/bin/activate.csh

Lastly, I test to make sure pimscache is now in my PATH, as I would expect it to be:


If the command had come back with something like 'command not found', something is wrong. When I'm done using pimscache I use the deactivate command to switch off the workflow's virtual environment, or I log out.

  • No labels