VLA Pipeline

VLA 2024 Priorities

Highest Priority:

Pipeline will crash if one calibration source gets fully flagged by flagtemplate or other mechanism PIPE-1729 - Getting issue details... STATUS
Use near-field RMS to adjust auto-masking noise threshold ( PIPE-1878 - Getting issue details... STATUS )
Spectral line stuff:
- Task to create VLA _line.ms ( PIPE-1414 - Getting issue details... STATUS )
- Ability to use _line.ms and cont.ms in different pipeline tasks. ( PIPE-1411 - Getting issue details... STATUS )
- Detection Spectral windows for spectral line processing ( PIPE-1412 - Getting issue details... STATUS )
- Spectral line calibration recipe definition ( PIPE-1344 - Getting issue details... STATUS )
- Zoom-in plots on non-128 MHz windows for inspection of bandpass in finalcals, semifinalBPdcals,testBPdcals, plotsummary for targets and cals ( PIPE-1413 - Getting issue details... STATUS ) Less critical than other items, but still desired.
- Enable basic cube imaging ( PIPE-1346 - Getting issue details... STATUS )
- Use MS tool hanning smoothing to enable per spw hanning smoothing (CASR-523; PIPE-672 - Getting issue details... STATUS )
- include additional flag versions in exported *.ms.flagversions.tgz ( PIPE-1553 - Getting issue details... STATUS )
- Add task to casa_piperestore script to support spectral line restores ( PIPE-2020 - Getting issue details... STATUS )
- Agg cont adjustments to exclude spectral line windows (unless only spectral line) ( PIPE-2132 - Getting issue details... STATUS )
Different bandpass calibrators in different bands - ( PIPE-669 - Getting issue details... STATUS )
Option to disable bad deformatter flagging completely or partially, ( PIPE-1183 - Getting issue details... STATUS )
Detect mosaicked fields and perform joint deconvolution with mosaic gridder ( PIPE-684 - Getting issue details... STATUS )
Manual and automatic refant ignore do not work together ( PIPE-1637 - Getting issue details... STATUS )
Enable time limits for antenna position updates PIPE-2052 - Getting issue details... STATUS

High Priority:

Image scaling and zooming ( PIPE-2019 - Getting issue details... STATUS and PIPE-1083 - Getting issue details... STATUS )
adjust theoretical sensitivity for Hanning smoothing and report theoretical sensitivity for VLA ( PIPE-2131 - Getting issue details... STATUS )
access to nmajor parameter and setting a default ( PIPE-2018 - Getting issue details... STATUS )
list number of major cycles in VLA imaging pipeline weblog ( PIPE-2039 - Getting issue details... STATUS )
flux calibrator RFI flagging should use residuals - ( PIPE-1274 - Getting issue details... STATUS )
hifv_restoredata should use flagversion backed up before statwt ( PIPE-1555 - Getting issue details... STATUS )
TestBPdcals uses wrong integration time multipliers in multi-band data ( PIPE-1703 - Getting issue details... STATUS PIPE-1586 - Getting issue details... STATUS )
Checkflag should properly execute when sources have mixed intents PIPE-1335 - Getting issue details... STATUS
PIPE-1554 - Getting issue details... STATUS
PIPE-986 - Getting issue details... STATUS
Aqua report PIPE-730 - Getting issue details... STATUS
Enhanced DTS issue identification and flagging ( PIPE-986 - Getting issue details... STATUS )
Fluxboot QA metric adjustments ( PIPE-989 - Getting issue details... STATUS )
fitorder=0 heuristic for fluxboot ( PIPE-1603 - Getting issue details... STATUS )
Abbreviate all image header strings in CASA images otherwise they will not load in CARTA ( PIPE-1615 - Getting issue details... STATUS )
Add options to turn off application of gaintables in hifv_priorcals ( PIPE-1665 - Getting issue details... STATUS )
PIPE-988 - Getting issue details... STATUS

Medium priority:

PIPE-1758 - Getting issue details... STATUS
PIPE-1737 - Getting issue details... STATUS
PIPE-1873 - Getting issue details... STATUS
PIPE-1905 - Getting issue details... STATUS
Only flag offending polarization for DTS issues PIPE-1435 - Getting issue details... STATUS
limit_short_solint='int' uses 0.0s ( PIPE-1597 - Getting issue details... STATUS )
remove hifv_targetflag and add its heuristic to hifv_checkflag ( PIPE-1532 - Getting issue details... STATUS )
PIPE-595 - Getting issue details... STATUS
PIPE-1755 - Getting issue details... STATUS

Low Priority:

Don't show setup Intent scans as calibrators ( PIPE-1711 - Getting issue details... STATUS )
when spws are in reverse frequency order, print them in proper frequency order ( PIPE-1589 - Getting issue details... STATUS
plot labels in semifinalbpdcals (PIPE-984)
parallactic angle vs time plot (PIPE-836)
Add API plot to VLA pipeline runs (PIPE-580)
hifv_circfeedpolcal should not add the KCROSS talbe to cal library after second call ( PIPE-1534 - Getting issue details... STATUS
hifv_circfeedpolcal should include CALIBRATE_POL_ANGLE intent for KCROSS solutions ( PIPE-1533 - Getting issue details... STATUS
PIPE-1406 - Getting issue details... STATUS
PIPE-1405 - Getting issue details... STATUS

VLA 2023 Priorities

Blocker:

Per-band manipulation of apply, antexclude, and usemedian for hifv_syspower ( PIPE-1487 - Getting issue details... STATUS )
Detect mosaicked fields and perform joint deconvolution with mosaic gridder ( PIPE-684 - Getting issue details... STATUS )
Use wproject gridder for single-field L and S-band imaging PIPE-1641 - Getting issue details... STATUS
Option to disable bad deformatter flagging completely or partially, ( PIPE-1183 - Getting issue details... STATUS )
Antenna position corrections missed if Bx,By=0 and Bz !=0 ( PIPE-1726 - Getting issue details... STATUS

Critical:

Different bandpass calibrators in different bands - ( PIPE-669 - Getting issue details... STATUS )
flux calibrator RFI flagging should use residuals - ( PIPE-1274 - Getting issue details... STATUS )
include additional flag versions in exported *.ms.flagversions.tgz ( PIPE-1553 - Getting issue details... STATUS )
hifv_restoredata should use flagversion backed up before statwt ( PIPE-1555 - Getting issue details... STATUS )
Checkflag should properly execute when sources have mixed intents PIPE-1335 - Getting issue details... STATUS
PIPE-1554 - Getting issue details... STATUS
PIPE-986 - Getting issue details... STATUS
Enhanced DTS issue identification and flagging ( PIPE-986 - Getting issue details... STATUS )
Fluxboot QA metric adjustments ( PIPE-989 - Getting issue details... STATUS )
fitorder=0 heuristic for fluxboot ( PIPE-1603 - Getting issue details... STATUS )
Allow hifv_syspower tot run on K, Ka, and Q-band data ( PIPE-1705 - Getting issue details... STATUS )
TestBPdcals uses wrong integration time multipliers in multi-band data ( PIPE-1703 - Getting issue details... STATUS PIPE-1586 - Getting issue details... STATUS )
Abbreviate all image header strings in CASA images otherwise they will not load in CARTA ( PIPE-1615 - Getting issue details... STATUS )
Add options to turn off application of gaintables in hifv_priorcals ( PIPE-1665 - Getting issue details... STATUS )
Pipeline will crash if one calibration source gets fully flagged by flagtemplate or other mechanism PIPE-1729 - Getting issue details... STATUS
Flux.csv uvmax is ignored if uvmin is 0 ( PIPE-1924 - Getting issue details... STATUS )

Major:

PIPE-1758 - Getting issue details... STATUS
Aqua report PIPE-730 - Getting issue details... STATUS
PIPE-1737 - Getting issue details... STATUS
Spectral line stuff:
- Task to create VLA _line.ms ( PIPE-1414 - Getting issue details... STATUS )
- Ability to use _line.ms and cont.ms in different pipeline tasks. ( PIPE-1411 - Getting issue details... STATUS )
- Detection Spectral windows for spectral line processing ( PIPE-1412 - Getting issue details... STATUS )
- Spectral line calibration recipe definition ( PIPE-1344 - Getting issue details... STATUS )
- Zoom-in plots on non-128 MHz windows for inspection of bandpass in finalcals, semifinalBPdcals,testBPdcals, plotsummary for targets and cals ( PIPE-1413 - Getting issue details... STATUS ) Less critical than other items, but still desired.
- Enable basic cube imaging ( PIPE-1346 - Getting issue details... STATUS )
- Use MS tool hanning smoothing to enable per spw hanning smoothing (CASR-523; PIPE-672 - Getting issue details... STATUS )
Only flag offending polarization for DTS issues PIPE-1435 - Getting issue details... STATUS
limit_short_solint='int' uses 0.0s ( PIPE-1597 - Getting issue details... STATUS )
remove hifv_targetflag and add its heuristic to hifv_checkflag ( PIPE-1532 - Getting issue details... STATUS )
PIPE-595 - Getting issue details... STATUS
PIPE-1755 - Getting issue details... STATUS

Minor:

Don't show setup Intent scans as calibrators ( PIPE-1711 - Getting issue details... STATUS )
when spws are in reverse frequency order, print them in proper frequency order ( PIPE-1589 - Getting issue details... STATUS
plot labels in semifinalbpdcals (PIPE-984)
parallactic angle vs time plot (PIPE-836)
Add API plot to VLA pipeline runs (PIPE-580)
hifv_circfeedpolcal should not add the KCROSS talbe to cal library after second call ( PIPE-1534 - Getting issue details... STATUS
hifv_circfeedpolcal should include CALIBRATE_POL_ANGLE intent for KCROSS solutions ( PIPE-1533 - Getting issue details... STATUS
PIPE-1406 - Getting issue details... STATUS
PIPE-1405 - Getting issue details... STATUS

VLASS 2022 Priorities

Cube Imaging pipeline PIPEREQ-142
~~Resolved calibrator handling PIPEREQ-132~~ Deferred
Imaging fails in some cases in 2021.2 ( PIPE-1173 - Getting issue details... STATUS )
Change VLASS parameter compressionusemedian default from False to True ( PIPE-1338 - Getting issue details... STATUS )
Change hifv_targetflag to flag target data using a .flagtargetstemplate.txt - (~~PIPE-37~~, PIPE-1355 - Getting issue details... STATUS )
Add reimaging_resources.tgz to manifest.xml ( PIPE-1407 - Getting issue details... STATUS )

VLA 2022 Priorities

Prioritized list:

Blocker:

Add plots to statwt weblog in PI pipeline( PIPE-1341 - Getting issue details... STATUS )
Flagging Heuristics on outlier weights (ticket to be created)
Generalized Compression correction for S-band and L-band ( PIPE-1155 - Getting issue details... STATUS / PIPEREQ-61 - Getting issue details... STATUS )
TEC maps fixes and behavior changes ( PIPE-1136 - Getting issue details... STATUS , PIPE-1365 - Getting issue details... STATUS )

Critical:

Backup flags in hifv_applycal prior to calibration ( PIPE-854 - Getting issue details... STATUS )
Enable self-calibration heuristics for continuum data only in new task (hif_selfcal) and operate on only the _cont.ms ( PIPE-1445 - Getting issue details... STATUS )
Spectral line stuff:
- Task to create VLA _line.ms ( PIPE-1414 - Getting issue details... STATUS )
- Ability to use _line.ms and cont.ms in different pipeline tasks. ( PIPE-1411 - Getting issue details... STATUS )
- Detection Spectral windows for spectral line processing ( PIPE-1412 - Getting issue details... STATUS )
- Spectral line calibration recipe definition ( PIPE-1344 - Getting issue details... STATUS )
- Zoom-in plots on non-128 MHz windows for inspection of bandpass in finalcals, semifinalBPdcals,testBPdcals, plotsummary for targets and cals ( PIPE-1413 - Getting issue details... STATUS ) Less critical than other items, but still desired.
- Enable basic cube imaging ( PIPE-1346 - Getting issue details... STATUS )
combine targetflag heuristics into checkflag - ( PIPE-1342 - Getting issue details... STATUS )
Change hifv_targetflag to flag target data using a .flagtargetstemplate.txt - (~~PIPE-37~~, PIPE-1355 - Getting issue details... STATUS )
flux calibrator RFI flagging should use residuals - ( PIPE-1274 - Getting issue details... STATUS )
Different bandpass calibrators in different bands - ( PIPE-669 - Getting issue details... STATUS )

Major:

Spectral Line syspower ( PIPE-1496 - Getting issue details... STATUS )
Detect mosaicked fields and perform joint deconvolution with mosaic gridder (PIPE-684)
wproject and n- w-planes heuristics for VLA imaging pipeline (J. Marvil input)
PIPE-1287 - fluxboot failing with target intent using pointing and a CG intent with the pointing setup ( PIPE-1287 - Getting issue details... STATUS )
PIPE-1350 - fluxboot shared intents ( PIPE-1350 - Getting issue details... STATUS )
PIPE-1335 - selection heuristics when delay calibrator mixed with CG intent
Long solution interval calculation improvement (PIPE-1002/PIPEREQ-64)
Short Solution interval calculation improvments (PIPE-1340)
Spws not specified in cont.dat should be handled as standard continuum (PIPE-1043)
Use MS tool hanning smoothing to enable per spw hanning smoothing (CASR-523; PIPE-672)
per-baseline RFI flagging - (PIPE-1286)
Fluxboot QA metric adjustments (PIPEREQ-8/PIPE-989)

Minor:

long solints: ( PIPE-1473 - Getting issue details... STATUS )
solint calc: ( PIPE-1002 - Getting issue details... STATUS )
Zoom feature for all hif_makeimages plots (PIPE-1083)
plot labels in semifinalbpdcals (PIPE-984)
parallactic angle vs time plot (PIPE-836)
Add API plot to VLA pipeline runs (PIPE-580)
Report total MS sizes at end of calibration on main weblog page (PIPE-1166)

multi-band weblog plots should not use shared time axis for plots (PIPE-1044)
Zeros flagged not reported properly PIPE-1033 - Getting issue details... STATUS )
Make an untarred MS when making a basic MS PIPE-749
PIPE-1098 - cont.dat failures for targets with some characters

Backburner:

Per-baseband delay solutions (PIPE-670)
Spectral window mapping for gain calibration (PIPE-671)
Antenna position corrections in a post-observation time window (CASR-556)
Gain table flagging extensions
- Ripple detection (bad switch problem) (PIPEREQ-11)
- Gain outlier detection (PIPEREQ-10)
Enhanced DTS issue identification and flagging (CASR-563/PIPE-986)
- Option to disable bad deformatter flagging completely or partially, (PIPE-1183)
Generalize hifv_circfeedpolcal for all bands
Enable experimental P-band calibration
Define requirements for the continuum imaging pipeline to use outlier fields to remove effects of bright sources rather than using large images
RFI flagging on uv-grid (CASR-496)
- avoids flagging of bright spectral lines
Enable data weights to be determined from switched power (CASR-476)
Seasonal model selection to override weather station in priorcals to compensate for bad weather station data (PIPE-1367)
Add Operator log to weblog (PIPE-1365)

VLA 2022 Categorized:

Bugs

PIPE-1287 - fluxboot failing with target intent using pointing and a CG intent with the pointing setup ( PIPE-1287 - Getting issue details... STATUS )
PIPE-1335 - selection heuristics when delay calibrator mixed with CG intent
PIPE-1098 - cont.dat failures for targets with some characters

Calibration and Flagging

Enhanced DTS issue identification and flagging (CASR-563/PIPE-986)
- Option to disable bad deformatter flagging completely or partially, (PIPE-1183)
Possible RFI flagging refinements
- per baseline flagging - (PIPE-1286)
- combine targetflag heuristics into checkflag - (PIPE-1342)
- flux calibrator RFI flagging should use residuals - (PIPE-1274)
Backup flags in hifv_applycal prior to calibration (PIPE-854)
Change hifv_targetflag to flag target data using a .flagtargetstemplate.txt - (~~PIPE-37~~, PIPE-1355 - Getting issue details... STATUS )
Add plots to statwt weblog in PI pipeline (PIPE-1341)
Long solution interval calculation improvement ( PIPE-1002 - Getting issue details... STATUS / PIPEREQ-64 - Getting issue details... STATUS )
Short Solution interval calculation improvments (PIPE-1340)
Per-baseband delay solutions (PIPE-670)
Spectral window mapping for gain calibration (PIPE-671)
- enables phase calibration if not enough S/N due to faint calibrator
Antenna position corrections in a post-observation time window (CASR-556)
Gain table flagging extensions
- Ripple detection (bad switch problem) (PIPEREQ-11)
- Gain outlier detection (PIPEREQ-10)
Fluxboot gain solution flagging (PIPEREQ-9)
Fluxboot QA metric adjustments (PIPEREQ-8)

Self-calibration

Enable self-calibration heuristics for continuum data only in new task (hif_selfcal) and operate on only the _cont.ms (PIPE-)
- Joint ALMA and VLA development and should work on both types of data
- SRDP heuristics team input

Spectral lines

Ability to use _line and cont.ms in different pipeline tasks.
Spectral line calibration recipe definition (PIPE-1344)
- Save pre-target RFI flagging and split non-RFI flagged data after hifv_statwt using a new task (PIPE-1343)
For VLA data, enable steps to run findcont to only run on line spws
- hif_makeimlist(specmode='mfs')
- hif_findcont()
- hif_uvcontfit()
- hif_uvcontsub()
Spectral window mapping for gain calibration (PIPE-671)
- enables phase calibration if not enough S/N on narrow spws
Use MS tool hanning smoothing to enable per spw hanning smoothing (CASR-523; PIPE-672)
Zoom-in plots on non-128 MHz windows for inspection of bandpass in finalcals, semifinalBPdcals,testBPdcals
Cont.dat interpretation to treat non-specified spws as continuum-only (PIPE-1043)

Weblog

Zoom feature for all hif_makeimages plots (PIPE-1083)
plot labels in semifinalbpdcals (PIPE-984)
parallactic angle vs time plot (PIPE-836)
Add API plot to VLA pipeline runs (PIPE-580)

Minor things

Report total MS sizes at end of calibration on main weblog page (PIPE-1166)
multi-band weblog plots should not use shared time axis for plots (PIPE-1044)
Zeros flagged not reported properly (PIPE-1033)
Make an untarred MS when making a basic MS PIPE-749

Imaging

Detect mosaicked fields and perform joint deconvolution with mosaic gridder (PIPE-684)
Enable basic cube imaging (PIPE-1346)
wproject and n- w-planes heuristics for VLA imaging pipeline (J. Marvil input)

Backburner (CASA development or low priority):

Generalize hifv_circfeedpolcal for all bands
Enable experimental P-band calibration
Define requirements for the continuum imaging pipeline to use outlier fields to remove effects of bright sources rather than using large images
RFI flagging on uv-grid (CASR-496)
- avoids flagging of bright spectral lines
Enable data weights to be determined from switched power (CASR-476)

2021 Priorities

Primary Development Priorities