NM Operations meetings

Summary of development requests, decisions and actions from SRDP NM Operations meetings July 2019 onwards:

Development requests:

1. The archive search results should indicate the QA standard (SRDP or non-SRDP) as well as whether it is a pass/fail
2. The qa_notes.html file from the weblog should be directly linked in the search results so users can click on it and look at the QA notes before deciding whether to download a dataset or not
   
3. For QA fails that are so bad that no usable calibration products are produced, the qa_fail script (or workspace manager, once that is done) should still ingest the weblog into the archive and make the qa_notes.html file available so users can see why it failed.
4. It would be good if the users could obtain the weblogs from the initial pipeline run, so they can see more easily what was flagged.
5. Ability to trigger restores on more than one dataset at a time.
   
   Items 1-3 sent to John T on 2nd August 2019

Decisions:

• 8/29/19: expand SRDP pilot to include X-band (in addition to Ku-band and higher).
• 10/23/19: use the NM User Support Calendar to record vacations etc.
• 11/20/19: flag outlying A-config antennas in wrong configuration for SRDP purposes once we are in the "official" D-config (otherwise risk trying to calibrate resolved calibrators).
• 12/11/19: AoD review for non-problematic datasets is performed only at the final stage, before archiving. DAs should flag and rerun as they see fit beforehand (but should also feel free to set a dataset to "Ready for AOD" if they have any questions prior to that).
• 2/26/20: add informational note to qa_notes.html if flagging percentage >50%. Decision of SRDP Pass/fail for heavily flagged datasets is at the discretion of the AoD on a case-by-case basis.
• 5/27/20: On the deployment of 3.8.2 with the MS flagging fix, Drew will update the email sent to non-SRDP PIs to include a suggestion that they can try archive-new for their restores/downloads. The ability of non-staff users to download directly to AOC lustre will be discussed at the next SRDP-DMS meeting. We will attapet to ingest calibration files for datasets that can only be run manually due to memory issues (only ~10/yr).
• 8/26/20 For runs that fail the pipeline (both regular CIPL and SRDP), if the reason for a pipeline fail is clear, the email sent to the PI will state this. This is especially important in cases relating to setup (poor calibrator choice etc). For SRDP, if the reason for the pipeline failure is not clear, a short investigation will be performed and the results communicated in the email to the PI.  
• 8/26/20 Plan for VLA imaging testing and implementation - at the end of November after VLASS 2.1 ends, try to send feedback to get any changes to the pipeline into a spring pipeline patch. Aim to start production when array moves to C-config (June 2021)
• 1/27/21 3C138 variability:
  • datasets on-hold to be delivered as-is with 3C48-style warning: "The flux density scale calibrator 3C 138 has been undergoing a flare since at least mid-December 2020.  At K and Ka-bands the magnitude of the flare is of order 40-50% compared to the Perley-Butler 2017 flux scale.  The effect is smaller at lower frequencies (10-20% at C and X-band),  and is larger at higher frequencies (more than a factor of two at Q-band).  If you care about the flux density scale of your observations at that level, monitoring datasets are publicly available in the archive under project code TCAL0009. From these observations, you may derive an updated flux density ratio to use for your observations." If you care about the flux density scale of your observations at that level, you may want to re-calibrate your data once new time-variable calibrator models have been put into CASA."  (change requested by NMOps). Both 3C138 and 3C48 warnings are now consistent.
  • For both 3C48 and 3C138, once the calibrator group have a page indicating how these calibrators have varied since Perley-Butler17 we will include data with these flux calibrators in SRDP, either linking to the flux variation page from the email, or applying the flux corrections via a script and probably remove the statement about time-variable models in CASA as these are still a ways off.
• 2/3/21 Reporting VLA data issues:  I set up a confluence page: https://open-confluence.nrao.edu/display/SRDP/VLA+data+issues. On that page are links (well, just one link for now) to SRDP JIRA tickets (under the research request category) describing the issues, with the idea being that if/when we find more examples of a problem we can add comments. (The idea behind having them as JIRA tickets rather than confluence pages is that when they either do not show up again, or the issue is fixed on the telescope the tickets can be closed out and won't hang around forever.) So the procedure is that when one of these issues is found, an AOD makes a new SRDP ticket detailing the issue (dataset, preferably a plot etc if that makes sense) (with label: VLA-data-issues) then puts the other AODs and SRDP DAs as watchers. If a DA finds the problem in another dataset, they can update the ticket with the dataset name and other details as they see fit.
• 4/21/21 Changing intents - we will not change intents in an EB to get it to pass SRDP, but instead add a note in the email to the user that changing/removing an intent may allow the pipeline to run correctly.
• 9/1/21 Resolved phase calibrators - if < 5% of the calibrator flux is resolved, it can still pass SRDP, but please add a comment to the qa_notes if there is evidence of resolved flux. If >5% note in the qa_notes or spreadsheet for the AOD to check.
• 4/25/22 super long sol int
• 4/25/22 compression
• 4/25/22 amp ramps due to model
• 5/05/22 missing fluxboot solutions for pol cal

Ramp-up schedule FY21

• 2020 Nov 20 (start of A-config): add C-band calibration to SRDP.
• 2021 Jan 18: add Multiband calibration to SRDP (down to C-band, including S and L later (for EBs unaffected by gain compression); not including P-band).
• 2021 Mar 5 (start of D-config): begin testing imaging pipeline and workflows (*dependency on SSA to provide scripts to process and ingest).
• 2021 Jun 10 (start of C-config): add S-band calibration to SRDP.
• 2021 Aug 15: begin running imaging as pilot on a small fraction of SRDP datasets.
• (2022 Mar 4 (start of A-config): add L-band calibration to SRDP.)

SSA tickets important for NM Ops:

SSA-6129 - Getting issue details... STATUS SDM files no longer have .loading appended - Fix version: unassigned; write requirement, design solution, determine priority

SSA-6495 - Getting issue details... STATUS

NMS software page

https://staff.nrao.edu/wiki/bin/view/DMS/NMSSoftwareReleases