The DSOC cluster will be offline October 28 and 29 for lustre improvements and to switch the cluster from RHEL6 to RHEL7, this document enumerates the steps we will take beforehand to prepare for it and the steps we will take afterwords to confirm things are working properly and restore access to it. Note that these upgrades won't touch the control systems for the AAT/PPI or VLASS, but they will touch the environment the workflows for both execute on.
1 Week Before the Shutdown (October 21)
- Stephan Witz to work with CIS to make sure the DNS TTLs for archive.nrao.edu and archive-new.nrao.edu are low
- Stephan Witz (SSA) to put MOTD banners up on the legacy archive announcing the downtime, will work with John Tobin on the messaging
- Stephan Witz (SSA) to work with CIS to replace the message on http://offline.nrao.edu, with work with John Tobin on the messaging
Morning of the Shutdown (October 28)
- CIS to change external DNS of archive.nrao.edu and archive-new.nrao.edu to point to offline.nrao.edu
- SSA to change the casa CAPO properties from the RHEL6 paths to RHEL7 paths
Morning After the Shutdown (October 30)
- Stakeholders (John Tobin, Mark Lacy ) test critical user-facing functions of the AAT/PPI under RHEL7
- User-facing things that need to be tested:
- Downloads of VLA EBs: SDM-only, SDM, basic MS, CMS
- Downloads of VLA calibrations
- Downloads of VLASS images
- Downloads of VLBA UVFits files
- Stakeholders (Mark Lacy, Drew Medlin) test critical operations-facing functions of the AAT/PPI under RHEL7
- Operations-facing things that need to be tested:
- EB ingestion
- CIPL being triggered
- CIPL working
- calibration ingestion (QAPass)
- GO/NO decision:
- If GO: Stephan Witzto with with CIS to undo the DNS change and MOTD banners, re-nable CIPL triggers
- If NO-GO: SSA to iterate with stakeholders until result is GO, note that this may mean putting out a bugfix release of AAT/PPI 3.6 and doing the same for VLASS