This is a proof of concept for running HERA calibration jobs at CHTC using HTCondor and no shared filesystem.
First, follow the Installing HERA Software on Submit Host instructions. Then, make the following modifications
Mar. 26, 2020 krowe: I was able to run this through HTCondor at CHTC via makeflow -T condor small.mf
...
---
> wget --no-verbose http://proxy.chtc.wisc.edu/SQUID/chtc/python37.tar.gz
> tar xfz python37.tar.gz
> tar xfz hera_calibration_packages.tar.gz
> date
> export PYTHONPATH=${PWD}/hera_calibration_packages
> export PATH=.:${PWD}/python/bin:${PATH}
>
> #CHTC's python37 tarball has bin/python3 and not bin/pyton
> (cd python/bin ; ln -s python3 python)
>
> #cd /lustre/aoc/projects/hera/krowe/makeflow_sample/raw_data
>
> timeout 24h ./do_EXTRACT_AUTOS.sh zen.2458098.44615.HH.uvh5
>
> # Perhaps wait and see where makeflow/condor put output and error.
> #if [ $? -eq 0 ]; then
> # cd /lustre/aoc/projects/hera/krowe/makeflow_sample/makeflow
> # touch zen.2458098.44615.HH.uvh5.EXTRACT_AUTOS.out
> #else
> # mv /lustre/aoc/projects/hera/krowe/makeflow_sample/makeflow/zen.2458098.44615.HH.uvh5.EXTRACT_AUTOS.log /lustre/aoc/projects/hera/krowe/makeflow_sample/makeflow/zen.2458098.44615.HH.uvh5.EXTRACT_AUTOS.log.error
> #fi
> # delete some files so they are not transferred back
> rm -f python37.tar.gz
extract_autos.py
This script was installed as part of hera_cal.git in the conda environment so it assumes it knows where python is. But, since this needs to run on the execution host, we need to change it to run the python37 we install via http://chtc.cs.wisc.edu/python-jobs.shtml where we also symlink python to python3.
1c1< #!/home/nu_kscott/hera/share/miniconda3/bin/python
---> #!/usr/bin/env python