Wrapper
A wrapper script like wrapper_zen.2458098.44615.HH.uvh5.EXTRACT_AUTOS.sh, is created by build_makeflow_from_config.py and called by makeflow_htcondor.sh idr2_2.mf. I see it having the following issues:
It runs conda activate base
Use a python-packages tarball instead of conda on the execution hosts.
Remove source ~/.bashrc
Remove conda activate base
Add wget http://proxy.chtc.wisc.edu/SQUID/chtc/python37.tar.gz
Add tar xfz python37.tar.gz
- Add (cd python bin ; ln -s python3 python) so the default python is python3
Add tar xfz hera_calibration_packages.tar.gz
Add export PYTHONPATH=${PWD}/hera_calibration_packages
Add export PATH=${PWD}/python/bin:${PATH}
- Add rm -f python37.tar.gz to the end of the script
It changes directory to /lustre/aoc/projects/hera/krowe/makeflow_sample/raw_data
This line should just be removed
It runs /lustre/aoc/projects/hera/krowe/hera_opm/pipelines/h1c/idr2/v2/task_scripts/do_EXTRACT_AUTOS.sh zen.2458098.44615.HH.uvh5
Instead run ./do_EXTRACT_AUTOS.sh zen.2458098.44615.HH.uvh5 zen.2458098.44615.HH.uvh5
zen.2458098.44615.HH.uvh5 will need to be copied to the scratch area on the execution host.
- We could just make it a dependency in the .mf file, but it is too large for HTCondor's transfer mechanism.
- http://chtc.cs.wisc.edu/file-avail-largedata
do_EXTRACT_AUTOS.sh requires ${src_dir}/_common.sh
Copy ${src_dir}/_common.sh to the working directory and make it a dependency in the .mf file
do_EXTRACT_AUTOS.sh calls extract_autos.py which is from hera_cal.git
- Copy extract_autos.py to the working directory and make it a dependency in the .mf file
- Copy extract_autos.py to the working directory and make it a dependency in the .mf file
On success it touches the .out file
- This is unnecessary. The .out file can just be the target for the rule in the .mf file
- This is unnecessary. The .out file can just be the target for the rule in the .mf file
On error it moves the .log file to .EXTRACT_AUTOS.log.error
- Is this really necessary?
- Is this really necessary?