You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Current »

Wrapper

A wrapper script like wrapper_zen.2458098.44615.HH.uvh5.EXTRACT_AUTOS.sh, is created by build_makeflow_from_config.py and called by makeflow_htcondor.sh idr2_2.mf.  I see it having the following issues:


  • It runs conda activate base

    • Use a python-packages tarball instead of conda on the execution hosts.

      • Remove source ~/.bashrc

      • Remove conda activate base

      • Add wget http://proxy.chtc.wisc.edu/SQUID/chtc/python37.tar.gz

      • Add tar xfz python37.tar.gz

      • Add (cd python bin ; ln -s python3 python) so the default python is python3

      • Add tar xfz hera_calibration_packages.tar.gz

      • Add export PYTHONPATH=${PWD}/hera_calibration_packages

      • Add export PATH=${PWD}/python/bin:${PATH}

      • Add rm -f python37.tar.gz to the end of the script

  • It changes directory to /lustre/aoc/projects/hera/krowe/makeflow_sample/raw_data

    • This line should just be removed

  • It runs /lustre/aoc/projects/hera/krowe/hera_opm/pipelines/h1c/idr2/v2/task_scripts/do_EXTRACT_AUTOS.sh zen.2458098.44615.HH.uvh5

    • Instead run ./do_EXTRACT_AUTOS.sh zen.2458098.44615.HH.uvh5 zen.2458098.44615.HH.uvh5

    • zen.2458098.44615.HH.uvh5 will need to be copied to the scratch area on the execution host.

    • do_EXTRACT_AUTOS.sh requires ${src_dir}/_common.sh

      • Copy ${src_dir}/_common.sh to the working directory and make it a dependency in the .mf file

    • do_EXTRACT_AUTOS.sh calls extract_autos.py which is from hera_cal.git

      • Copy extract_autos.py to the working directory and make it a dependency in the .mf file

  • On success it touches the .out file

    • This is unnecessary.  The .out file can just be the target for the rule in the .mf file

  • On error it moves the .log file to .EXTRACT_AUTOS.log.error

    • Is this really necessary?

I am confused as to the difference between the .log, .out and .log.error files.


idr2.2.mf


Even though condor copies do_EXTRACT_AUTOS.sh to the scratch area, it doesn’t run it.  Instead, it runs the full-path version (/lustre/aoc/projects/hera/krowe/…)  This is because the .mf file runs /lustre/.../wrapper_zen.2458098.44615.HH.autos.uvh5.EXTRACT.AUTOS.sh which in turn runs /lustre/.../do_EXTRACT_AUTOS.sh  To fix this, the .mf file will need to be generated differently.



Makeflow uses the classic Make syntax like so


target : prerequisites


        recipe


where it is expected that the recipe will update the target.  So, I think what you want is the wrapper script to output into a file (perhaps .log) that is the target.  E.g.

zen.2458098.44615.HH.uvh5.EXTRACT_AUTOS.log: /home/nu_kscott/hera/test1/wrapper_zen.2458098.44615.HH.uvh5.EXTRACT_AUTOS.sh do_EXTRACT_AUTOS.sh _common.sh ../share/makeflow_sample/raw_data/zen.2458098.44615.HH.uvh5 hera_calibration_packages.tar.gz extract_autos.py

        ./wrapper_zen.2458098.44615.HH.uvh5.EXTRACT_AUTOS.sh > zen.2458098.44615.HH.uvh5.EXTRACT_AUTOS.log 2>&1

What is different?

  • The target (zen.2458098.44615.HH.uvh5.EXTRACT_AUTOS.log) was changed from the .out file to the .log file.  This is common in makefiles where the target depends on the result of that target.  The .out file is created by the wrapper_*.sh script in the recipe. Honestly, using either the .log or .out file is ok.

  • The prerequisites contain all the files we want HTCondor to copy to the execution host.  hera_calibration_packages.tar.gz is about 250MB and zen.2458098.44615.HH.autos.uvh5 is about 5GB.  They are both too big for HTCondor's transfer mechanism and while it works in testing, we will need to either make them smaller or transfer them some other way.  http://chtc.cs.wisc.edu/file-avail-largedata

  • I replaced the absolute path to wrapper_zen.2458098.44615.HH.uvh5.EXTRACT_AUTOS.sh with ./ to cause makeflow to tell HTCondor to transfer this file to the scratch area on the execution host and execute it from that scratch area.

  • I removed the absolute path from zen.2458098.44615.HH.uvh5.EXTRACT_AUTOS.log so it can be written to the scratch area and HTCondor will copy it back from the execution host.

  • Build python-packages tarball.  python-packages

    • If over 100MB have it installed in CHTC’s squid proxy.  Using xz instead of gz compression reduces the file from 240MB to 178MB but increases the time to uncompress it from 65s to 215s.



  • No labels