...
Try to create and follow some form of structure using headers and bullets.
Data Reuse Mechanism
In v8.9 can cache job in put files on the execute machine.
File Transfer Improvements
AWS S3, box.com
DAGMan Data Flow
DAG can work like Make and run or not run jobs
based on the time of files. Does this replace what Makeflow does? Seems like it is simpler than Makeflow. Data Flow will skip nodes if the defined output file is newer than the input file. It is in the latest 8.9.7 release.
But, even though the input file is the older that both the output files and the executable and adding the knob to the 99-nrao file on the central manager/submit host and the execution host and running condor_reconfig, this job still runs. So either dataflow doesn't work or I don't know how to work it.
Docker
Condor has docker images (regular and mini)
docker run htcondor/cm
docker run -t -I htcondor/mini
...
docker_image = dev7_and_HEP_stack
docker vs singularity
HTCondor seems to prefer singularity because it doesn't start processes from a daemon and therefore processes can be better tracked by condor. condor_ssh_to_job works with singularity.
Youtube
https://www.youtube.com/channel/UCd1UBXmZIgB4p85t2tu-gLw
Center for High Throughput Computing
These seem mostly intro-level stuff.
HTMap
seems like a good approach to make the pipeline run imaging in the HTC environment
https://htmap.readthedocs.io/en/latest/
HTMap is a library that wraps the process of mapping Python function calls out to an HTCondor pool. It provides tools for submitting, managing, and processing the output of arbitrary functions.
Python tutorial
need python3-condor RPM or use pip install. It doesn't get installed by default with yum. I pointed this out to CHTC and they submitted a ticket. Should try installing locally with pip in a user's account
pip3 install htcondor
/dev/shm
/dev/shm is now job-private like /tmp and /var/tmp in version 8.9. It was already restricted by cgroups but now in 8.9 jobs can't see other job's /dev/shm and it gets cleaned when a job exits. (tested krowe May 28, 2020)
Versions
It is unlikely there will be a version 8.8.10 but instead 9.0
...