Please track notes from HTCondor week here, particularly new features we may want to investigate. It could be some time before we can look at them and I don't want to forget. If possible include:
- Name or very brief description of the feature (e.g. DAGman flow)
- Any links provided by the talk or that you can track down in semi-realtime
- Broader description of each item and maybe some context of where it might be useful
The primary function of this is as a note to future versions of us so no need to go into great detail, just want to avoid head scratching questions about 'what was that thing, that was going to help with that other thing'.
Try to create and follow some form of structure using headers and bullets.
Data Reuse Mechanism
In v8.9 can cache job in put files on the execute machine.
File Transfer Improvements
AWS S3, box.com
DAGMan Data Flow
DAG can work like Make and run or not run jobs
based on the time of files. Does this replace what Makeflow does?
Docker
Condor has docker images (regular and mini)
docker run htcondor/cm
docker run -t -I htcondor/mini
Could we use docker images for interactive use? That way docker could run a proper sshd? University in Bonn (High Energy Physics) does something like this.
condor_submit -i -append '+ContainerOS="CentOS7"'
Youtube
https://www.youtube.com/channel/UCd1UBXmZIgB4p85t2tu-gLw
Center for High Throughput Computing
These seem mostly intro-level stuff.
HTMap
seems like a good approach to make the pipeline run imaging in the HTC environment
https://htmap.readthedocs.io/en/latest/
HTMap is a library that wraps the process of mapping Python function calls out to an HTCondor pool. It provides tools for submitting, managing, and processing the output of arbitrary functions.