Versions Compared

Old Version 247

changes.mady.by.user K Scott Rowe

Saved on

compared with

New Version 248

changes.mady.by.user K Scott Rowe

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Look at the masks in the cgroup.

  • Bug in condor_annex: Underscores in the AnnexName prevent the annex from moving into the pool.
    • Also when I try to terminate an annex with underscores (e.g. krowe_annex_casa5) with the command condor_off -annex krowe_annex_casa5 I get the following error
      • Found no ClassAds when querying pool (local)
      • Can't find addresses for master's for constraint 'AnnexName =?= "krowe_annex_casa5"'

      • Perhaps you need to query another pool.

  • Bug in condor_annex: The following will wait for an annex named krowe - annex - casa5 (note the spaces).  If I pass $(myannex) as an argument to a shell script, the spaces are not there.
    • include.htc
      • myannex = krowe-annex-casa5
    • submit.htc
      • include : include.htc
      • executable = /bin/sleep
      • arguments = 127
      • +MayUseAWS = True
      • requirements = AnnexName == $(myannex)
      • queue
    • Actually, I think this isn't a bug but a limitation on using macros.  The AnnexName needs to be quoted but how can I quote a macro?  Note, I have the same problems with AnnexNames that don't have hyphens (E.g. krowetest).
      • No: requirements = AnnexName == "$(myannex)"
      • No: myannex = "krowe-annex-casa5"
      • No: myannex = \"krowe-annex-casa5\"
      • No: myannex = "\"krowe-annex-casa5\""
    • Idea: +annex = "krowe-annex-casa5"
    • requirements = AnnexName == my.annex

Answered Questions:

  • JOB ID question from Daniel

    • When I submit a job, I get a job ID back. My plan is to hold onto that job ID permanently for tracking. We have had issues in the past with Torque/Maui because the job IDs got recycled later and our internal bookkeeping got mixed up. So my questions are:

       - Are job IDs guaranteed to be unique in HTCondor?
       - How unique are they—are they _globally_ unique or just unique within a particular namespace (such as our cluster or the submit node)?

    • A Job ID (ClusterID.ProcID)
    • DNS name of the schedd and ctime of the job_queued.log file.
    • It is unique to a schedd.
    • We should talk with Daniel about this.  They should craft their own ID.  It could be seeded with a JobID but should not depend on just it.
  • UpgradingHTCondor without killing jobs?
    • schedd can be upgraded and restarted without loosing state assuming the restart is less than the timeout.
    • currently restarting execute services will kill jobs.  CHTC is working on improving this.
    • negotiator and collector can be restarted without killing jobs.
    • CHTC works hard to ensure 8.8.x is compatible with 8.8.y or 8.9.x is compatible with 8.9.y.
  • Leaving data on execution host between jobs (data reuse)
    • Todd is working on this now.
  • Ask about installation of CASA locally and ancillary data (cfcache)
    • CHTC has a Ceph filesystem that is available to many of their execution hosts (notibly the larger ones)
    • There is another software filesystem where CASA could live that is more used for admin usage but might be available to us.
    • We could download the tarball each time over HTTP.  CHTC uses a proxy server so it would often be cached.
  • Environment:  Is there a way to have condor "login" when a job starts thus sourcing /etc/proflie and the user's rc files? Currently, not even $HOME is set.
    • A good analogy is Torque does a su - _username_ while HTCondor just does a su _username_
    • WORKAROUND: setting getenv = True which is like the -V option to qsub, may help. It doesn't source rc files but does inherit your current environment. This may be a problem if your current environment is not what you want on the cluster node. Perhaps the cluster node is a different OS or architecture.
    • ANSWER: condor doesn't execute things with a shell.  You could set your executable as /bin/bash and then have the arguments be the executable you used to have.  I just changed our stuff to staticly set $HOME and I think that is good enough.
  • Flocking: Suppose I have two hosts in the same pool.  testpost-master is a submit-host and testpost-serv-1 is both a submit-host and the central-manager.  testpost-serv-1 is configured to flock to CHTC but testpost-master is not. Is it possible to submit a job on testpost-master that will flock to CHTC by somehow leveraging testpost-serv-1?  In other words, do I have to setup flocking and an external IP on every submit host?
    • ANSWER: there isn't a good way to do this.  So eventually we will need to make testpost-master flock to CHTC and possibly remove the ability of testpost-serv-1 to flock.

...

  • Look in the job log file not the dag log file
  • 040 (150.000.000) 2020-06-15 13:05:45 Started transferring input files
            Transferring to host: <10.64.10.172:9618?addrs=10.64.10.172-9618&alias=nmpost072.aoc.nrao.edu&noUDP&sock=slot1_1_72656_7984_60>
    ...
    040 (150.000.000) 2020-06-15 13:06:04 Finished transferring input files
  • Rank and Premption: Can we use Rank to set "preferences" without requiring job preemption?
    • ANSWER: There are 2 kinds of rank (job rank, machine rank).  job rank (RANK=... in a submit file) is purely a preference.  That does not preempt.  Machine rank (in startd.config) will preempt.  Negotiator pre-job rank is a third type of rank that works at a pool level and is often used to pack jobs efficiently.

...

  • /staging/nu_jrobnett
  • Requirements = (Target.HasCHTCStaging == true)
  • Quota: 100GB, 100K files

...

  • /squid/nu_jrobnett
  • only accessable via this path on the submit hosts.  Execution hosts will need to access it via HTTP.
  • transfer_input_files = http://proxy.chtc.wisc.edu/SQUID/nu_jrobnett/casa.tgz

Software area We can use this in run-time applications.  Think of it like /usr/local.

...

/software/nu_jrobnett/casa/casa-pipeline-release-5.6.1-8.el7

...

...




Answered Questions:

  • JOB ID question from Daniel

    • When I submit a job, I get a job ID back. My plan is to hold onto that job ID permanently for tracking. We have had issues in the past with Torque/Maui because the job IDs got recycled later and our internal bookkeeping got mixed up. So my questions are:

       - Are job IDs guaranteed to be unique in HTCondor?
       - How unique are they—are they _globally_ unique or just unique within a particular namespace (such as our cluster or the submit node)?

    • A Job ID (ClusterID.ProcID)
    • DNS name of the schedd and ctime of the job_queued.log file.
    • It is unique to a schedd.
    • We should talk with Daniel about this.  They should craft their own ID.  It could be seeded with a JobID but should not depend on just it.
  • UpgradingHTCondor without killing jobs?
    • schedd can be upgraded and restarted without loosing state assuming the restart is less than the timeout.
    • currently restarting execute services will kill jobs.  CHTC is working on improving this.
    • negotiator and collector can be restarted without killing jobs.
    • CHTC works hard to ensure 8.8.x is compatible with 8.8.y or 8.9.x is compatible with 8.9.y.
  • Leaving data on execution host between jobs (data reuse)
    • Todd is working on this now.
  • Ask about installation of CASA locally and ancillary data (cfcache)
    • CHTC has a Ceph filesystem that is available to many of their execution hosts (notibly the larger ones)
    • There is another software filesystem where CASA could live that is more used for admin usage but might be available to us.
    • We could download the tarball each time over HTTP.  CHTC uses a proxy server so it would often be cached.
  • Environment:  Is there a way to have condor "login" when a job starts thus sourcing /etc/proflie and the user's rc files? Currently, not even $HOME is set.
    • A good analogy is Torque does a su - _username_ while HTCondor just does a su _username_
    • WORKAROUND: setting getenv = True which is like the -V option to qsub, may help. It doesn't source rc files but does inherit your current environment. This may be a problem if your current environment is not what you want on the cluster node. Perhaps the cluster node is a different OS or architecture.
    • ANSWER: condor doesn't execute things with a shell.  You could set your executable as /bin/bash and then have the arguments be the executable you used to have.  I just changed our stuff to staticly set $HOME and I think that is good enough.
  • Flocking: Suppose I have two hosts in the same pool.  testpost-master is a submit-host and testpost-serv-1 is both a submit-host and the central-manager.  testpost-serv-1 is configured to flock to CHTC but testpost-master is not. Is it possible to submit a job on testpost-master that will flock to CHTC by somehow leveraging testpost-serv-1?  In other words, do I have to setup flocking and an external IP on every submit host?
    • ANSWER: there isn't a good way to do this.  So eventually we will need to make testpost-master flock to CHTC and possibly remove the ability of testpost-serv-1 to flock.


  • It seems the transfer mechanism won't transfer symlinks to directories (e.g. data/vlass.ms → /lustre/aoc/...) Is there a way around this?
    • ANSWER: there is no flag to chase symlinks at the moment.  The top level dir (e.g. data) could be a symlink if transfer_input_files=data/ but it will then transfer the contents of data instead of data itself.
    • If symlink → data and transfer_input_files=symlink I get the error Transfer of symlinks to directories is not supported.
    • if symlink → ../data and transfer_input_files=symlink/ it transfers the contents not the directory.  In other words I don't have a data directory in scratch I have a VLASS... directory.
    • If data/VLASS → /some/path/VLASS and transfer_input_files=data/VLASS/
    • If data/VLASS → /some/path/VLASS and transfer_input_files=data/ I get the error Transfer of symlinks to directories is not supported.

  • DAG log time stamps,  is there a way to differentiate data import/export time and process run time.
    • Look in the job log file not the dag log file
    • 040 (150.000.000) 2020-06-15 13:05:45 Started transferring input files
              Transferring to host: <10.64.10.172:9618?addrs=10.64.10.172-9618&alias=nmpost072.aoc.nrao.edu&noUDP&sock=slot1_1_72656_7984_60>
      ...
      040 (150.000.000) 2020-06-15 13:06:04 Finished transferring input files


  • Rank and Premption: Can we use Rank to set "preferences" without requiring job preemption?
    • ANSWER: There are 2 kinds of rank (job rank, machine rank).  job rank (RANK=... in a submit file) is purely a preference.  That does not preempt.  Machine rank (in startd.config) will preempt.  Negotiator pre-job rank is a third type of rank that works at a pool level and is often used to pack jobs efficiently.


  • Update on software store for CASA either on shared Ceph storage or admin software storage
    • Staging area for datasets 100MB - TBs.  This is where we could try keeping the cfcache assuming doing so doesn't overwhelm the filesystem.
      • /staging/nu_jrobnett
      • Requirements = (Target.HasCHTCStaging == true)
      • Quota: 100GB, 100K files
    • Squid area for 100MB - 1GB input or shared software.  This is where we could keep casa.tgz and then have the execution host retrieve it via HTTP.
      • /squid/nu_jrobnett
      • only accessable via this path on the submit hosts.  Execution hosts will need to access it via HTTP.
      • transfer_input_files = http://proxy.chtc.wisc.edu/SQUID/nu_jrobnett/casa.tgz

    • Software area We can use this in run-time applications.  Think of it like /usr/local.

  • Flocking: When we flock to CHTC what is the data path for transfer_input_files?  Is it our submit host and CHTC's execution host, is CHTCs submit host involved ?
    • Dataflow is from our schedd (submit host) to their execute host but CCB will reverse the connection.  Their execution hosts are publicly addressable but that may not be necessary.

  • How can we choose the data path for transfer_input_files to our clients given multiple networks.  Currently we assume it will use the 1Gb link but we have IB links.    Is there a way for condor to use the IB link just for transferring files, is that hostname based ? Other ideas?
    • CHTC doesn't have a good solution for this.
    • We could upgrade from 1Gb to 10Gb
    • We could use the IB names for everything (problematic for submit hosts that don't have IB)
    • We could not use transfer mechanism and instead use something else like scp
    • We could use a custom transfer plugin

  • Are there known issues with distributed scratch via NFS or Lustre w.r.t tmpdir or other,  e.g. OpenMPI complains about tmpdir being on network FS?
    • Some problems with log files on the submit host but rare.
  • Any general best practices to support MPI in terms of class ads or other.
    • Use the shared memory transport for security

  •  Is there a way DAGMan can be told to ignore errors, in some cases we want a DAG to mindlessly continue vs retry.
    • The job is considered successful based on the return of the post script.  If there isn't a post script, the success is based on the return of the job.

  • Transfer mechanism: Documentation implies that only files with an mtime newer than when the transfer_input_files finished will be transferred back to the submit host.  While running a dag, the files in my working directory (which is in both transfer_input_files and transfer_output_files) seem to always have an mtime around the most recent step in the DAG suggesting that the entire working directory is copied from the execution host to the submit host at the end of each DAG step.  Perhaps this means the transfer mechanism only looks at the mtime of the files/dirs specified in transfer_output_files and doesn't descend into the directories.
    • Subdirectories are treated differently
    • SOLUTION: I think casa just touches every file and therefore condor is forced to copy everything in the working directory.  I have been unable to reproduce the problem outside of casa.
    • SOLUTION: If you specify a directory, HTCondor will transfer the entire directory not just files with new mtime.


  • Does the trasnfer mechanism accept any sort of regular expression?  E.g. transfer_input_files=*.txt
    • No

  • Can the transfer mechanism accept manifest files?  E.g. a file that is a list of files?
    • Use include : <some file> in the submit script where <some file> contains the full transfer_input_files line
    • use queue FILES from manifest Which defines the submit variable $(FILES) which could be used in a transfer_input_files like: transfer_input_files = $(FILES)
    • Perhaps a plugin

  • What other options are there than holding a job?  I find myself not noticing, sometimes for hours, that a job is on hold.  Is there a way to make jobs fail instead of getting held?  I assume others will make this mistake like me.
    • I see I can set periodic_remove = (JobStatus == 5) but HTCondor doesn't seem to think that is an error so if I have notification = Error I don't get any email.
    • Greg will look into adding a Hold option to notification
    • The HTCondor idea of held jobs is that you submitted a large DAG of jobs, one step is missing a file and you would like to put that file in place and continue the job instead of the whole DAG failing and having to be resubmitted.  This makes sense but it would be nice to be notified when a job gets held.

    • Greg wrote "notification = error in the submit file is supposed to send email when the job is held by the system, but there's a bug now where it doesn't.  I'll fix this."


  • What limits are there to transfer_input_files?  I would sometimes get Failed to transfer files when the number of files was around 10,000
    • ANSWER: There is a memory limit because of ClassAds but in general there isn't a defined limit.
  • Is there a way to generate the dag.dot file without having to submit the job?
    • The -no_submit option doesn't create the .dot file
    • Is adding NOOP to all the JOB commands the right thing to do?  The DAG still gets submitted but then quickly ends.
    • ANSWER: You need to submit the DAG.  NOOP is the current solution
  • Is there a way to start a dag at a given point?  E.g. if there are 5 steps in the dag, can you start the job at step 3?
    • Is the answer again to add NOOP to the JOB commands you don't want to run?
    • ANSWER: Splices may work here but NOOP is certainly a tool to use.
  • I see at CHTC jobs now require a request_disk setting.  How does one launch interactive jobs?
    • ANSWER: This is a bug.
  • For our initial tests, we want to flock jobs to CHTC that transfer about 60GB input and output.  Eventually we will reduce this significantly but for now what can we do?
    • Globus or rsync to move data?  If Globus, how to do so in an automated way (E.g. no password)?
    • ANSWER: Using the Transfer Mechanism from our submit server to CHTC's execution host is ok with CHTC because it doesn't interfere with their submit host.  Outbound from their execute hosts is also allowed (scp, rsync, etc).
  • Use +Longjob = true attribute for 72+ hour jobs.  This is a CHTC-specific knob.


  • How can I know if my job swapped?
    • ANSWER: CHTC nodes have no or minimal swap space.


  • Condor Annex processing in AWS.  Is there support for spot market
    • ANSWER: Condor Annex does indeed support the spot market.  It is a bit more work to set up because you don't say "give my X of Y", but "I'll pay d1 dollars for machines like X1 and d2 for machines like X2, etc.".


  • What network mask should we use to allow ssh from CHTC into NRAO?  Is it a class B or several class Cs?
    • ANSWER: The ip (v4 !) ranges for CHTC execute nodes are

      128.104.100.0/22

      128.104.55.0/24

      128.104.58.0/23

      128.105.244.0/23


  • Is there a ganglia server or some other monitor service at CHTC we can view?

    • ANSWER: We have a bunch of ganglia and grafana graphs for the system, but I think they are restricted to campus folks and tend to show system-wide utliization and problem



  • I have a machine with an externaly-accessable, non-NATed address (146.88.10.48) and an internal, non-routable address (10.64.1.226).  I want to install condor_annex on this machine such that I can submit jobs to AWS from it.  I don't necessarily need to submit jobs to local execute hosts from this machine.  Should I make this machine a central manager, a submit host, both, or does it matter?

    • ANSWER: I think instances in AWS will need to contact both the schedd (submit host) and collectord (central manager) from the internet using port 9618.  So either both submit host and central manager need external connections and IPs with port 9618 open or combine them into one host with an external IP and port 9618 open to the Internet.

  • Last time I configured condor_annex I was using an older version of condor (8.8.3 I think) and used a pool password for security.  Now I am using 8.9.7.  Is there a newer/better security method I should use?

    • ANSWER: The annex still primarily uses pool password, so let's stick with that for now.




  • How can I find out what hosts are available for given requirements (LongJob, memory, staging)
    • condor_status -compact -constraint "HasChtcStaging==true" -constraint 'DetectedMemory>500000' -constraint "CanRunLongJobs isnt Undefined"
    • Answer: yes this is correct but it doesn't show what other jobs are waiting on the same resources.  Which is fine.
  • It looks to me like most hosts at CHTC are setup to run LongJobs.  The following shows a small list of about 20 hosts so I assume all others can run long jobs.  Is the correct?
    • condor_status -compact -constraint "CanRunLongJobs is Undefined"
    • JongJobs is for something like 72 hours.  So it might be best to not set it unless we really need it like step23.
    • Answer: yes this is correct but it doesn't show what other jobs are waiting on the same resources.  Which is fine.



  • Is port 9618 needed for flocking or just for condor_annex?
    • ANSWER: Greg thinks yes 9618 is needed for both flocking and condor_annex.
  • Are there bugs in the condor.log output of a DAG node?  For example, I have a condor.log file that clearly shows the job taking about three hours to run yet at the bottom lists user time of 13 hours and system time of 1 hour.  https://open-confluence.nrao.edu/download/attachments/40541486/step07.py.condor.log?api=v2

    • And as for the cpu usage report, there could very well be a bug, but first, is your job multi-threaded or multi-process?  If so, the cpu usage will be the aggregate across all cpu cores.

    • Yes they are all parallel jobs to some extent so I accept your answer for that job.  But I have another job that took 21 hours of wallclock time and yet the condor.log shows 55 minutes of user and 5:34 hours of system time.  https://open-confluence.nrao.edu/download/attachments/40541486/step05.py.condor.log?api=v2
    • ANSWER: if you look, the user time is actually 6 days and 55 minutes.  I missed the 6 in there.


  • Given a DAG where some steps are to always  run at AOC and some are to always  run in CHTC how can we dictate this.  Right now local jobs flock  to CHTC if local resources are full.  Can we make local jobs idle instead of flock?
    • ANSWER: Use PoolNames.  I need to make a testpost PoolName.
    • PoolName = "TESTPOST"

    • STARTD_ATTRS = $(STARTD_ATTRS) PoolName


  • It seems that when using DAGs the recommended method is to define variables in the DAG script instead of submit scripts.  This makes sense as it allows for only one file, the DAG script, that needs to be edited to make changes.  But, is there a way to get variables into the DAG script from the command line or environment or an include_file or something?
    • ANSWER: There is an INCLUDE syntax but there is no command-line or environment variable way to get vars into a DAG.
  • We are starting to run 10s of jobs in CHTC requiring 40GB as part of a local DAG.  Are there any options we can set to improve their execution chance.  What memory footprint (32, 20, 16, 8GB) would significantly improve their chances.
    • ANSWER: only use +LongJob if the job needs more than 72 hour, which is the default "walltime".
  • How can we set AWS Tags with condor_annex?  We'd like this to track jobs and set billing tags.  Looks like there isn't really a way.
    • SOLUTION: Greg wrote (Oct. 5, 2020) "we've just now added code to the condor_annex to support setting multiple aws tags on the annex command line".  K. Scott expects it will take a while before that code makes it to released software.
    • Launch Templates didn't work.  I don't think condor_annex supports Launch Templates.
    • Use aws-user-data options to condor_annex?
      • I have tried all sorts of user-data and default-user-data-file options.  On-demand apparently no longer works and I was never able to get something working with spot-fleet.  I think all things user-data are non-functional.
    • I tried setting a tag in the role defined in config.json (aws-ec2-spot-fleet-tagging-role) but that tag didn't translate to the instance.
    • I tried adding a tag to the AMI when creating a new AMI (EC2 → Instances → Actions → Image → Create Image).  Didn't work.
    • What about selftagging?  The instance figures out its instance id and runs aws.
      • wget -qO- http://instance-data/latest/meta-data/instance-id
        • returns nothing when logged in as nobody (condor_ssh_to_job)
        • returns nothing when logged in as centos (ssh -i ~/.ssh/...)
        • returns instanceid when logged in as root (ssh as centos then sudo su)
        • Aha!  There is a firewall (iptables) rule blocking exactly this.  But I can't figure out what file sets this iptables rule on boot.
    • I tried adding tags to the json file using both ResourceType set to instance and spot-fleet-request.  Neither created an instance with my tag.

  • "TagSpecifications": [
    {
    "ResourceType": "instance",
    "Tags": [
    {
    "Key": "budget",
    "Value": "VLASS"
    }
    ]
    {
    ],

  • What are the clever solutions to submitting N different DAG jobs with each having different parmeters?

    • T10t34

      • J220200-003000

        • bin, working, data

      • J220600-003000

        • bin, working, data

      • ...

    • T10t35

      • J170743-393000

        • bin, working, data

      • J171241-383000

        • bin, working, data

      • ...

    • ANSWERS:

    • INCLUDE syntax for DAGs

    • include syntax for submit files

    • make a template of files

    • use a PRE script that populates things

    • usedagdir


  • I had a job killed because it exceeded 72 hours even though I set +LongJobs = true  in the submit file 
    • 2385.0 krowe 9/22 20:43 Error from slot1_1@e2008.chtc.wisc.edu: Job failed to complete in 72 hrs
    • ANSWER: the knob is sinuglar +LongJob = true
  • What are the options to setting up HTCondor to both flock to CHTC and annex to AWS? Multiple submit hosts?  Multiple CMs? etc.
    • ANSWER: Philosophy is for everyone to submit in one place and let condor sort out where it goes.
    • CHTC flocks annex jobs to a different CM that actually starts the annex.
      • Submit annex job on SM.  It then flocks to a different CM that can create the annex
  • As we feared, referencing the cache of convolution functions (cfcache) directly from staging performed poorly.  This is due to a fstat() pathology that fares poorly on distributed filesystems.  Jobs ran 3 to 4 times faster when we copied cfcache from /staging to local disk.  I ran a small data set test with full parameters at CHTC that copied cfcache from /staging to local disk and step05 took only 16.7 hours instead of the 56.8 hours it had taken using cfcache on /staging.



  • Condor_annex bug: Edit  /usr/libexec/condor/condor-annex-ec2 and comment out the line chkconfig condor || exit 1 because this line is a hold-over from older versions that put condor in init.d. Now that it is in systemd, this line causes condor to exit.
    • SOLUTION: Greg submitted ticket on this.
  • Debugging held jobs.  I had thought that setting when_to_transfer_output = ON_EXIT_OR_EVICT  would copy the scratch area back to the submit machine so files there could be inspected.  But that doesn't seem to happen for me.
  • Memory issue: Greg did find a bug deep in the code that may cause jobs to be killed because of memory issues. HTCondor occationally gets a short read when looking at the process table via /proc, and then something like 2/3 of the processes are missing.
    • SOLUTION: CHTC will work towards a solution.


  • How can we have the .dag.* files written to a different directory?  -usedagdir doesn't help.
    • ANSWER: There isn't a way to tell condor_submit_dag where to put the logs
  • Is there a for-loop structure available to DAG scripts or a range mechanic?
    • No
  • If 8.9.9 requires Globus from EPEL then it may have trouble being installed on a Globus endpoint because the EPEL version of Globus conflicts with the Globus.org version.
    • I told them about it. I have not tried installing HTCondor-8.9.9 so I am only guessing it will be a problem.



  • Is there a recommended way to start annexes from a DAG?  We have been using PRE scripts but sometimes it seems to fail.
    • SOLUTIONS:
      • CHTC is working on a BEGIN syntax (provision) that will block a DAG node from starting until the annex is ready.
      • We could have the script not return until the annex is ready.
      • We could also have the job require a specific name that the create_annex creates.


  • How can I set a variable in a DAG file that I can then use in the submit file in a conditional?  None of the following seem to work
    • DAG:

      • VARS step01 CHTC=""

      • VARS step05 CHTC="True"
    • Submit:
      • if defined $(CHTC)
        • requirements = PoolName == "CHTC"
      • endif
    • or
    • DAG:
      • #VARS step01 CHTC="True"
      • VARS step05 CHTC="True"
    • Submit:
      • if defined $(CHTC)
        • requirements = PoolName == "CHTC"
      • endif
    • or
    • DAG:
      • VARS step01 CHTC="False"
      • VARS step05 CHTC="True"
    • Submit:
      • chtc_var = $(CHTC)
      • if $(chtc_var)
        • requirements = PoolName == "CHTC"
      • endif
    • even though when I pass $(chtc_var) as arguments to the shell script, the shell script sees it as True.
    • or
    • DAG:

      • VARS node1 file="chtc.htc"

      • VARS node2 file="aws.htc"
    • Submit:
      • include : $(file)
10/20/20 08:54:36 From submit: ERROR: on Line 9 of submit file:
10/20/20 08:54:36 From submit: Submit:-1:Error "", Line 0, Include Depth 1: can't open file
10/20/20 08:54:36 From submit:
10/20/20 08:54:36 From submit: ERROR: Failed to parse command file (line 9).
10/20/20 08:54:36 failed while reading from pipe.
10/20/20 08:54:36 Read so far: Submitting job(s)ERROR: on Line 9 of submit file: Submit:-1:Error "", Line 0, Include Depth 1: can't open fileERROR: Failed to parse command file (line 9).
10/20/20 08:54:36 ERROR: submit attempt failed
    • Yet I can use a variable defined in a DAG for things like arguments and request_memory.
    • I can also use file = $CHOICE(myindex, chtc.htc, aws.htc) where myindex is defined in a DAG it will set $(file) to the file I want to include but again if I use include : $(file) I get an error
10/20/20 11:58:58 From submit: Submitting job(s)ERROR on Line 13 of submit file: $CHOICE() macro: myindex is invalid index!
10/20/20 11:58:58 failed while reading from pipe.
10/20/20 11:58:58 Read so far: Submitting job(s)ERROR on Line 13 of submit file: $CHOICE() macro: myindex is invalid index!
10/20/20 11:58:58 ERROR: submit attempt failed
  • Perhaps use requirements.  Greg will send an example
    • SOLUTION:
    • DAG:
      • JOB step05 step05.htc
      • #VARS step05 SITE="chtc"
      • #VARS step05 SITE="aws"
    • Submit:
      • +NRAOAttr = "$(SITE)"
      • Requirements = My.NRAOAttr == "chtc" ? PoolName == "CHTC" : PoolName =!= "CHTC"

      • Requirements = My.NRAOAttr == "chtc" ? (Target.HasCHTCStaging == true) : (Target.HasCHTCStaging =!= true)

      • myannex = "krowe-annex"
      • +MayUseAWS = True

      • Requirements = My.NRAOAttr == "aws" ? AnnexName == $(myannex) : AnnexName =!= $(myannex)

    • I would set myannex in the DAG but when I do that it tries to find an AnnexName of "krowe - annex" (note spaces)
  • ANSWER: My conclusion is that there are limitations on what one can do with variables in the submit file that were defined in the DAG file.


  • Is there a config option that will cause condor to not start?  We have diskless nodes and it is easier to modify the config file then change systemd.
    • SOLUTION: Either set START_MASTER = False or START_DAEMONS = False depending on desired outcome.


  • Torque has this command called pbsnodes that can not only offline/drain a node but keeps a note about it that all can see in one place.  I know I can use condor_off to drain a node but is there a central place keep notes so I can remember a month later why I set a certain node to drain?
    • ANSWER: there is no place to keep such notes but Greg likes the idea and may look into it.
    • May want to use condor_drain instead of condor_off.  condor_off will kill the startd when all jobs finish and it no longer shows up in condor_status.  condor_drain will leave the node in condor_status.
    • condor_drain doesn't work for me because it immediatly sets jobs idle instead of letting them run to completion.  This is why I use condor_off -startd -peaceful instead.


  • How can you tell which job is associated with an email given the email message doesn't include a working dir or the assigned batch_name?
    • CHTC will look into adding such information to the email condor sends.



  • Bug in condor_annex: Underscores in the AnnexName prevent the annex from moving into the pool.
    • Also when I try to terminate an annex with underscores (e.g. krowe_annex_casa5) with the command condor_off -annex krowe_annex_casa5 I get the following error
      • Found no ClassAds when querying pool (local)
      • Can't find addresses for master's for constraint 'AnnexName =?= "krowe_annex_casa5"'

      • Perhaps you need to query another pool.

    • Greg has noted this bug


  • Bug in condor_annex: The following will wait for an annex named krowe - annex - casa5 (note the spaces).  If I pass $(myannex) as an argument to a shell script, the spaces are not there.
    • include.htc
      • myannex = krowe-annex-casa5
    • submit.htc
      • include : include.htc
      • executable = /bin/sleep
      • arguments = 127
      • +MayUseAWS = True
      • requirements = AnnexName == $(myannex)
      • queue
    • Actually, I think this isn't a bug but a limitation on using macros.  The AnnexName needs to be quoted but how can I quote a macro?  Note, I have the same problems with AnnexNames that don't have hyphens (E.g. krowetest).
      • No: requirements = AnnexName == "$(myannex)"
      • No: myannex = "krowe-annex-casa5"
      • No: myannex = \"krowe-annex-casa5\"
      • No: myannex = "\"krowe-annex-casa5\""
    • Idea: +annex = "krowe-annex-casa5"
    • requirements = AnnexName == my.annex
    • Greg has noted this bug

...

  • Some problems with log files on the submit host but rare.

...

  • Subdirectories are treated differently
  • SOLUTION: I think casa just touches every file and therefore condor is forced to copy everything in the working directory.  I have been unable to reproduce the problem outside of casa.
  • SOLUTION: If you specify a directory, HTCondor will transfer the entire directory not just files with new mtime.

...

  • I see I can set periodic_remove = (JobStatus == 5) but HTCondor doesn't seem to think that is an error so if I have notification = Error I don't get any email.
  • Greg will look into adding a Hold option to notification
  • The HTCondor idea of held jobs is that you submitted a large DAG of jobs, one step is missing a file and you would like to put that file in place and continue the job instead of the whole DAG failing and having to be resubmitted.  This makes sense but it would be nice to be notified when a job gets held.

  • Greg wrote "notification = error in the submit file is supposed to send email when the job is held by the system, but there's a bug now where it doesn't.  I'll fix this."

  • What limits are there to transfer_input_files?  I would sometimes get Failed to transfer files when the number of files was around 10,000
    • ANSWER: There is a memory limit because of ClassAds but in general there isn't a defined limit.
  • Is there a way to generate the dag.dot file without having to submit the job?
    • The -no_submit option doesn't create the .dot file
    • Is adding NOOP to all the JOB commands the right thing to do?  The DAG still gets submitted but then quickly ends.
    • ANSWER: You need to submit the DAG.  NOOP is the current solution
  • Is there a way to start a dag at a given point?  E.g. if there are 5 steps in the dag, can you start the job at step 3?
    • Is the answer again to add NOOP to the JOB commands you don't want to run?
    • ANSWER: Splices may work here but NOOP is certainly a tool to use.
  • I see at CHTC jobs now require a request_disk setting.  How does one launch interactive jobs?
    • ANSWER: This is a bug.
  • For our initial tests, we want to flock jobs to CHTC that transfer about 60GB input and output.  Eventually we will reduce this significantly but for now what can we do?
    • Globus or rsync to move data?  If Globus, how to do so in an automated way (E.g. no password)?
    • ANSWER: Using the Transfer Mechanism from our submit server to CHTC's execution host is ok with CHTC because it doesn't interfere with their submit host.  Outbound from their execute hosts is also allowed (scp, rsync, etc).
  • Use +Longjob = true attribute for 72+ hour jobs.  This is a CHTC-specific knob.
  • How can I know if my job swapped?
    • ANSWER: CHTC nodes have no or minimal swap space.
  • Condor Annex processing in AWS.  Is there support for spot market
    • ANSWER: Condor Annex does indeed support the spot market.  It is a bit more work to set up because you don't say "give my X of Y", but "I'll pay d1 dollars for machines like X1 and d2 for machines like X2, etc.".
  • What network mask should we use to allow ssh from CHTC into NRAO?  Is it a class B or several class Cs?
    • ANSWER: The ip (v4 !) ranges for CHTC execute nodes are

      128.104.100.0/22

      128.104.55.0/24

      128.104.58.0/23

      128.105.244.0/23

  • Is there a ganglia server or some other monitor service at CHTC we can view?

    • ANSWER: We have a bunch of ganglia and grafana graphs for the system, but I think they are restricted to campus folks and tend to show system-wide utliization and problem

  • I have a machine with an externaly-accessable, non-NATed address (146.88.10.48) and an internal, non-routable address (10.64.1.226).  I want to install condor_annex on this machine such that I can submit jobs to AWS from it.  I don't necessarily need to submit jobs to local execute hosts from this machine.  Should I make this machine a central manager, a submit host, both, or does it matter?

    • ANSWER: I think instances in AWS will need to contact both the schedd (submit host) and collectord (central manager) from the internet using port 9618.  So either both submit host and central manager need external connections and IPs with port 9618 open or combine them into one host with an external IP and port 9618 open to the Internet.

  • Last time I configured condor_annex I was using an older version of condor (8.8.3 I think) and used a pool password for security.  Now I am using 8.9.7.  Is there a newer/better security method I should use?

    • ANSWER: The annex still primarily uses pool password, so let's stick with that for now.
  • How can I find out what hosts are available for given requirements (LongJob, memory, staging)
    • condor_status -compact -constraint "HasChtcStaging==true" -constraint 'DetectedMemory>500000' -constraint "CanRunLongJobs isnt Undefined"
    • Answer: yes this is correct but it doesn't show what other jobs are waiting on the same resources.  Which is fine.
  • It looks to me like most hosts at CHTC are setup to run LongJobs.  The following shows a small list of about 20 hosts so I assume all others can run long jobs.  Is the correct?
    • condor_status -compact -constraint "CanRunLongJobs is Undefined"
    • JongJobs is for something like 72 hours.  So it might be best to not set it unless we really need it like step23.
    • Answer: yes this is correct but it doesn't show what other jobs are waiting on the same resources.  Which is fine.
  • Is port 9618 needed for flocking or just for condor_annex?
    • ANSWER: Greg thinks yes 9618 is needed for both flocking and condor_annex.
  • Are there bugs in the condor.log output of a DAG node?  For example, I have a condor.log file that clearly shows the job taking about three hours to run yet at the bottom lists user time of 13 hours and system time of 1 hour.  https://open-confluence.nrao.edu/download/attachments/40541486/step07.py.condor.log?api=v2

    • And as for the cpu usage report, there could very well be a bug, but first, is your job multi-threaded or multi-process?  If so, the cpu usage will be the aggregate across all cpu cores.

    • Yes they are all parallel jobs to some extent so I accept your answer for that job.  But I have another job that took 21 hours of wallclock time and yet the condor.log shows 55 minutes of user and 5:34 hours of system time.  https://open-confluence.nrao.edu/download/attachments/40541486/step05.py.condor.log?api=v2
    • ANSWER: if you look, the user time is actually 6 days and 55 minutes.  I missed the 6 in there.

...

STARTD_ATTRS = $(STARTD_ATTRS) PoolName

...

  • ANSWER: There is an INCLUDE syntax but there is no command-line or environment variable way to get vars into a DAG.

...

  • ANSWER: only use +LongJob if the job needs more than 72 hour, which is the default "walltime".

...

  • SOLUTION: Greg wrote (Oct. 5, 2020) "we've just now added code to the condor_annex to support setting multiple aws tags on the annex command line".  K. Scott expects it will take a while before that code makes it to released software.
  • Launch Templates didn't work.  I don't think condor_annex supports Launch Templates.
  • Use aws-user-data options to condor_annex?
    • I have tried all sorts of user-data and default-user-data-file options.  On-demand apparently no longer works and I was never able to get something working with spot-fleet.  I think all things user-data are non-functional.
  • I tried setting a tag in the role defined in config.json (aws-ec2-spot-fleet-tagging-role) but that tag didn't translate to the instance.
  • I tried adding a tag to the AMI when creating a new AMI (EC2 → Instances → Actions → Image → Create Image).  Didn't work.
  • What about selftagging?  The instance figures out its instance id and runs aws.
    • wget -qO- http://instance-data/latest/meta-data/instance-id
      • returns nothing when logged in as nobody (condor_ssh_to_job)
      • returns nothing when logged in as centos (ssh -i ~/.ssh/...)
      • returns instanceid when logged in as root (ssh as centos then sudo su)
      • Aha!  There is a firewall (iptables) rule blocking exactly this.  But I can't figure out what file sets this iptables rule on boot.
  • I tried adding tags to the json file using both ResourceType set to instance and spot-fleet-request.  Neither created an instance with my tag.

...

  • T10t34

    • J220200-003000

      • bin, working, data

    • J220600-003000

      • bin, working, data

    • ...

  • T10t35

    • J170743-393000

      • bin, working, data

    • J171241-383000

      • bin, working, data

    • ...

  • ANSWERS:

  • INCLUDE syntax for DAGs

  • include syntax for submit files

  • make a template of files

  • use a PRE script that populates things

  • usedagdir

  • I had a job killed because it exceeded 72 hours even though I set +LongJobs = true  in the submit file 
    • 2385.0 krowe 9/22 20:43 Error from slot1_1@e2008.chtc.wisc.edu: Job failed to complete in 72 hrs
    • ANSWER: the knob is sinuglar +LongJob = true
  • What are the options to setting up HTCondor to both flock to CHTC and annex to AWS? Multiple submit hosts?  Multiple CMs? etc.
    • ANSWER: Philosophy is for everyone to submit in one place and let condor sort out where it goes.
    • CHTC flocks annex jobs to a different CM that actually starts the annex.
      • Submit annex job on SM.  It then flocks to a different CM that can create the annex
  • As we feared, referencing the cache of convolution functions (cfcache) directly from staging performed poorly.  This is due to a fstat() pathology that fares poorly on distributed filesystems.  Jobs ran 3 to 4 times faster when we copied cfcache from /staging to local disk.  I ran a small data set test with full parameters at CHTC that copied cfcache from /staging to local disk and step05 took only 16.7 hours instead of the 56.8 hours it had taken using cfcache on /staging.
  • Condor_annex bug: Edit  /usr/libexec/condor/condor-annex-ec2 and comment out the line chkconfig condor || exit 1 because this line is a hold-over from older versions that put condor in init.d. Now that it is in systemd, this line causes condor to exit.
    • SOLUTION: Greg submitted ticket on this.
  • Debugging held jobs.  I had thought that setting when_to_transfer_output = ON_EXIT_OR_EVICT  would copy the scratch area back to the submit machine so files there could be inspected.  But that doesn't seem to happen for me.
  • Memory issue: Greg did find a bug deep in the code that may cause jobs to be killed because of memory issues. HTCondor occationally gets a short read when looking at the process table via /proc, and then something like 2/3 of the processes are missing.
    • SOLUTION: CHTC will work towards a solution.
  • How can we have the .dag.* files written to a different directory?  -usedagdir doesn't help.
    • ANSWER: There isn't a way to tell condor_submit_dag where to put the logs
  • Is there a for-loop structure available to DAG scripts or a range mechanic?
    • No
  • If 8.9.9 requires Globus from EPEL then it may have trouble being installed on a Globus endpoint because the EPEL version of Globus conflicts with the Globus.org version.
    • I told them about it. I have not tried installing HTCondor-8.9.9 so I am only guessing it will be a problem.
  • Is there a recommended way to start annexes from a DAG?  We have been using PRE scripts but sometimes it seems to fail.
    • SOLUTIONS:
      • CHTC is working on a BEGIN syntax (provision) that will block a DAG node from starting until the annex is ready.
      • We could have the script not return until the annex is ready.
      • We could also have the job require a specific name that the create_annex creates.
  • How can I set a variable in a DAG file that I can then use in the submit file in a conditional?  None of the following seem to work
    • DAG:

      • VARS step01 CHTC=""

      • VARS step05 CHTC="True"
    • Submit:
      • if defined $(CHTC)
        • requirements = PoolName == "CHTC"
      • endif
    • or
    • DAG:
      • #VARS step01 CHTC="True"
      • VARS step05 CHTC="True"
    • Submit:
      • if defined $(CHTC)
        • requirements = PoolName == "CHTC"
      • endif
    • or
    • DAG:
      • VARS step01 CHTC="False"
      • VARS step05 CHTC="True"
    • Submit:
      • chtc_var = $(CHTC)
      • if $(chtc_var)
        • requirements = PoolName == "CHTC"
      • endif
    • even though when I pass $(chtc_var) as arguments to the shell script, the shell script sees it as True.
    • or
    • DAG:

      • VARS node1 file="chtc.htc"

      • VARS node2 file="aws.htc"
    • Submit:
      • include : $(file)
10/20/20 08:54:36 From submit: ERROR: on Line 9 of submit file:
10/20/20 08:54:36 From submit: Submit:-1:Error "", Line 0, Include Depth 1: can't open file
10/20/20 08:54:36 From submit:
10/20/20 08:54:36 From submit: ERROR: Failed to parse command file (line 9).
10/20/20 08:54:36 failed while reading from pipe.
10/20/20 08:54:36 Read so far: Submitting job(s)ERROR: on Line 9 of submit file: Submit:-1:Error "", Line 0, Include Depth 1: can't open fileERROR: Failed to parse command file (line 9).
10/20/20 08:54:36 ERROR: submit attempt failed
    • Yet I can use a variable defined in a DAG for things like arguments and request_memory.
    • I can also use file = $CHOICE(myindex, chtc.htc, aws.htc) where myindex is defined in a DAG it will set $(file) to the file I want to include but again if I use include : $(file) I get an error
10/20/20 11:58:58 From submit: Submitting job(s)ERROR on Line 13 of submit file: $CHOICE() macro: myindex is invalid index!
10/20/20 11:58:58 failed while reading from pipe.
10/20/20 11:58:58 Read so far: Submitting job(s)ERROR on Line 13 of submit file: $CHOICE() macro: myindex is invalid index!
10/20/20 11:58:58 ERROR: submit attempt failed
  • Perhaps use requirements.  Greg will send an example
    • SOLUTION:
    • DAG:
      • JOB step05 step05.htc
      • #VARS step05 SITE="chtc"
      • #VARS step05 SITE="aws"
    • Submit:
      • +NRAOAttr = "$(SITE)"
      • Requirements = My.NRAOAttr == "chtc" ? PoolName == "CHTC" : PoolName =!= "CHTC"

      • Requirements = My.NRAOAttr == "chtc" ? (Target.HasCHTCStaging == true) : (Target.HasCHTCStaging =!= true)

      • myannex = "krowe-annex"
      • +MayUseAWS = True

      • Requirements = My.NRAOAttr == "aws" ? AnnexName == $(myannex) : AnnexName =!= $(myannex)

    • I would set myannex in the DAG but when I do that it tries to find an AnnexName of "krowe - annex" (note spaces)
  • ANSWER: My conclusion is that there are limitations on what one can do with variables in the submit file that were defined in the DAG file.
  • Is there a config option that will cause condor to not start?  We have diskless nodes and it is easier to modify the config file then change systemd.
    • SOLUTION: Either set START_MASTER = False or START_DAEMONS = False depending on desired outcome.
  • Torque has this command called pbsnodes that can not only offline/drain a node but keeps a note about it that all can see in one place.  I know I can use condor_off to drain a node but is there a central place keep notes so I can remember a month later why I set a certain node to drain?
    • ANSWER: there is no place to keep such notes but Greg likes the idea and may look into it.
    • May want to use condor_drain instead of condor_off.  condor_off will kill the startd when all jobs finish and it no longer shows up in condor_status.  condor_drain will leave the node in condor_status.
    • condor_drain doesn't work for me because it immediatly sets jobs idle instead of letting them run to completion.  This is why I use condor_off -startd -peaceful instead.
  • How can you tell which job is associated with an email given the email message doesn't include a working dir or the assigned batch_name?
    • CHTC will look into adding such information to the email condor sends.