...
- define how SSA uses Torque/Moab and what they should do for Slurm
Why can't we implement nodescheduler in Slurm?
In a word, uniqueuser.
Moab has an option to qsub (-l naccesspolicy=uniqueuser) that
prevents the job from running on a node where the user is already
runing a job. This allows my /etc/cgrules.conf hack to add any
shells beloging to a specific user to the cgroup of the job running
on the node without needing to choose which of the user's job to
select.
I don't see a similar function in Slurm so there is no way for my
/etc/cgrules.conf hack to put shells into the right cgroup.
We could have nodescheduler check the nodes and assign one that the
user isn't currently using but this is starting to turn
nodescheduler into a scheduler of its own and I think may be more
complication than we want to maintain. It would also have to
find nodes with all the other requirements (free cores, free mem, etc)
pam_slurm_adopt.so seems like it might help because it moves shells
into the cgroup of the user's job expected to run the longest. That's
nice but now if a user has two interactive jobs on the same node, all
shells will be put in the cgroup of just one of the jobs thus never
utilizing the resources reserved for the other job.
srun --jobid seems like it might help because it logs a user into a
running reservation based on the jobid. But, it doesn't tunnel X11
reliably and only allows for one connection to a job at a time.