...
Interactive: The ability to assign all or part of a node to a user with shell level access (nodescheduler, qsub -I, etc), minimal granularity is per NUMA node, finer would be useful.
- nodescheduler: Was written before I understood what qsub -I did. Had I known, I may have argued to use that instead of nodescheduler as it is much simpler, is consistent with other installations of Torque, and may have forced some users to use batch processing which is way more efficient. My current thought is to ditch nodescheduler if we switch to slurm and/or htcondor. Users could just request interactive shells or perhaps nodevnc could be a replacement.
- nodescheduler likes
- It's not tied to any tty so a user can login multiple times from multiple places to their reserved node without requiring something like screen, tmux, or vnc. It also means that users aren't all going through nmpost-master.
- Its creation is asynchronous. If the cluster is full you don't wait around for your reservation to start, you get an email message when it is ready.
- It's time limited (e.g. two weeks). We might be able to do the same with a queue/partition setting but could we then extend that reservation?
- We get to define the shape of a reservation (whole node, NUMA node, etc). If we just let people use qsub -I they could reserve all sorts of sizes which may be less efficient. Then again it may be more efficient. But either way I think nodescheduler it is simpler for our users.
- It's not tied to any tty so a user can login multiple times from multiple places to their reserved node without requiring something like screen, tmux, or vnc. It also means that users aren't all going through nmpost-master.
- nodescheduler: dislikes
- With Toruqe/Moab asking for a NUMA node doesn't work as I would like. Because of bugs and limitations, I still have to ask for a specific amount of memory. The whole point of asking for a NUMA node was that I didn't need to know the resources of a node ahead of time but could just ask for half of a node. Sadly, that doesn't work with Torque/Moab.
- Because of the way I maintain the cgroup for the user, with /etc/cgrules.conf, I cannot let a user have more than one nodescheduler job on the same node or it will be impossible to know which cgroup an ssh connection should use. The interactive commands (qsub -I, etc) don't have this problem.
- Slurm
- srun --pty bash This logs the user into an interactive shell on a node with defaults (1 core, 1 GB memory)
- I don't see how Slurm can reserve NUMA nodes so we will may have to just reserve X tasks with Y memory.
- I don't know how to keep Slurm from giving a user multiple portions of the same host. With Moab I used naccesspolicy=uniqueuser which prevents the ambiguity of which ssh connection goes to which cgroup.
- srun --pty bash This logs the user into an interactive shell on a node with defaults (1 core, 1 GB memory)
- HTCondor
- condor_submit -i This logs the user into an interactive shell on a node with defaults (1 core equivelent, 0.5 GB memory)
- I don't see how Slurm can reserve NUMA nodes so we may have to just reserve X tasks with Y memory.
- Could run a sleep job just like we do with Torque and use condor_ssh_to_job which seems to do X11 properly. We would probably want to make gygax part of the nmpost pool.
- I don't know how to keep htcondor from
- think I need to worry about giving a user multiple portions of the same host . With Moab I used naccesspolicy=uniqueuser which prevents the ambiguity of which ssh connection goes to which cgroup
- if we are using condor_ssh_to_job.
- nodevnc
- Given the limitation of Slurm and HTCondor and that we already recommend users use VNC on their interactive nodes, why don't we just provide a nodevnc script that reserves a node (via torque, slurm or HTCondor), start a vnc server and then tells the user it is ready and how to connect to it? If someone still needs/wants just simple terminal access, then qsub -I or srun --pty bash or condor_submit -i might suffice.
- Given the limitation of Slurm and HTCondor and that we already recommend users use VNC on their interactive nodes, why don't we just provide a nodevnc script that reserves a node (via torque, slurm or HTCondor), start a vnc server and then tells the user it is ready and how to connect to it? If someone still needs/wants just simple terminal access, then qsub -I or srun --pty bash or condor_submit -i might suffice.
- nodescheduler: Was written before I understood what qsub -I did. Had I known, I may have argued to use that instead of nodescheduler as it is much simpler, is consistent with other installations of Torque, and may have forced some users to use batch processing which is way more efficient. My current thought is to ditch nodescheduler if we switch to slurm and/or htcondor. Users could just request interactive shells or perhaps nodevnc could be a replacement.
Access: Would like to prevent users from being able to login to nodes unless they have a proper reservation.
- Slurm
- Has a pam_slurm.so module similar to pam_pbssimpleauth.so.
- HTCondor
- How do we restrict access to condor nodes to only those users with valid jobs running? Since I don't think we will be using nodescheduler with HTCondor, this isn't needed.
- How do we restrict access to condor nodes to only those users with valid jobs running? Since I don't think we will be using nodescheduler with HTCondor, this isn't needed.
- Slurm
Reservations: The ability to reserve nodes far in the future for things like CASA classes and SIW would be very helpful. It would need to prevent HTCondor from starting jobs on these nodes as reservation time approaches.
- Slurm
- scontrol create reservation starttime=now duration=5 nodes=testpost001 user=root
- scontrol create reservation starttime=2022-05-3T08:00:00 duration=21-0:0:0 nodes=nmpost[020-030] user=root reservationname=siw2022
- scontrol show res The output of this kinda sucks. Hopefully there is a better way to see all the reservations.
- HTcondor
- This isn't really something HTCondor is designed to do. We will use Slurm for this.
- This isn't really something HTCondor is designed to do. We will use Slurm for this.
- Slurm
...