How-to: Scheduling via SLURM
Action
To schedule research computing work using SLURM, follow the instructions below.
Instructions
- 1 SLURM
- 2 General Purpose Computing
- 3 Frequently asked questions
- 3.1 View the current status, or resources available, of batch nodes
- 3.2 View jobs currently running, and waiting in queue
- 3.2.1 viewing the job queue
- 3.3 View the resources requested for an active job
- 3.4 Maximum resources allowed
- 3.5 Request access to more nodes, or a longer time limit
- 3.6 Schedule a non-interactive job
- 3.7 Schedule an interactive job
- 3.8 View the resources used by a completed job
- 3.9 Restrict a job to a certain CPU architecture
- 3.10 Spawn on the infiniband nodes
- 3.11 Allocate GPU resources
- 3.12 Run jupyter notebook on the cluster
SLURM
All jobs on the general purpose cluster request resources via SLURM. SLURM, is open source software that allocates resources to users for their computations, provides a framework for starting, executing and monitoring compute jobs, and arbitrates contention for resources by managing a queue of pending work. SLURM is widely used in the high performance computing (HPC) landscape and it is likely you will encounter it outside of our systems. For more information please see https://slurm.schedmd.com/
General Purpose Computing
All resources on the general purpose cluster are submitted using the SLURM scheduler. For more information, please read the Frequently asked Questions. Jobs can be submitted from the following head nodes:
head.its.albany.edu
Or from the large memory machine:
lmm.its.albany.edu
Resource information
All users have access to the "batch" partition for general purpose computing.
The batch partition is comprised of 1040 CPU cores (2080 threads) and 31 compute nodes. Note that a job can only request 3 nodes and may only be active for 14 days. If you need an exception to this, please contact askIT@albany.edu
$ sinfo -p batch -o "%n, %c, %m" | sort
HOSTNAMES, CPUS, MEMORY
uagc20-01, 64, 186415
uagc20-02, 64, 186415
uagc20-03, 64, 186415
uagc20-04, 64, 186415
uagc20-05, 64, 186415
uagc20-06, 64, 186415
uagc20-07, 64, 186415
uagc20-08, 64, 186415
uagc20-09, 64, 186415
uagc20-10, 64, 186415
uagc20-11, 64, 186415
uagc20-12, 64, 186415
uagc20-13, 64, 186415
uagc20-14, 64, 186415
uagc20-15, 64, 186415
uagc21-01, 80, 380500
uagc21-02, 80, 380500
uagc21-03, 80, 380500
uagc21-04, 80, 380500
uagc21-05, 80, 380500
uagc21-06, 80, 380500
uagc21-07, 80, 380500
uagc21-08, 80, 380500
uagc23-01, 128, 380681
uagc23-02, 128, 380681
uagc23-03, 128, 380681
uagc23-04, 128, 380681
uagc23-05, 128, 380681
uagc24-01, 128, 510949
uagc24-02, 128, 510949
uagc24-03, 128, 510949
uagc24-04, 128, 510949
uagc24-05, 128, 510949
uagc24-06, 128, 510949
Frequently asked questions
SLURM documentation can be found at the SLURM website (https://slurm.schedmd.com); but below are answers to frequently asked questions which demonstrate several useful SLURM commands.
View the current status, or resources available, of batch nodes
sinfo is commonly used to few the status of a given cluster or node, or how many resources are available to schedule.
Viewing available resources
-bash-4.2$ sinfo -p batch -o "%n, %a, %C, %e, %O" | sort
HOSTNAMES, AVAIL, CPUS(A/I/O/T), FREE_MEM, CPU_LOAD
rhea-09, up, 0/0/96/96, N/A, N/A
rhea-10, up, 0/0/96/96, N/A, N/A
uagc19-01, up, 2/38/0/40, 93621, 0.00
uagc19-02, up, 12/28/0/40, 54240, 15.90
uagc19-03, up, 14/26/0/40, 67920, 17.12
uagc19-04, up, 36/4/0/40, 75889, 16.60
uagc20-01, up, 0/64/0/64, 189368, 0.00
uagc20-02, up, 0/64/0/64, 189359, 0.00
uagc20-03, up, 0/64/0/64, 189367, 0.00
uagc20-04, up, 0/64/0/64, 189461, 0.00
uagc20-05, up, 32/32/0/64, 98151, 32.09
uagc20-06, up, 32/32/0/64, 97446, 32.10
uagc20-07, up, 2/62/0/64, 188191, 0.00
uagc20-08, up, 32/32/0/64, 94065, 32.11
uagc20-09, up, 40/24/0/64, 180208, 48.21
uagc20-10, up, 64/0/0/64, 8985, 64.35
uagc20-11, up, 0/64/0/64, 189303, 0.00
uagc20-12, up, 64/0/0/64, 9337, 64.24
uagc20-13, up, 24/40/0/64, 176151, 0.00
uagc20-14, up, 0/64/0/64, 189364, 0.00
uagc20-15, up, 0/64/0/64, 189343, 0.02
uagc21-01, up, 80/0/0/80, 278987, 1.47
uagc21-02, up, 20/60/0/80, 371334, 0.50
uagc21-03, up, 0/80/0/80, 381238, 0.00
uagc21-04, up, 0/80/0/80, 381046, 0.00
uagc21-05, up, 0/80/0/80, 290550, 0.00
uagc21-06, up, 80/0/0/80, 339206, 25.01
uagc21-07, up, 80/0/0/80, 247070, 4.17
uagc21-08, up, 80/0/0/80, 338115, 25.03
uagc21-09, up, 0/80/0/80, 380172, 0.75
uagc21-10, up, 0/80/0/80, 380682, 0.20
uagc21-11, up, 0/80/0/80, 381265, 0.00
uagc21-12, up, 0/80/0/80, 260703, 0.00Note that %a reports CPUS as allocated/idle/other/available. In this example, uagc20-10 has all of it's threads allocated (64 out of 64), and is showing a CPU load of 64.30 (or that 64.30 threads are active). Whereas, many of the other nodes have lower utilization. We can use this information to make smart decisions about how many resources we request.
View jobs currently running, and waiting in queue
squeue will show jobs currently waiting in the queue or running, for all partitions that you have access to.
viewing the job queue
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
140574 batch g.slurm [netid] PD 0:00 1 (Resources)
140486 batch g.slurm [netid] R 21:53:54 1 rhea-04
140290 batch run.sh [netid] R 2-19:09:35 1 rhea-01
140216 batch shell1_5 [netid] R 3-08:48:18 1 rhea-09
135093 batch g.sh [netid] R 28-19:56:31 1 rhea-08
135087 batch g.sh [netid] R 28-20:43:49 1 rhea-10
135090 batch g.sh [netid] R 28-20:49:42 1 rhea-07At the time this command was run, there were 7 jobs running or waiting in queue. JOBID 140574 is waiting in the queue due to inadequate available resources, while the other jobs have been running for a few days.
View the resources requested for an active job
scontrol show job [jobid] will generate a report with information about how a job was scheduled.
Note that once a job is completed, this report can no longer be generated via scontrol. See How do I view the resources used by my job? for accessing similar information upon job completion.
Viewing a running job's allocation
$ scontrol show job ######
JobId=###### JobName=g.slurm
UserId=[netid](52639) GroupId=faculty(972) MCS_label=N/A
Priority=1 Nice=0 Account=rit QOS=normal
JobState=PENDING Reason=Resources Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:00 TimeLimit=14-00:00:00 TimeMin=N/A
SubmitTime=2019-02-13T07:48:25 EligibleTime=2019-02-13T07:48:25
StartTime=2019-02-14T11:53:10 EndTime=2019-02-28T11:53:10 Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
LastSchedEval=2019-02-13T08:42:58
Partition=batch AllocNode:Sid=headnode7:86819
ReqNodeList=(null) ExcNodeList=(null)
NodeList=(null)
NumNodes=1 NumCPUs=32 NumTasks=1 CPUs/Task=32 ReqB:S:C:T=0:0:*:*
TRES=cpu=32,mem=87.50G,node=1
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=32 MinMemoryCPU=2800M MinTmpDiskNode=0
Features=avx2 DelayBoot=00:00:00
Gres=(null) Reservation=(null)
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/path/to/command/
WorkDir=/path/to/workdir/
StdErr=/path/to/stderr/
StdIn=/dev/null
StdOut=/path/to/stdout/
Power=Here, the job requested 32 CPUs on one node, with 87.5GB of memory, at 2019-02-13T07:48:25, with a constraint of Features=avx2.
NumNodes=1 NumCPUs=32 NumTasks=1 CPUs/Task=32 ReqB:S:C:T=0:0:*:* TRES=cpu=32,mem=87.50G,node=1
Features=avx2
Maximum resources allowed
$ scontrol show partition batch
PartitionName=batch
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
AllocNodes=ALL Default=YES CpuBind=none QoS=generaluse
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
MaxNodes=6 MaxTime=14-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
Nodes=uagc20-[01-15],uagc21-[01-04,06-08],uagc23-[01-05],uagc24-[01-06]
PriorityJobFactor=100 PriorityTier=100 RootOnly=NO ReqResv=NO OverSubscribe=FORCE:1
OverTimeLimit=NONE PreemptMode=OFF
State=UP TotalCPUs=2928 TotalNodes=33 SelectTypeParameters=NONE
JobDefaults=(null)
DefMemPerCPU=1500 MaxMemPerNode=UNLIMITED
TRES=cpu=2928,mem=10428824M,node=33,billing=2928batch has some important restrictions. A job can only request 3 nodes and will run for 14 days before being automatically terminated. If you need an exception to this rule, please contact askIT@albany.edu
Request access to more nodes, or a longer time limit
On a case by case basis, ITS will grant users temporary access to more than the default job limitations. Please contact askIT@albany.edu if you would like to request access to more nodes, or a longer time limit.
Schedule a non-interactive job
There are many ways to schedule jobs via slurm. For non-interactive jobs, we recommend using sbatch with a shell script that runs your script. We will use #SBATCH commands to allocate the appropriate resources required for our script. Below is an example workflow of how to submit a python script via sbatch to batch.
First ssh into head.its.albany.edu. On windows, you can use an ssh client such as PuTTY, on mac, simply use the terminal. Replace [netid] below with your username and type in your password at the prompt. You will not see your password, but it is being typed.
$ ssh [netid]@head.its.albany.edu Warning: Permanently added the ECDSA host key for IP address '169.226.65.82' to the list of known hosts. [netid]@head.its.albany.edu's password: Warning: No xauth data; using fake authentication data for X11 forwarding. Last login: Wed Jan 30 13:49:20 2019 from lmm.its.albany.edu ================================================================================ This University at Albany computer system is reserved for authorized use only. http://www.albany.edu/its/authorizeduse.htm Headnodes: head.its.albany.edu General Purpose Computing: lmm.its.albany.edu - Large memory x2go headnode: eagle.arcc.albany.edu Questions / Assistance - askIT@albany.edu ================================================================================Next, change directories to /network/rit/misc/software/examples/slurm/
$ cd /network/rit/misc/software/examples/slurm//network/rit/misc/software/examples/slurm/run.sh contains #SBATCH commands that will request the appropriate amount of resources for our python code, then execute the code.
$ more run.sh #!/bin/bash #SBATCH -p batch #SBATCH --cpus-per-task=4 #SBATCH --mem-per-cpu=100 #SBATCH --mail-type=ALL #SBATCH -o /network/rit/home/%u/example-slurm-%j.out # Now, run the python script /network/rit/misc/software/examples/slurm/simple_multiprocessing.pyTo submit the job, we simply run sbatch run.sh. Keep note of the Job ID that is output to the terminal, it will be different that what is shown below.
$ sbatch run.sh Submitted batch job 140584The job will output a file to your home directory called ~/example-slurm-[jobid].out. We will view it using the "more" command. You should see output similar to below.
$ more ~/example-slurm-140584.out USER [netid] was granted 4 cores and 100 MB per node on [hostname]. The job is current running with job # [jobid] Process D waiting 3 seconds Process D Finished. Process C waiting 1 seconds Process C Finished. Process E waiting 4 seconds Process E Finished. Process A waiting 5 seconds Process A Finished. Process B waiting 2 seconds Process B Finished. Process F waiting 5 seconds Process F Finished.Congratulations, you just ran your first job on the cluster!
Schedule an interactive job
To spawn a terminal session on a cluster node run:
srun --partition=batch --nodes=1 --time=01:00:00 --cpus-per-task=2 --mem=400 --pty $SHELL -iTo spawn a terminal on the DGX cluster, you’ll also need to pass progate rules as such:
srun --propagate=NONE --nodes=1 --time=01:00:00 --cpus-per-task=2 --mem=40gb --pty $SHELL -i