Slurm Learning Path: From Installing It to Shipping a Job

What this page is: not an introduction to Slurm, and not another install guide. It is a signpost: the five Slurm articles already on this site, arranged in the order you actually work through them, each stop giving a goal, which article to read, which command to run, and how to know you are done.

If you only need to submit jobs and do not touch the cluster, jump to Stage 3.

Three starting points, three different first pages

Where you are	Start at	Why
A rack of bare metal, no manager yet	Stage 1	Pick one of the two install guides (Rocky or Ubuntu) by distro
The cluster is already built and you just inherited it	Stage 2	Learn to observe before you change; do not open slurm.conf first
You only need to submit and manage jobs	Stage 3	The first two stages are noise to you

Stage 1: Bring up the controller and the compute side

Goal: slurmctld and slurmd both active, sinfo lists the nodes, none of them down.

Read: one of the two depending on distro — Rocky Linux 9 end to end or Ubuntu 22.04 + Slurm 24.05. Same skeleton, different package manager and systemd unit names.

Do (keep this order; changing it multiplies the debugging cost):

1. Fix hostnames and resolution first: the SlurmctldHost in slurm.conf must be a name hostname -s can resolve;
2. Unify munge: the same /etc/munge/munge.key on every node, owned by munge, mode 400. This is Slurm's only manual trust root — mismatched keys show up as nodes permanently down with nothing but an auth failure in the log;
3. Generate the node lines with slurmd -C instead of copying CPU and memory by hand. It reads the real values on that machine and avoids a too-small RealMemory silently degrading jobs;
4. Start the services last, and only then touch scheduling policy.

Accept:

systemctl status slurmctld slurmd --no-pager | grep -E "Active|slurm"
sinfo -N -l                       # one line per node; STATE must not be down/inval
scontrol show node node01 | grep -E "State|RealMemory|CfgTRES|Gres"

Trap: a node showing down* — the asterisk means someone set that state manually. Read sinfo -R for the reason, fix it, then scontrol update NodeName=node01 State=RESUME. Resuming without fixing the cause just puts it back to down on the next heartbeat.

Stage 2: Make the nodes actually usable (mandatory if you inherited it)

Goal: nodes idle, GPUs recognized as allocatable resources, jobs not killed for memory they believed they had.

Read: the first half of Slurm Job Submission and Management Guide for reading sinfo/scontrol/squeue; the slurm.conf section of the install article for configuration.

Four things to verify:

Check	Command	Expected
GPUs registered as GRES	scontrol show node node01 | grep Gres	a gpu:... count, not empty
Allocatable CPU / memory	sinfo -N -o "%N %c %m %e"	%c matches the real core count
cgroups in effect	scontrol show config | grep -E "ProctrackType|TaskPlugin"	proctrack/cgroup + task/cgroup
Accounting recorded	sacct -a -n | head	rows present; empty means slurmdbd is not wired in

The six parameters that decide whether it works (leave the rest at defaults):

SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory     # allocate by cores + memory together
GresTypes=gpu
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup
DefMemPerCPU=4000                       # default memory per core, so omitting --mem is not unlimited

cons_tres (consumable trackable resources) rather than the older cons_res is the prerequisite for allocating on cores, memory and GRES together.

Stage 3: Submit your first job

Goal: run a command interactively with srun, run a script with sbatch, and see what it is doing via squeue/scontrol.

Read: Slurm Job Submission and Management Guide.

Minimum working sequence:

# 1) smoke test: 2 nodes x 8 tasks x 1 GPU each
srun -N2 --ntasks-per-node=8 --gres=gpu:1 hostname

# 2) validate syntax without queueing (saves "queued two hours, then failed on a typo")
sbatch --test-only job.sbatch

# 3) submit and watch
sbatch job.sbatch
squeue -u "$USER" -o "%.10i %.20j %.10P %.2t %.10M %R"

A job.sbatch skeleton you can use as is:

#!/bin/bash
#SBATCH -J train-demo
#SBATCH -p gpu
#SBATCH -N 2
#SBATCH --ntasks-per-node=8
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=4
#SBATCH --mem=64G
#SBATCH -t 02:00:00
#SBATCH -o logs/%j.out
#SBATCH -e logs/%j.err

set -euo pipefail
module load cuda/12.4
srun python -u train.py --epochs 10

Accept: the job moves PD → R → CD, and sacct shows Elapsed/ExitCode/MaxRSS.

sacct -j <jobid> --format=JobID,JobName,Partition,Elapsed,State,ExitCode,MaxRSS

Trap: a job stuck in PD is usually not a queue problem but a request larger than the node actually has (--mem=128G on a 120G node, or --gres=gpu:2 on a node registering one GPU). The Reason field of scontrol show job <id> says so directly.

Stage 4: Make the job run right, not merely run

Goal: multi-node task distribution matches the request, CPU affinity does not drift, hybrid MPI does not idle.

Read: Slurm + Docker distributed training template for the container case; MPI specifics are in MPI with Slurm: scheduling and performance tuning; the per-node communication ceiling is in Collective Communication Primitives and Libraries.

Three frequent mistakes:

Symptom	Cause	Fix
16 tasks requested, one process doing work	process count comes from --ntasks, not from a Python loop	srun --ntasks=16, or use --ntasks-per-node
MPI hangs at startup	mpirun wrapped inside srun, tasks expanded twice	pick one: srun ./app, or mpirun with its bootstrap options
half the expected performance	processes and threads drifting across NUMA domains	--cpu-bind=cores, OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

A stable hybrid layout (2 MPI ranks per node × 8 threads, pinned):

srun --ntasks-per-node=2 --cpus-per-task=8 --cpu-bind=cores \
     bash -c 'export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK; ./solver'

Accept: srun hostname | sort | uniq -c matches the request; MaxRSS in sacct stays under the request; sstat -j <id> -a shows per-task CPU efficiency.

Stage 5: Make the environment reproducible (containers)

Goal: the same image gives the same result on any node, without root.

Read: Containerizing HPC Clusters: Apptainer SIF, GPU injection and Slurm integration; for Docker rather than Apptainer see the Docker-based training template.

The point: inside Slurm a container is just "the command after srun" — srun apptainer exec --nv x.sif python train.py. The real decisions are three: where the image lives (do not let 200 nodes pull the same file at once), which directories get mounted (--containall wipes the defaults), and whether the in-container driver generation matches the host.

Accept: srun apptainer exec --nv x.sif nvidia-smi lists a table for every allocated GPU, and the indices agree with CUDA_VISIBLE_DEVICES.

6. One-pass acceptance checklist

Tick in order; only then call the cluster "ready":

• [ ] sinfo -N -l: every node idle/mixed, none down, none inval
• [ ] scontrol show node <n> | grep Gres: GPU count matches reality
• [ ] srun -N1 --ntasks=1 hostname: single-node smoke test
• [ ] srun -N2 --ntasks-per-node=8 hostname: cross-node, task count correct
• [ ] sbatch --test-only job.sbatch: syntax and resource request valid
• [ ] a real job finishes and sacct shows Elapsed/ExitCode/MaxRSS
• [ ] deliberately scancel once; the job is reclaimed cleanly, no CG leftover
• [ ] (optional) slurmdbd connected; sshare shows account shares
• [ ] (optional) srun apptainer exec --nv x.sif nvidia-smi passes

7. Troubleshooting quick table

Symptom	Look here first	One-line fix
node down, log says auth failure	munge key and permissions	same key everywhere, owner munge, mode 400
every node shows down*	somebody set the state manually	fix the cause, then scontrol update ... State=RESUME
job sits in PD	Reason in scontrol show job	almost always a request above the node's real capacity
job instantly FAILED, non-zero ExitCode	%j.err	set -euo pipefail stops at the first error
MaxRSS above the request	cgroups not active	switch ProctrackType/TaskPlugin to the cgroup variants
GPU allocated but nvidia-smi fails	driver / container generation	in-container libs must match the host generation
slurmctld starts then exits	run slurmctld -Dvvv in the foreground once	a config parse error prints on the first screen

Three closing takeaways

• Order beats tricks: hostnames and keys, then node lines from slurmd -C, then services, then policy. Get the order wrong and every later step is cleaning up after an earlier one;
• sinfo and scontrol are your dashboard: every "the job won't run" question is answered in Reason, State or Gres;
• Done means accepted: srun smoke test, sbatch --test-only, sacct — those three passing are worth more than a page of slurm.conf.

Further Reading

• Rocky Linux 9 HPC Cluster Deployment, End to End
• Ubuntu 22.04 + Slurm 24.05 Cluster Deployment
• Slurm Job Submission and Management Guide
• Slurm in Practice: a Docker-based Distributed Training Template
• Containerizing HPC Clusters: Apptainer SIF, GPU Injection and Slurm Integration
• Collective Communication Primitives and Libraries: from MPI to NCCL

Related Articles

• Kubernetes GPU Scheduling: From Device Plugin to Shared GPUs
• Kubernetes Inference Service in Production: vLLM from Deployment to High Availability
• K8s AI Scheduling Ledger: GPU Sharing, Topology & Elasticity
• NVIDIA PyTorch Docker Deployment Guide