MPI + Slurm in Practice: Scheduling and Performance Tuning

1. Scenario: From Manual mpirun to Cluster Jobs

In the previous article we always ran mpirun by hand. On a real cluster, Slurm schedules jobs: allocates nodes, binds resources, and queues execution. This article runs both submission paths (interactive srun and batch sbatch), aligns resource parameters with MPI process layout, and closes with an HPL tuning retrospective from measured data.

2. Path 1: srun

--ntasks sets the total process count; --ntasks-per-node sets processes per node (the counterpart of mpirun -perhost):

# 8 nodes, 8 processes per node, 64 MPI processes total
srun -N 8 --ntasks-per-node=8 -p hpc ./xhpl

# or specify the total explicitly
srun -N 8 --ntasks=64 -p hpc ./xhpl

Key point: Slurm already generates the allocated node list (slurm.hosts), so no hostfile is needed; srun starts MPI processes according to the allocation.

3. Path 2: sbatch Batch Script

#!/bin/bash
#SBATCH -J hpl-run
#SBATCH -N 8
#SBATCH --ntasks-per-node=8
#SBATCH -p hpc
#SBATCH -o hpl_%j.log
#SBATCH --exclusive

module load mpi
export I_MPI_FABRICS=shm:ofi
export FI_PROVIDER=tcp
export I_MPI_PIN=1

mpirun -np 64 -genv I_MPI_FABRICS shm:ofi -genv FI_PROVIDER tcp ./xhpl

Submit with sbatch run_hpl.sh. The job enters the queue; check with squeue, review history and timing with sacct.

4. Process Layout and Pinning: The First Gate to Performance

Where MPI processes land and which network path they take directly determines performance:

Setting	Role	Practical advice
--ntasks-per-node ( / -perhost)	processes per node	match CPU topology (e.g., for 104-core nodes pick divisors like 8-13)
I_MPI_FABRICS=shm:ofi	shared memory within a node, OFI across nodes	correct default for mixed fabric
FI_PROVIDER=tcp	OFI provider	use verbs/mlx for IB, efa/tcp depending on RoCE hardware
I_MPI_PIN=1	pin processes to cores	prevent migration, reduce cache thrashing

The native measured form of the Intel MPI run command:

mpirun -perhost ${MPI_PER_NODE} -np ${MPI_PROC_NUM} \
  -genv I_MPI_FABRICS shm:ofi -genv FI_PROVIDER tcp \
  -hostfile hostfile

5. Tuning Retrospective: 104-Core Single Node → 80 Nodes

Measured cluster: 104 cores per node, ~2.0 GHz, 1.5 TB memory per node.

Single-node theoretical peak (Rpeak):

Rpeak = 104 cores × 2.0 GHz × 32 FLOPs/cycle = 6656 Gflops

80-node HPL measured parameters:

Parameter	Measured	Note
N (matrix size)	3,145,728	~80%-90% of memory
NB (block size)	384	balance cache and communication granularity
Grid P×Q	10 × 8	80 processes, as square as possible
BCASTs	0 (1rg)	1-level recursive broadcast
lookahead	0	pipeline depth

Efficiency curve from single node to multi-node:

Scale	Key practice	Observation
1 node (104 cores)	all cores busy + pinning	highest efficiency; shm means zero network cost
4-8 nodes	square up P×Q, larger NB	network becomes a secondary bottleneck
36 nodes (GPU+CPU)	separate pools, CPU jobs independent	mixed loads must avoid resource contention
80 nodes	N scales with memory, P×Q matches topology	efficiency drops but total throughput rises

Three reusable conclusions:

1. Square the grid first, then scale it: the closer P×Q is to a square, the fewer communication steps; the grid should match per-node core counts (10×8 = 80 processes = 8 nodes × 10 processes).
2. Choosing the network provider matters more than tuning parameters: shared memory within a node, direct transport across nodes; a wrong FI_PROVIDER (e.g., forcing verbs on a TCP environment) raises latency by an order of magnitude.
3. Memory buys N, communication buys shape: N defines problem size (larger is more efficient); NB/P×Q/BCAST define communication efficiency. Validate the shape at small scale first, then enlarge N.

6. Troubleshooting Quick Reference

Symptom	Check
mpirun: command not found	module load mpi or source /opt/intel/oneapi/setvars.sh
Job stuck in queue	squeue -u $USER for priority/partition; --exclusive waits for whole idle nodes
Cross-node latency abnormally high	confirm FI_PROVIDER matches hardware; compare small-message ping-pong
Results change with process count	check MPI_Reduce type lengths (MPI_LONG_LONG), whether P×Q divides evenly
Sudden efficiency drop	check I_MPI_PIN=0 migration, NB too small, lookahead too deep