GPU Cluster Utilization Governance: Fragmentation, Queuing, and Scheduling Efficiency

Note: This is a review article (no first-hand experimental data), based on public scheduling practice and operations experience.

Experiment Objective

The biggest hidden waste in a large cluster is not "broken machines" but "machines idle while jobs queue" — a cluster at 60% GPU utilization usually has a scheduling problem, not a GPU problem. Goal: a methodology for utilization governance — first define "what utilization actually means" (the definition decides everything), then diagnose the four causes of fragmentation and queuing (over-request, granularity mismatch, priority chaos, occupy-without-run), then governance actions: job packing, Backfill, priority and QoS. After reading you can answer "why is my cluster utilization low, and where do I look first."

Code Walkthrough

Part 1: Utilization definitions and diagnostic scripts

# Definition 1: GPU time utilization (the most common "utilization")
# = Σ(time GPU occupied by jobs) / (GPU count × window)
# Data source: sacct / SLURM accounting
sacct -S 2026-10-01 -E 2026-10-08 --format=JobID,AllocNodes,ReqTRES,AllocTRES,Start,End,State

# Definition 2: GPU space utilization (instant snapshot)
# = allocated GPUs at a moment / total GPUs
squeue -o "%i %C %t %R" | head -20   # running + queued now

# Definition 3: compute utilization (MFU's sibling, training side)
# = actual compute / theoretical compute, from DCGM SM utilization (ch02)

Key points:

1. All three definitions are needed: time utilization shows "is the cluster full," space utilization shows "is it empty right now," compute utilization shows "are the occupied cards actually busy" — saying "70% utilization" without stating the definition is meaningless.
2. sacct is the authoritative source for time utilization: SLURM's accounting DB records each job's request/allocation/start/end, making statistics reproducible.
3. Diagnostic trio: low time utilization → few jobs / scheduling-policy problems; high time but low space → jobs too coarse (an 8-GPU job holding 8 cards but using 4 cards' compute); high space but low compute → occupy-without-run (model loaded but not serving, jobs stuck on IO).

Part 2: Backfill and job packing (governance examples)

# SLURM config snippet: Backfill + partition granularity
PartitionName=gpu Nodes=ALL DefaultTime=24:00:00 MaxTime=120:00:00
  MaxNodes=32 State=UP
# Backfill lets small jobs fill gaps while big jobs wait (default on)
SchedulerParameters=bf_continue,bf_window=3600
# Granularity governance: min/max cards per job, preventing 8-card jobs blocking racks
MaxTRESPerNode=gpu:8

Key points:

• Backfill is the #1 utilization contributor: without it, while a 32-GPU job waits, the cluster would rather idle than run a 4-GPU small job; with it, small jobs fill the big job's waiting window and space utilization rises immediately.
• Job granularity (MaxTRESPerNode/MinTRES): cap cards per job to stop "an 8-card job stuck on 7-card headroom" fragmentation; MinTRES prevents tiny jobs from burning scheduling overhead.
• Backfill window (bf_window): Backfill only fills windows known to open; too long makes small jobs wait, too short under-fills — set 30-60 minutes by typical job duration.

Principles

Four low-utilization causes and governance

Cause	Signature	Governance
over-request	job requests 8 cards, uses 4 cards' compute	verify real demand, train users, quota constraints
granularity mismatch	big-job fragmentation, small-job under-fill	Backfill, job packing, granularity limits
priority chaos	long jobs squat, short jobs starve	priority = fair + backfill, QoS tiers
occupy-without-run	loaded but not serving, stuck on IO/comm	DCGM alerting (ch02), timeout reclaim

Core insight: utilization governance is "scheduling policy × monitoring feedback × user habits." Scheduling policy (Backfill/QoS) is the lever, but without monitoring feedback (which jobs occupy-without-run) you cannot govern directionally — ch02's SM-utilization metrics are the data foundation here.

Governance loop

periodic statistics from sacct/squeue (weekly)
  → attribute by the three definitions (time/space/compute)
  → locate the dominant cause (fragmentation/queuing/occupy-without-run)
  → apply the matching action (Backfill/granularity/QoS/alerting)
  → re-measure next period → iterate

Governance is not one-shot: re-measure two weeks after turning on Backfill — utilization gains can be canceled by user application habits (scheduling got smoother, so users over-request more), requiring quotas and training together.

Results

(Estimated, not measured)

Action	Space-utilization gain	Cost
enable Backfill	+10%-25%	scheduler CPU, queue prediction
granularity limits	+5%-15%	users must change habits
QoS tiers (interactive/batch)	+5%-10%	config complexity
occupy-without-run monitoring + reclaim	+5%-20%	needs monitoring linkage

Engineering Takeaways

1. Define first: unify the three utilization definitions before discussing governance; weekly reports must state the definition — otherwise "improvement" is unverifiable.
2. Backfill is the best-ROI first move: enable Backfill and bf_window first; most clusters gain a step in space utilization immediately.
3. Govern with data, not feelings: export sacct periodically and group analysis by granularity/duration/priority to find the dominant "bad job" class and govern it directionally.
4. Connect to observability: occupy-without-run is only discoverable via SM utilization (ch02 DCGM) — utilization governance and cluster observability must share one data set.

Next article preview: utilization governance depends on "scheduling + monitoring"; heterogeneous scheduling opened in ch07; next we add the AMD side — "ROCm and the AMD GPU Training Ecosystem: From Porting to Tuning."