GPU Job Orchestration on Kubernetes: Kueue/Volcano Queues and Quotas

Note: This is a review article (no first-hand experimental data), based on public docs and deployment practice.

Experiment Objective

K8s's default scheduler is "online-service" thinking (Deployments start anytime), while training is "batch job" thinking (queuing, quotas, preemption, whole-card requests) — running training on bare K8s hits walls everywhere. Goal: dissect GPU job orchestration on K8s — the queue model of Kueue/Volcano (why queues are needed) → quotas and priorities (multi-tenant) → batch jobs and preemption → the SLURM correspondence. After reading you can answer: how to implement "SLURM-style" training queuing and quotas on K8s.

Code Walkthrough

Part 1: Kueue queues and quotas (recommended start)

# Kueue: resource quotas in "queues"; jobs queue for quota
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata: name: gpu-flavor
spec:
  nodeLabels: { cloud.google.com/gke-accelerator: nvidia-h100 }  # node selection

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata: name: team-a-cq
spec:
  namespaceSelector: { matchLabels: { team: a } }
  resources:
    - name: nvidia.com/gpu
      flavors: [ { name: gpu-flavor, resources: [ { name: nvidia.com/gpu, nominalQuota: 32 } ] } ]
  preemption: { reclaimWithinCohort: Any }  # preemption within the cohort

apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata: name: team-a-lq
spec: clusterQueue: team-a-cq

Key points:

1. The queue model solves "quotas + queuing": ClusterQueue defines "how many GPUs this team has"; LocalQueue points a namespace's jobs to the queue — training jobs no longer grab GPUs directly but queue for quota; the "SLURM partition" equivalent on K8s.
2. Preemption is job-level: reclaimWithinCohort lets low-priority jobs in the cohort be preempted by high-priority ones (with PreemptionPolicy) — "big jobs wait, small jobs run first," echoing ch02's Backfill thinking.
3. Kueue is a "lightweight queue layer": it does not change the K8s scheduler, only job admission/queuing/quota — simple to deploy, production-proven; the recommended start for training orchestration on K8s.

Part 2: Volcano — the batch scheduler

# Volcano: standalone scheduler supporting gang scheduling
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata: name: training-queue
spec: weight: 10  # queue weight (quota share)

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata: name: dl-train
spec:
  schedulerName: volcano
  minAvailable: 4          # gang: all 4 cards ready before start
  queue: training-queue
  tasks:
    - replicas: 4
      template: { spec: { containers: [{ name: t, image: pytorch:24.01 }] } }

Key points:

• Gang scheduling is training-specific: distributed training (DDP/FSDP/3D parallel) requires all cards ready simultaneously before starting; otherwise ready cards idle-wait or time out — minAvailable implements "whole-group scheduling," which the bare K8s scheduler cannot (linking ch06's k8s-gpu-scheduling article).
• Volcano is a "full scheduler": heavier than Kueue (takes over scheduling, affinity/topology awareness), fits "K8s must carry HPC workloads"; Kueue+default scheduler fits "lightweight start."
• They are not mutually exclusive: Kueue manages quota/queuing, Volcano manages gang scheduling — composable (Kueue's community supports Volcano integration).

Principles

The three gaps of running training on K8s

Gap	Symptom	Solution
no queues/quotas	jobs grab GPUs directly, no fairness	Kueue ClusterQueue
no gang scheduling	distributed jobs idle-wait on partial readiness	Volcano gang scheduling
no preemption/priority	long jobs squat, short jobs starve	preemption + priority classes

Core insight: K8s defaults to "instant start" (online services); training needs "queuing + whole-group + preemption" (batch) — Kueue/Volcano port SLURM's batch semantics into K8s. In mixed clusters (online inference + offline training on one K8s), queues/quotas let both load classes share GPUs fairly.

Correspondence with SLURM

SLURM	K8s + Kueue/Volcano
Partition	ClusterQueue / Volcano Queue
QoS (priority/preemption)	preemption + priorityClass
jobs (srun/sbatch)	Kueue Workload / Volcano Job
Backfill	inter-cohort preemption + quota elasticity
GRES (GPU resources)	nvidia.com/gpu extended resources

Migration insight: SLURM teams moving to K8s first map "partition→queue, QoS→preemption, job→Workload"; once the semantics transfer, ops habits (queuing/quotas/priority) reuse directly.

Results

(Estimated, not measured)

Option	Deploy complexity	Capability
bare K8s	low	no queues/gang; training pitfalls
Kueue	mid	quota/queuing/preemption; gang needs help
Volcano	mid-high	gang/topology; heavier
Kueue + Volcano	high	full capability (recommended at scale)

Engineering Takeaways

1. Start with Kueue; add Volcano on demand: quotas/queuing are most teams' need; Kueue is light; add Volcano when gang/topology needs appear.
2. Queues and quotas are the key to multi-tenant governance: per-team ClusterQueue, quotas, preemption — training jobs no longer "first-come-first-served," fairness is immediate.
3. Gang scheduling is the distributed-training floor: minAvailable = whole-group card count, avoiding "partial-ready idle wait" — mandatory for DDP/FSDP jobs.
4. Link with monitoring/cost: report utilization per queue (ch02) and cost (ch06's cost article) so quota adjustments have data.

Next article preview: with K8s job orchestration complete, ch06 and ch10 both advanced further; the next batch could return to ch04 storage or ch05 provisioning gaps, or pivot to SEO wrap-up. Your call.