K8s AI Scheduling Ledger: GPU Sharing, Topology & Elasticity

Slurm manages batch jobs, K8s manages microservices — but when AI training/inference runs on K8s, "how GPUs are split, given, and placed" becomes the core problem. This article settles three ledgers of K8s AI scheduling: resource abstraction, topology awareness, elasticity, linking to K8s GPU Scheduling Practice.

1. Resource Abstraction Ledger: From Whole Card to Fragments

K8s natively treats GPUs as "integer resources"; AI needs three abstractions:

Layer	Mechanism	Granularity	Typical impl
Whole-card allocation	Device Plugin + extended resources	starts at 1 card	NVIDIA Device Plugin
Memory slicing	MIG / vGPU	within-card slices	MIG, virtualization
Time-slice sharing	GPU time-slice reuse	multi-pod per card	time-slice plugin

Takeaway: whole-card is simplest (good isolation, no contention); slicing fits "big card, small model" fragmentation; time-slicing fits latency-insensitive tasks but risks contention (see MIG Partitioning and CUDA MPS).

2. Topology-Aware Scheduling Ledger: Wrong Placement Wastes Comm

Multi-card training is extremely sensitive to card-node-switch placement (see Topology & Comm Ledger):

Pod requests 8 cards
   ├─ 8 cards on one node (NVLink/PCIe, best)
   ├─ 2 nodes × 4 cards in one rack (cross-node, ok)
   └─ 4 nodes × 2 cards across racks (cross-switch, worst)

K8s solutions:

• Node level: node labels + nodeSelector/affinity to pin training pods to one node pool;
• Pod level: topologySpreadConstraints to co-locate 8 cards;
• Scheduler level: Volcano / topology-aware schedulers scoring by "GPU topology + network distance".

Takeaway: multi-card training must use topology-aware means (labels/affinity/scheduler), or collectives amplify latency across switches; estimate the loss with the traffic model in Collective Communication.

3. Elasticity Ledger: Inference Elastics, Training Stays

Scenario	Scaling strategy	Key constraints
Inference (online)	HPA/KEDA by QPS, GPU utilization	cold start sec-min, needs warm pool
Training (batch)	queue + priority + preemption	mid-task shrink = restart from checkpoint, careful

Takeaway: inference fits elasticity, training doesn't — training elasticity's value is "filling fragment time with small jobs" via queues (Volcano Queue / Kueue) for fair scheduling & preemption, not crude scaling.

4. Positioning vs Slurm

	Slurm	K8s
Model	batch queue (job-partition-QOS)	declarative Pod/Deployment
GPU slicing	manual MIG/vGPU binding	Device Plugin unified reporting
Elasticity	weak (job granularity)	strong (HPA/KEDA)
Ecosystem	HPC traditional stack	microservice/cloud-native stack

Takeaway: prefer Slurm for training (stable, mature queues, see Slurm User Guide), K8s for inference (elastic, rolling updates, see K8s + vLLM Production) — coexistence is the norm in AI computing centers.

5. Reproducible Python: Topology Gain Estimate

def topology_comm_penalty(n_gpus_per_node, n_nodes, bw_nvlink_gbps, bw_net_gbps, msg_mb):
    """8-card AllReduce: intra-node vs cross-node time (ms, approx)"""
    intra = msg_mb * (n_gpus_per_node - 1) / (bw_nvlink_gbps / 8) / 1e3
    inter = msg_mb * (n_nodes - 1) / (bw_net_gbps / 8) / 1e3
    return round(intra, 2), round(inter, 2)

topology_comm_penalty(8, 2, 400, 200, 16)
# intra-node 0.28 ms vs cross-node 1.2 ms → cross-node comm ≈ 4.3× time
# note: when cards > single-node capacity, prefer same-rack aggregation, avoid cross-switch

6. Checklist

1. Pin training pods to the same node/rack with node affinity;
2. Inference: HPA/KEDA elasticity + warm pool; training: queue + priority, never mid-task shrink;
3. Evaluate MIG/slicing for fragmented VRAM; time-slicing only for latency-insensitive tasks;
4. For 8+ card multi-node training, use Volcano/topology schedulers scoring by network distance;
5. After launch, measure comm ratio (see Communication Ledger) and scheduling success rate.

---

Organized and written by this site for learning purposes; formulas and examples are the site's own; specific versions and measurements are subject to on-site conditions.