Kubernetes GPU Scheduling: From Device Plugin to Shared GPUs

Summary: K8s does not natively understand GPUs—it treats them as "one allocatable integer resource". Device Plugin is what makes GPUs schedulable, and sharing/partitioning is what makes them efficient. This post explains: how Device Plugin works, how to choose among three shared-GPU options, how to monitor and quota, and the division of labor between K8s and Slurm.

0. Experiment Goals

• Understand the Device Plugin mechanism: how K8s exposes GPUs as schedulable resources
• Master three shared-GPU routes: Time-Slice / MIG / MPS
• Monitor GPU usage and quotas with Prometheus + DCGM
• Clarify the K8s–Slurm division of labor

1. Device Plugin: The Key to K8s Recognizing GPUs

How it works:

• The K8s scheduler only understands integer resources in resources.requests/limits (cpu/memory). To make GPUs schedulable, a Device Plugin (an official K8s mechanism) does three things:
  1. Reports how many GPUs the node has to Kubelet (ListAndWatch)
  2. Allocates/initializes the assigned GPU when a Pod lands (injecting CUDA_VISIBLE_DEVICES in Allocate)
  3. Monitors GPU health

# Pod using a GPU (standard form)
apiVersion: v1
kind: Pod
metadata:
  name: gpu-infer
spec:
  containers:
  - name: infer
    image: my-llm:latest
    resources:
      limits:
        nvidia.com/gpu: 1        # request 1 GPU (resource name exposed by the plugin)
    env:
    - name: NVIDIA_VISIBLE_DEVICES
      value: all                # plugin injects the actual visible GPU IDs

Engineering notes:

• NVIDIA Device Plugin is the de facto standard (DaemonSet); choose community or official per node GPU model.
• Pods request nvidia.com/gpu: N as whole GPUs; no "half GPU" by default—use the sharing options below.
• Naked risk: if the node has drivers but no Device Plugin, GPUs are invisible to K8s (nvidia.com/gpu = 0) and scheduling fails directly.

2. Three Shared-GPU Options: Time-Slice / MIG / MPS

Option	Mechanism	Isolation	Best for	K8s support
Time-Slice	multiple Pods share one GPU round-robin	weakest (no memory isolation)	low-load inference, dev/test	official plugin
MIG	hardware partition of SM/memory	strong (instance isolation)	multi-tenant, strong isolation	plugin supports MIG instances
MPS	multi-process shared context	medium (software sharing)	small-task concurrency throughput	custom scheduler extension

Step-by-step:

• Time-Slice: simplest; nvidia.com/gpu exposed as shares (e.g., 0.5 = round-robin). Risk: memory not isolated—one memory-hungry Pod can OOM others on the same card. Only for controlled environments.
• MIG: A100/H100 hardware partitioning (up to 7 instances); each MIG instance schedules as an independent resource. Strongest isolation but fixed config (instance specs cannot change dynamically); see MIG Partitioning.
• MPS: software sharing with the best small-task throughput (see CUDA MPS); K8s integration requires mounting the MPS pipe directory into Pods and injecting env vars.

Selection memory: time-slice for dev, MIG for tenants, MPS for throughput.

3. GPU Monitoring and Quotas

# DCGM (NVIDIA Data Center GPU Manager) exports Prometheus metrics
# After deploying nvidia-dcgm-exporter DaemonSet, scrape into Prometheus/Grafana
curl -s localhost:9400/metrics | grep -E "DCGM_FI_DEV_GPU_UTIL|DCGM_FI_DEV_MEM_COPY_UTIL|DCGM_FI_DEV_GPU_TEMP" | head -5

Key metrics (alert suggestions):

Metric	Meaning	Suggested alert
DCGM_FI_DEV_GPU_UTIL	GPU utilization	persistent <20% = idle waste
DCGM_FI_DEV_MEM_COPY_UTIL	memory bandwidth utilization	>90% = bottleneck
DCGM_FI_DEV_GPU_TEMP	temperature	>85°C
DCGM_FI_DEV_ECC_...	ECC errors	>0 immediately (hardware issue)

Quota management: native ResourceQuota + LimitRange cap namespace GPU totals; multi-tenant strong isolation = MIG + namespace quotas.

4. K8s and Slurm: Division of Labor

How it works:

• Slurm manages HPC batch jobs: MPI, scientific computing, fixed-resource long training—mature queuing, preemption, topology-aware scheduling.
• K8s manages cloud-native services: online inference APIs, microservices, autoscaling, rolling updates—K8s' strengths.
• Common architectures:
  1. Separate GPU pools: Slurm nodes and K8s nodes physically separated, each manages its own (simple and reliable)
  2. Slurm-on-K8s: use K8s to manage bare-metal/VM nodes while Slurm handles job scheduling (advanced)
  3. Event-driven integration: K8s scales GPU Pods at inference peaks; training jobs queue on Slurm—wired via Autoscaler/Webhook

Engineering takeaways:

1. Online inference on K8s (elasticity + rolling updates), batch training on Slurm (queuing + topology)—use each's strengths.
2. Physically separate GPU pools is the most robust: avoids K8s autoscaling snatching Slurm nodes and causing job-queue avalanches.
3. Choose sharing by scenario: dev/test time-slice, production inference MPS for throughput, multi-tenant MIG for strong isolation.
4. Wire DCGM into monitoring: utilization/temperature/ECC fully managed—don't wait for user tickets.
5. Device Plugin is a precondition: new-node checklist must include "kubectl describe node | grep nvidia.com/gpu".
6. Quota first: set ResourceQuota before multi-team sharing to avoid "GPU wars".

5. Engineering Takeaways

1. Device Plugin = the foundation: no plugin = GPUs invisible to K8s.
2. Integer-resource limit: nvidia.com/gpu: 1 is whole-card only; partitioning via time-slice/MIG/MPS.
3. Time-slice for dev, MIG for tenants, MPS for throughput.
4. Time-slice does not isolate memory: avoid in production; OOM risk.
5. DCGM fully managed: utilization/temperature/ECC into Prometheus.
6. K8s online, Slurm batch: separate GPU pools, coordinate without snatching.
7. Quota first: ResourceQuota + MIG for multi-tenant governance.

6. Further Reading

• NVIDIA Device Plugin docs
• DCGM Exporter
• KB: 06. Cloud & Scheduling · Slurm Practice · CUDA MPS · MIG Partitioning