Kubernetes Inference Service in Production: vLLM from Deployment to High Availability

Experiment Goal

Running vLLM inference on a GPU cluster with Kubernetes — "kubectl apply a Deployment" is only the starting point. Productionization answers orchestration-side questions:

• How to design the Deployment so GPU resources are allocated and isolated correctly?
• How to upgrade/release the service without interruption or overcommitting VRAM?
• How to scale elastically under load spikes instead of wedging GPUs?
• When something breaks, how does monitoring pinpoint "model service vs platform layer" first?

This article walks through productionizing vLLM inference from the Kubernetes orchestration view: Deployment design, GPU quotas, rolling updates & graceful shutdown, HPA, and monitoring. Note: this is a review + architecture-analysis route; code snippets are generic practice examples without first-hand experiment data; versions and parameters follow official docs.

Deployment Design: Pin GPU Resources Correctly

1. GPU Resource Declaration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-serve
spec:
  replicas: 2
  selector:
    matchLabels: { app: vllm-serve }
  template:
    metadata:
      labels: { app: vllm-serve }
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          command: ["python", "-m", "vllm.entrypoints.openai.api_server"]
          args:
            - --model
            - /models/Qwen2.5-72B-Instruct
            - --max-model-len
            - "32768"
            - --gpu-memory-utilization
            - "0.92"
            - --port
            - "8000"
          ports:
            - containerPort: 8000
          resources:
            requests:
              nvidia.com/gpu: "1"   # must match Device Plugin resource name
              cpu: "8"
              memory: "64Gi"
            limits:
              nvidia.com/gpu: "1"
              cpu: "8"
              memory: "64Gi"

Key points:

• GPU resource name (nvidia.com/gpu) is registered by the Device Plugin; different plugins (NVIDIA/Ascend/custom) expose different names — distinguish in mixed clusters;
• requests must equal limits (GPUs are non-compressible; declaring only requests leaves an oversubscription hole);
• --gpu-memory-utilization 0.92 reserves VRAM headroom for KV cache and model loading — never set 1.0.

2. Scheduling Constraints & Affinity

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: gpu-type
                    operator: In
                    values: ["h20", "a100"]
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule

• nodeSelector/affinity by GPU model: with mixed models, use node labels to pin services to the right machine, avoiding "scheduled to a node without the card";
• topologySpreadConstraints spreads replicas: multi-replica services spread across nodes so a single node failure doesn't take down all replicas;
• Multi-GPU services (tensor parallel) need cards on the same node — use node-level required constraints or a custom scheduler.

3. Model Loading & Persistence

• Mount model weights via PVC or read-only volume to avoid image bloat and repeated downloads;
• For shared weights across replicas, use a ReadOnlyMany volume (object-storage/NFS CSI) or embed in the image;
• Cold-start loading a 70B model is slow — use StartupProbe to relax readiness instead of letting ReadinessProbe flap.

Rolling Updates & Graceful Shutdown: Release Without Interruption

Rolling out a new model/parameters with a blunt kubectl rollout restart creates a window where "the new Pod is still loading, the old Pod is already terminated."

Graceful Shutdown Config

spec:
  template:
    spec:
      terminationGracePeriodSeconds: 300   # room for model unload & request drain
      containers:
        - name: vllm
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 10"]  # wait for traffic removal

• Give terminationGracePeriodSeconds room: vLLM must drain in-flight requests and unload VRAM; the 30s default is not enough;
• preStop sleep: let the Service endpoint be removed first (EndpointSlice updates have latency), then trigger process exit;
• With ReadinessProbe: new Pod takes traffic only when the model is ready; the old Pod terminates only after health checks remove it.

Rolling Strategy

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0   # keep service capacity

maxUnavailable: 0 guarantees available replicas during release (cost: peak extra GPU in small clusters — trade off); maxSurge caps parallel new replicas to avoid spiking cluster VRAM.

Elastic Scaling: The Right Way to Use HPA

GPU services should not use CPU metrics for HPA (low CPU ≠ low load). Recommended:

1. Based on concurrent requests/queue length: vLLM exposes Prometheus metrics (e.g. vllm:num_requests_running, vllm:num_requests_waiting); scale via KEDA or HPA custom metrics on in-flight requests;
2. Lower bound protection: minReplicas at least the minimum replicas the model can serve; maxReplicas bounded by cluster GPU total to avoid saturating the fleet;
3. Scale up fast, scale down slow: model loading is slow; scale-down cooldown must exceed model load time to prevent flapping.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-hpa
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: vllm-serve }
  minReplicas: 2
  maxReplicas: 8
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 600   # scale-down cooldown
  metrics:
    - type: Object
      object:
        metric: { name: vllm_num_requests_waiting }
        describedObject: { apiVersion: v1, kind: Service, name: vllm-serve }
        target: { type: AverageValue, averageValue: "10" }

Monitoring & Alerting: Layer-Based Diagnosis

Inference failures should quickly distinguish "model layer vs platform layer":

Layer	Example metrics	Example alerts
Model service	vllm:num_requests_running/waiting, P99 latency, tokens/s throughput	P99 spike, waiting queue saturated, OOM
GPU layer	DCGM metrics (util, VRAM, temp, NVLink BW, ECC errors)	VRAM pinned at 100%, temp over threshold, NVLink degraded, ECC growth
Platform layer	Pod status, restarts, endpoint count, node allocatable GPUs	CrashLoop, GPU unschedulable

Engineering observation:

• GPU utilization ≠ service health: 0% util can be idle or wedged (requests queued but no compute). Alert on both utilization and request queue.
• VRAM is the hard constraint: inference OOM usually manifests as CUDA out of memory crash-restart. Link --gpu-memory-utilization and HPA cap to avoid replica growth blowing VRAM.
• Node allocatable GPU monitoring: GPU fragmentation (each node left with 1 card that can't form multi-GPU services) is a hidden inference-cluster bottleneck; alert on allocatable counts plus scheduling failures.

Production Checklist

• [ ] GPU requests=limits; resource name matches Device Plugin
• [ ] Multi-replica affinity by model + node spread; multi-GPU services co-located
• [ ] Model weights via read-only volume; StartupProbe covers cold start
• [ ] terminationGracePeriodSeconds ≥ 300; preStop drains traffic
• [ ] Rolling maxUnavailable=0; release drill verified
• [ ] HPA on request metrics + scale-down cooldown; cap bounded by GPU total
• [ ] Monitoring covers model/GPU/platform layers; alerts identify failure source

Engineering Takeaways

• Inference productionization = a set of orchestration details: Deployment declaration, rolling strategy, graceful shutdown, elasticity, layered monitoring — any missing link surfaces as an incident after go-live.
• Write the immutable facts first: GPU resources, model affinity, VRAM caps hard-coded in the deployment manifest — business iteration can't silently break platform constraints.
• Release is a drill: every model upgrade runs the same rolling flow until "release without interruption" becomes muscle memory.

Next article preview: with inference services stable on K8s, the cluster keeps growing — multi-vendor accelerators (NVIDIA/Ascend/Cambricon) start coexisting. The next article, Unified Resource Management for Heterogeneous Accelerators, dissects mixed-vendor scheduling from the Slurm view.