LLM Inference SRE: SLOs, Capacity Planning, and Degradation Strategies

Summary: Shipping an inference service is just the beginning; "serving it stably" is another matter. This post covers the LLM-inference SRE trio: how to define SLOs, how to compute capacity, and what to do under overload. Unlike the deployment posts in chapter 12 (vLLM/TRT flags), this one focuses on the service-governance layer: metrics, alerts, capacity models, and degradation.

0. Experiment Goals

• Define inference-service SLOs (time-to-first-token / total latency / availability)
• Build a capacity model: the math relationship among concurrency, throughput, and KV memory
• Design degradation strategies: multi-level fallbacks under overload
• Land an alerting and capacity-planning SOP

1. SLOs: Define "Normal" First

# Example SLOs (online chat scenario)
- TTFT (time to first token) P95 < 2s
- Total generation latency P95 < 30s (scaled by max_tokens)
- Availability: 99.5% (monthly)
- Error rate (4xx/5xx/timeouts) < 1%

How it works:

• The SLO trio: TTFT (first token; the user's perceived "responsiveness"), total latency/generation speed, availability. Don't only watch availability—"not down but slow as if down" is the most common inference-service incident.
• Metrics come from vLLM/TRT native exports (the Prometheus table in KB vLLM Production Deployment), plus gateway-level: request volume, error rate, timeout rate.
• Alert grading: P1 (availability broken / error rate spiking, immediate), P2 (TTFT P95 over threshold for 5min), P3 (capacity watermark near the ceiling).

2. Capacity Planning: Concurrency, Throughput, KV Memory

Do the math (8B model · 80G card · 8K context example):

Per-card concurrency ceiling ≈ KV memory budget ÷ KV per request
- KV budget = 80G × 0.90 - weights (~16G) ≈ 56G
- KV per 8K-context request ≈ 1GB (FP16) → concurrency ≈ 56
- FP8 KV halves it → 0.5GB per request → concurrency ≈ 112

Capacity (QPS) ≈ concurrency ÷ average request time
- avg time 5s, concurrency 56 → QPS ≈ 11
- need 100 QPS → about 9-10 replicas (with redundancy)

Engineering notes:

• Capacity = concurrency × throughput, not "number of GPUs": memory sets the concurrency ceiling, GPU compute sets per-request speed; their product is capacity.
• Planning formula: replicas = targetQPS × avgTime ÷ per-card concurrency × redundancy(1.3).
• Context length is a variable: long-context requests compress the concurrency ceiling (see the KV ledger in Long-Context Inference)—capacity planning must use the business's actual context-length distribution, not max-model-len.

3. Overload Degradation: Multi-Level Fallbacks

Overload signals: queue count > threshold / TTFT P95 over / gateway CPU pressure
   │
   ├─ Level 1: rate limiting (gateway) — per-tenant/IP limits, protect the backend from being hammered
   ├─ Level 2: queue & priority — demote long tasks, prioritize short ones
   ├─ Level 3: quality fallback — lower max_tokens / disable retrieval / simplify prompts
   └─ Level 4: read-cache fallback — hot questions hit cache and return directly (below)

Engineering notes:

• Rate limiting is the first gate: gateway-level limit_req by tenant quota (nginx/gateway config in vLLM Production Deployment); don't let overloaded requests pile into the inference queue.
• Degrade quality before failing: cut max_tokens from 2048 to 1024, retrieval from TOP6 to TOP3—the user perceives "a shorter answer", not "service unavailable".
• Multi-level caching: KV prefix caching (hit saves prefill) + semantic caching (hot questions return historical answers) + static page caching—three levels cut real inference pressure by an order of magnitude.

4. Capacity Planning & Scaling SOP

# Scale-out decision (example): monitoring-panel trigger conditions
# When queue count > 50 for 5min, or TTFT P95 > 2s for 10min
# Action: add one replica to the gateway upstream (same flags, same version) → add after /health passes
# Scale-in: utilization persistently < 30% with no queue → remove the replica (drain first, then stop)

# Alert loop: Prometheus + Alertmanager → on-call group → handling records archived

Engineering takeaways:

1. Define SLOs before monitoring: TTFT/total latency/availability, graded alerts.
2. Capacity = concurrency × throughput: memory sets the concurrency ceiling, compute sets per-request speed.
3. Plan capacity by business context distribution, not max-model-len.
4. Rate limiting is the first gate: gateway per-tenant quotas; don't let requests pile into the inference queue.
5. Degrade quality before failing: lowering max_tokens/retrieval depth is far more graceful than "unavailable".
6. Three-level caching: KV prefix + semantic + static, cutting real inference pressure an order of magnitude.
7. Automated scaling: metric trigger → add replica → health check → live, all scripted.

5. Further Reading

• Google SRE book (SLO chapter)
• vLLM Production Deployment
• KB: 12. Inference Engines · Long-Context Inference · Private LLM Serving