Inference Capacity Planning: From QPS to GPU Instance Count

Note: This is a review article (no first-hand experimental data), based on public practice and capacity engineering.

Experiment Objective

"100 QPS online — how many cards do I need?" is the most asked question before serving goes live, yet the answer is usually a guess. Goal: a complete inference capacity planning method — business QPS modeling (peak/fluctuation) → per-instance throughput baseline (continuing ch12's benchmark article) → latency budget and concurrency constraints → instance-count and redundancy formula → load-test acceptance. After reading you can answer "how many cards, how much headroom, when to scale" with a computable basis.

Code Walkthrough

Part 1: QPS modeling and instance estimation

def capacity_plan(target_qps, peak_ratio, instance_throughput,
                  p99_budget_s, kv_overhead=0.1):
    """Estimate required GPU instances (with fluctuation and redundancy)"""
    peak_qps = target_qps * peak_ratio            # peak modeling (e.g. 3× mean)
    # usable per-instance throughput: constrained by latency budget
    usable = instance_throughput * (1 - kv_overhead)
    instances = peak_qps / usable
    # redundancy: N+1 baseline, 2N for critical services
    return {
        'peak_qps': peak_qps,
        'instances_min': int(instances) + 1,      # N+1
        'instances_critical': int(instances) * 2, # 2N
    }

# e.g. 100 QPS mean, 3× peak, 40 QPS per card (within P99 budget), KV overhead 10%
print(capacity_plan(100, 3.0, 40))
# → peak 300 QPS / ~36 QPS usable per card / ~9 cards (N+1) / 18 cards (2N)

Key points:

1. Peak modeling decides everything: sizing by the mean will blow up in production — the online inference peak/mean ratio is often 2-5× (time-of-day effects, campaigns). Fix peak_ratio by business characteristics first, then size.
2. Per-instance throughput must be "within the latency budget": frameworks like vLLM raise throughput with concurrency, but P99 rises too; the throughput used in planning is "the max stable throughput within the P99 budget," not max throughput — the "budgeted throughput" definition from ch12's benchmark article applied.
3. KV overhead is the safety factor: with long-context requests or low prefix-reuse, KV Cache eats VRAM (ch12's KV article), lowering effective per-instance concurrency — kv_overhead reserves for it.

Part 2: Scale-out watermarks and load-test acceptance

# Scale-out trigger (monitoring expression, linking ch06 elasticity)
# GPU utilization (DCGM SM) sustained > 70% AND P99 > 80% of budget → scale out
# utilization < 40% sustained 30 min → scale in

# Load-test acceptance (ch12 benchmark method)
python -m vllm.entrypoints.benchmark.benchmark_serving \
  --model your-model --num-prompts 10000 \
  --request-rate 40 --max-concurrency 512 \
  --percentile-metrics ttft,itl,tpot

Key points:

• Dual-signal scale watermark: utilization (GPU side) and P99 (user side) — only watching utilization misses "queue buildup where latency already blew up but utilization looks fine"; only watching P99 reacts late.
• Load tests need concurrency caps and request distributions: max-concurrency simulates real concurrency, --request-rate simulates arrival rhythm (ch12's arrival modeling); the conclusion is "at XX QPS this config gives P99 = what," not "max throughput."
• Acceptance definition: validate planning with the P99 curve — load-test at the planned QPS; P99 must stay under budget; if not, add instances or tune, don't wait for production to blow up.

Principles

Inference capacity vs training compute: different planning logic

Dimension	Training compute (ch10)	Inference capacity (this article)
Objective	work per unit time (card-hours)	requests per unit time (QPS)
Constraints	compute+VRAM+time window	latency (P99)+concurrency+peak
Elasticity	job-level (queueing)	instance-level (seconds-minutes)
Headroom logic	by peak batch	N+1/2N redundancy + fluctuation margin

Core difference: training can queue (slower is OK); inference cannot (exceeded latency is an incident) — so inference capacity must be "peak × redundancy," while training compute can be "mean × utilization correction."

Three determinants of per-instance throughput

1. Hardware: HBM bandwidth (ch01: token throughput ≈ bandwidth/weights), compute.
2. Framework and quantization: vLLM continuous batching, KV quantization, weight precision (ch12's KV ledger).
3. Request characteristics: input/output length ratio, prefix-reuse rate, batch shapes — same hardware, different loads: per-card throughput can vary 5-10×, so planning must use your own load's baseline, not someone else's numbers.

Watermark design

Low watermark (scale-in): SM < 40% for 30 min → scale in 1 instance
High watermark (scale-out): SM > 70% AND P99 > 80% budget → scale out 1
Safety buffer: reserve ~20% instance headroom for bursts (campaigns/failover)

Key insight: scaling is "preventive" — GPU cold start takes 5-15 minutes (ch06); by the time P99 exceeds you cannot catch up. Watermarks must lead latency degradation, driven by "utilization trend," not "latency now."

Results

(Estimated, not measured; example service)

Input	Estimate	Conclusion
100 QPS mean, 3× peak, 40 QPS/card	9 cards (N+1) / 18 (2N)	critical services at 2N
30% long-context share	effective throughput -20%-30%	more instances, or KV quantization
high prefix reuse (RAG)	throughput ×2-3	prefer prefix caching

Engineering Takeaways

1. Peak modeling + latency budget + redundancy, all three: sizing by mean, no latency budget, or no redundancy — any one missing blows up in production.
2. Use your own load's baseline, not other people's numbers: request characteristics move per-card throughput 5-10×; the planning baseline must come from your own load test (ch12's method).
3. Watermarks watch trends; scale ahead: utilization trend + P99 dual signal; cold-start time decides the lead.
4. Link with cost: 2N redundancy is a "double the cost" decision — spot instances for redundancy (ch06) cut cost without sacrificing critical-path stability.

Next article preview: inference capacity planning answers "how many cards"; next, back to the AI×Science home turf — "Scientific Foundation Models: Engineering Panorama from Data to Training."