Inference Optimization Boundaries: When Batching, KV Management, and Speculative Decoding Actually Pay Off

1. Scenario: Every Optimization Has a Range

vLLM is fast, KV quantization saves memory, speculative decoding accelerates — all with boundaries. This article uses the three ledgers from Part 1 to compute when each works and when it doesn't, so you don't blindly copy parameters. Engine measurements live in Chapter 12 (vllm-principles, vllm-production-deployment, speculative-decoding).

2. Continuous Batching: Amortizing the Bandwidth Ledger

Decode reads all weights per token; a single request is bandwidth-bound. Batching lets requests share one weight read:

$$B\_per\_token \approx \frac{N \times bytes}{batch\_size}$$

batch	70B FP16 weights read/token	Theoretical TPOT floor (3TB/s)
1	140 GB	~47 ms
8	17.5 GB	~5.8 ms
64	2.2 GB	~0.7 ms

Boundary: bigger batches amortize bandwidth better, but memory (KV grows per request) and scheduling overhead rise. Continuous batching's value: it doesn't wait for the whole batch — requests enter/leave dynamically, keeping "effective batch" high. When is the gain largest? Dense arrivals and long outputs. In idle periods (sparse arrivals) there is almost no gain. Quantitative check: if effective batch < 4 and concurrency cannot rise, upgrading bandwidth or switching to MoE beats tuning batch parameters.

3. KV Quantization and Prefix Caching: Saving Memory and First-Token Latency

• KV quantization (INT8/INT4): drops the bytes term in Part 1's KV formula from 2 to 1/0.5 — saves memory and KV read bandwidth, at some precision cost. When it works: long context/high concurrency where KV dominates (70B 32K ctx KV≈5.2GB → ≈1.3-2.6GB); short contexts gain little.
• Prefix caching: requests sharing a common prefix reuse computed KV — saves duplicated prefill compute. When it works: fixed system prompts, RAG with shared document prefixes, multi-turn context — the higher the hit rate, the more valuable; random per-request prefixes gain almost nothing. The site's vllm-production-deployment already measures prefix-caching TTFT gains; this article adds the boundary: gain ceiling = duplicated prefill FLOPs / total FLOPs.

4. Speculative Decoding: Profitable Only When Verification Beats Autoregression

Speculative decoding drafts N tokens with a small model (or single-token predictor) and verifies them in one forward pass of the target model. Whether it pays off depends on two timings:

$$speedup \approx \frac{N \times t_{draft} + t_{target}}{t_{verify} + t_{draft}}$$

Intuition: verifying a batch of tokens costs the same compute as generating one token but with more parallelism — under small batches and bandwidth limits, a verification batch uses bandwidth better than one-by-one autoregression. Therefore:

• Works: large target model (clear decode bandwidth bottleneck), high draft hit rate (code/structured text), small batch (compute to spare, bandwidth short).
• Useless or negative: small target model (bandwidth not saturated), low draft hit rate (verification wasted), large batch (bandwidth already amortized, autoregression near the limit).

The site's speculative-decoding article gives the measurement workflow; this article adds the decision formula: measure target single-token latency $t$ and draft hit rate $p$; if $p$ is low or batch already high, skip it when speedup <10%.

5. Optimization Applicability Quick Reference

Technique	Ledger saved	Effective when	Ineffective when
Continuous batching	bandwidth	dense arrivals, long outputs	sparse arrivals, low effective batch
KV quantization	memory + cache bandwidth	long context / high concurrency	short context
Prefix caching	prefill compute	high prefix hit rate	random prefixes
Speculative decoding	decode latency	large model + small batch + high hit rate	small model / large batch / low hit rate
Weight quantization	memory + weight bandwidth	acceptable precision	precision-sensitive tasks

6. Deployment Checklist

• [ ] Measure current effective batch and bandwidth utilization before adding batch vs switching techniques
• [ ] Compute KV share (KV/total memory) before KV quantization; below ~10% the gain is limited
• [ ] Evaluate prefix caching by hit rate; don't enable blindly
• [ ] Use the formula before speculative decoding; measure draft hit rate and single-token latency first
• [ ] Recheck every optimization against Part 1's three ledgers — avoid "optimizing into a corner"

Method note: the boundary-condition quantification framework follows the open-source book Understanding AI Infra (Li Bojie, Apache-2.0); formulas and boundaries here are the site's own.