KV Cache Compression Ledger: GQA, Quantization & Paging

In long-context inference the memory heavyweight isn't weights — it's the KV cache, growing linearly with sequence length and deciding how many requests you can serve concurrently. Four levers compress it: architecture (GQA/MQA), quantization (INT8/INT4), memory (paging), sharing (prefix cache). This article settles all four, linking to Long-Context Inference.

1. The KV Baseline

KV per token (recap of Inference Resource Accounting):

$$KV\_per\_token = 2 \times L \times K \times H \times bytes$$

Example: Llama-3-8B (L=32, K=8, H=128, FP16) → 128 KB/token; 32K context → 4 GB/request. One request eats 4GB; an 80G card serves ~20 concurrent long-context requests — that's the motivation for compression.

2. Architecture: MHA → GQA → MQA

Architecture	KV heads	KV/token (vs MHA)	Quality loss
MHA	32 (=Q heads)	1×	baseline
GQA	8 (group-shared)	1/4	small
MQA	1 (fully shared)	1/32	noticeable

Takeaway: GQA is today's mainstream compromise (tiny quality loss, 4× KV saving); MQA saves more but only for quality-insensitive scenarios. Count GQA into the KV ledger at model selection — cheaper than compressing afterward.

3. Quantization: KV INT8 / INT4

Store KV at lower precision, dequantize to FP16 for compute:

Scheme	KV memory	Quality impact	Note
FP16 baseline	2B/token	—
INT8	1B/token (−50%)	minimal	mainstream default
INT4	0.5B/token (−75%)	slight at long context	needs calibration

Takeaway: INT8 KV is a free win — nearly lossless, half memory; use INT4 when memory is tight and some precision loss is acceptable (stacks well with weight quantization).

4. Memory Layer: PagedAttention

KV isn't allocated as one big contiguous block; it's paged like an OS (vLLM's PagedAttention):

• Allocate per block (e.g. 16 token/page) — less fragmentation, higher VRAM utilization;
• Share prefix pages across requests (identical prompt parts stored once);
• Effect: 2-4× concurrent requests on the same VRAM — vLLM's core selling point (see vLLM Principles).

5. The Combo: Four Levers Together

Combo	KV/token	32K context/request	80G-card concurrency (~)
MHA+FP16	128 KB	4 GB	~18
GQA+FP16	32 KB	1 GB	~72
GQA+INT8	16 KB	0.5 GB	~144
GQA+INT8+paging+prefix	~16 KB (deduped lower)	<0.5 GB	200+

Takeaway: architecture (GQA) → quantization (INT8) → memory (paging), each step is "free" — the complete path from "a dozen concurrent requests" to "hundreds" for long-context serving.

6. Reproducible Python

def kv_compression(L, K, H, seq, bytes_per=2, head_share=1, quant=1, paging=1.0):
    """KV memory (GB): head_share=GQA sharing factor, quant=quantization factor, paging=dedup factor"""
    per_token = 2 * L * K * H * bytes_per / head_share / quant * 1e-6   # KB
    return per_token * seq * paging / 1e6                                # GB

kv_compression(32, 8, 128, 32768, head_share=4, quant=2, paging=0.8)
# ≈ 0.21 GB/request vs MHA+FP16 baseline 4 GB → saves 95%

7. Checklist

1. Check the KV architecture at model selection: prefer GQA; compute the KV ledger with the actual K;
2. Enable KV INT8 by default at launch; INT4 only when memory is tight (run a quality regression);
3. Choose engines with paging + prefix sharing (vLLM etc.) — concurrency gains are immediate;
4. Measure KV memory and concurrency ceilings with Acceptance & Baselines;
5. Validate each lever separately for quality and gain; don't flip everything at once (hard to attribute).

---

Organized and written by this site for learning purposes; formulas and examples are the site's own; specific versions and measurements are subject to on-site conditions.