Inference Resource Accounting: Memory, Compute, and Bandwidth per Generation

1. Scenario: Don't Just Count Weights

Before putting a model online, three questions always come up: how many GPUs, how fast, what QPS? All three need the same thing — splitting one generation into three ledgers: memory, compute, and bandwidth. The most common mistake is counting only weights: a 70B model is 140 GB in FP16, and you might think two 80 GB GPUs are enough — until the KV cache grows and the workspace overflows, and you OOM.

This article follows the knowledge base's "derive design from constraints" method: first list what is moved, how much, how many times, through where, and who must wait for it, then do order-of-magnitude estimates against hardware capacity, bandwidth, and compute. All formulas are reproducible; a Python accounting function is at the end.

2. Ledger One: Memory

An inference instance's memory = weights + KV cache + activation/workspace. Missing any one of them is a bug.

Weights (loaded and resident):

$$W = N \times bytes\_per\_param$$

Precision	bytes/param	70B model
FP32	4	280 GB
FP16 / BF16	2	140 GB
INT8	1	70 GB
INT4 (GPTQ/AWQ)	0.5~0.6	35~42 GB

KV cache per token:

$$KV\_per\_token = 2 \times L \times K \times H \times bytes$$

L=layers, K=KV heads, H=head dim. Under GQA, K is far smaller than Q heads — the main switch for KV size. Example (BF16, bytes=2):

Model	L	K	H	KV/token	4K ctx	32K ctx
7B (GQA)	28	4	128	28 KB	115 MB	917 MB
70B (GQA)	80	8	128	160 KB	655 MB	5.2 GB

Activation/workspace: prefill's large activation tensors plus engine workspace (vLLM reserves 90% of GPU memory for KV by default via gpu_memory_utilization). Estimate as the remainder after weights + KV.

Total: memory ≈ weights + KV(planned for peak concurrent context) + headroom(≥10%). This is why the vLLM deployment guide says "70B starts with 4 GPUs" and "KV quantization saves 20-50%".

3. Ledger Two: Compute

Generation has two phases with very different compute:

• Prefill: processes the whole input in parallel. ~ $2 \times N$ FLOPs per token.
• Decode: generates one token at a time, ~ $2 \times N$ FLOPs per step, reusing prefill's KV — small compute per step, but bandwidth-hungry.

One request (S input tokens, T output tokens):

$$FLOPs \approx 2N \cdot S + 2N \cdot T$$

Example (70B, S=1024, T=512): $2 \times 70e9 \times 1536 \approx 2.15 \times 10^{14}$ FLOPs. Against a single GPU FP16 peak (e.g., H800 ≈ 989 TFLOPs BF16, assume ~50% efficiency), prefill throughput reaches tens-hundreds of tokens/ms; decode's bottleneck is usually bandwidth, not compute.

4. Ledger Three: Bandwidth

Decode is bandwidth-starved: every generated token requires reading all weights from HBM once. Per-GPU bandwidth ledger:

$$B\_needed = N \times bytes\_per\_param \quad (bytes/token)$$

Model	FP16 weights read/token	Theoretical TPOT @3TB/s HBM
7B	14 GB	~4.7 ms/token
70B	140 GB	~47 ms/token

This "full weight read per token" is the floor for TPOT — and explains why single-GPU decode cannot beat physical bandwidth, why MoE (activating only some experts) helps, and why small batches waste compute while large batches amortize bandwidth (next article).

Prefill is the opposite: input tokens are computed in parallel, weights are read fewer times, compute-bound. The two phases pull in opposite directions, which is why inference systems optimize them separately — the physical motivation behind vLLM's continuous batching.

5. Accounting Tool: Python (runs anywhere)

def kv_bytes(num_layers, kv_heads, head_dim, seq_len, bytes_per=2):
    return 2 * num_layers * kv_heads * head_dim * seq_len * bytes_per

def weights_bytes(num_params, bytes_per=2):
    return num_params * bytes_per

def flops_request(num_params, input_tokens, output_tokens):
    return 2 * num_params * (input_tokens + output_tokens)

def decode_tpot(num_params, bytes_per, hbm_bw):
    """bandwidth-bound per-token time floor (seconds)"""
    return num_params * bytes_per / hbm_bw

# e.g. 70B, GQA(80 layers/8 KV heads/128 dim), FP16, 4K ctx, 3TB/s HBM
N = 70e9
print(weights_bytes(N) / 1e9, "GB weights")          # 140 GB
print(kv_bytes(80, 8, 128, 4096) / 1e6, "MB KV")      # 655 MB
print(decode_tpot(N, 2, 3e12) * 1e3, "ms/token")      # ~46.7 ms

6. Practical Checklist

• [ ] All three memory items (weights + KV + workspace) are accounted; never weights only
• [ ] KV planned for peak concurrent context (not average), GQA config confirmed
• [ ] Decode bottleneck verified with bandwidth (TPOT ≥ weight bytes / HBM bandwidth), not peak FLOPs
• [ ] Prefill and decode estimated separately: prefill is compute-bound, decode is bandwidth-bound
• [ ] Quantization gains recomputed: INT4 cuts the weight ledger, KV quantization cuts the cache ledger — don't mix them

Next, From Model Architecture to Resource Requirements maps attention, KV reuse, and MoE sparsity onto these formulas.

Method note: the order-of-magnitude and "five data-movement questions" framework follows the open-source book Understanding AI Infra: Quantitative Analysis and System Design (Li Bojie, Apache-2.0, bojieli.github.io/ai-infra-book); formulas and examples here are the site's own derivations.