From Model Architecture to Resource Requirements: Attention, KV, and MoE

1. Scenario: Architecture Decides the Ledger

Models with the same parameter count can have very different memory, bandwidth, and compute needs depending on architecture. Building on the three ledgers from Part 1 (memory/compute/bandwidth), this article quantifies the impact of attention, KV reuse, GQA, and MoE sparsity — so you can compute "how much resource this architecture eats" before choosing it.

2. Attention: Compute Grows with Sequence Length

Standard multi-head attention (MHA) attends each token to all history. During prefill, attention compute grows quadratically with sequence length:

$$FLOPs_{attn} \propto S^2 \times H$$

• S=sequence length, H=attention heads. Going from 4K to 32K input (8×) raises attention compute ~64×.
• During decode, each new token attends only to existing KV ($S$ steps), linear compute — which is why long contexts make prefill heavy and TTFT long.
• Engineering consequence: FlashAttention reorders KV in blocks and avoids moving the whole KV into on-chip storage — it cuts movement, not the mathematical complexity (see llm-lab-07-flash-attention in Chapter 11).

3. KV Cache: Linear in Context, GQA Is the Switch

From Part 1: KV per token = 2 × L × K × H × bytes. Here is how architecture changes K:

Architecture	KV heads (K)	Relation to Q heads	32K ctx KV (70B-class, BF16)
MHA	= Q heads (e.g., 64)	one KV per query head	~42 GB
GQA	8	8 query heads share 1 KV	~5.2 GB
MQA	1	all query heads share 1 KV	~0.65 GB

Same context, MHA→GQA cuts KV ~8×. This is the system-level reason mainstream open models (Qwen, Llama 3 family) moved to GQA: it saves memory and bandwidth, not model quality. Quantitatively: KV does not scale with batch (shared), it grows with total concurrent context — plan deployment as "concurrency × average context × KV/token".

4. MoE: Sparsity Rewrites the Bandwidth Ledger

The key of mixture-of-experts: total weights are large, but each token activates only some experts.

$$B\_needed = N_{active} \times bytes \quad (bytes/token)$$

Model	Total params	Active params	FP16 read/token	vs 7B bandwidth cost
Dense 7B	7B	7B	14 GB	1×
Dense 70B	70B	70B	140 GB	10×
MoE 70B-A3B	~70B	~3B	~6 GB	0.43×

MoE's accounting logic: buy memory for total parameters, pay bandwidth for active parameters. On the same GPUs, a 70B MoE (3B active) has decode bandwidth cost at 43% of a 7B dense model — the source of its throughput/cost advantage. The price is routing communication (All-to-All) and expert load imbalance, which need the network ledger (Chapter 03) and scheduling (see Part 4's communication item).

5. Architecture Selection Checklist

• [ ] Long-context-heavy → compute the KV ledger first; prefer GQA/MQA + KV quantization; MHA is nearly unusable above 32K
• [ ] Decode-heavy (chat/streaming) → look at the active-parameter bandwidth ledger; MoE wins clearly; prefill-heavy (batched offline) → look at the compute ledger
• [ ] Memory-tight → weight quantization (INT4 cuts the weight ledger) + KV quantization (cuts the cache ledger); compute each saving separately
• [ ] Multi-GPU → confirm GQA/MoE communication patterns: KV sharding vs expert All-to-All, different network ledgers (Chapters 03 & 11)
• [ ] Always recompute on model change: Dense/MoE/GQA combinations at similar parameter counts can differ by an order of magnitude; never select by parameter count alone

Method note: the order-of-magnitude and "five data-movement questions" framework follows the open-source book Understanding AI Infra (Li Bojie, Apache-2.0); examples here are the site's own, model configs per official releases.