MoE Inference Ledger: Routing, Load Balance & Expert Parallelism

Mixture-of-Experts (MoE) uses sparse activation for "big params + low compute" — but the inference ledger is far more than compute: all experts stay in VRAM, routing can imbalance, and fetching experts across cards costs All-to-All. This article settles MoE inference's three ledgers, linking to DeepSeek MoE/MLA.

1. Memory Ledger: Sparse Activation, Dense Residency

MoE activates only a few experts per token (e.g. top-2), but all expert weights are in VRAM:

$$Mem_{MoE} = \underbrace{W_{dense}}_{\text{attention/FFN shared}} + \underbrace{E \times W_{expert}}_{\text{all E experts}}, \quad \text{active compute only } k/E$$

Model	Total params	Active params/token	VRAM
Dense 70B	70B	70B	140 GB+
MoE (8 experts, top-2)	~50B	~5B	~100 GB (all experts)

Takeaway: MoE saves compute (fewer activations), not VRAM (all resident) — use total params for the memory ledger and active params for the compute ledger; don't mix the two.

2. Routing & Load Balance: The Cost of Imbalance

The router assigns tokens to experts; imbalance means:

• Hot experts: individual experts queue, becoming the latency bottleneck;
• Cold experts: idle, wasting resident VRAM;
• Training's auxiliary load-balancing loss mitigates this, but inference load drifts over time.

Engineering countermeasures: measure routing distribution with real request traffic before launch; if a hot expert exceeds ~2× the mean, adjust routing policy or retrain the router.

3. Communication Ledger: Expert Parallelism & All-to-All

Experts spread across cards (expert parallelism, EP); routed tokens travel to target cards via All-to-All:

$$Comm \propto \text{cross-card token ratio} \times \text{activation size}, \quad T_{alltoall} \approx \frac{(N-1)}{N} \times \frac{Msg}{BW}$$

Key trade-off:

• Larger EP: fewer experts per card, less VRAM pressure, but more cross-card tokens and heavier comm;
• Smaller EP: lighter comm, but each card hosts more experts;
• Conclusion: EP size = VRAM-vs-comm compromise — run the All-to-All formula from Communication Ledger first.

4. Optimizations: Prefetch, Offload & Quantization

Technique	Principle	Gain
Expert prefetch	preload likely-next-layer experts	hide expert-fetch latency
Expert offload	cold experts to CPU/remote, fetch on use	big VRAM drop, slight latency
Expert quantization	store expert weights at low precision	half VRAM, slight quality drop
Routing cache	reuse routing results for similar prompts	save routing compute & comm

Takeaway: when VRAM is tight, "offload cold experts + keep hot ones resident" is the best ROI; prefetch pays off at low concurrency.

5. Reproducible Python: EP Communication & Memory

def moe_ep_ledger(n_experts, expert_gb, world, cross_ratio, msg_mb, bw_gbps):
    """Expert parallelism: per-card VRAM (GB) and All-to-All time (ms)"""
    mem = n_experts * expert_gb / world
    t_ms = (world - 1) / world * msg_mb * cross_ratio / (bw_gbps / 8) / 1e3
    return round(mem, 1), round(t_ms, 2)

moe_ep_ledger(n_experts=8, expert_gb=12, world=8, cross_ratio=0.4, msg_mb=8, bw_gbps=400)
# (12 GB/card, 0.56 ms/token) → EP8: VRAM acceptable, comm controlled

6. Checklist

1. MoE memory ledger by total params, compute ledger by active params — keep the two apart;
2. Measure routing distribution before launch; handle hot experts first (>2× mean);
3. Size EP with the All-to-All formula (comm ratio >30% → shrink EP or fix topology);
4. When VRAM is tight: offload cold experts + keep hot resident, then add expert quantization;
5. Run Acceptance & Baselines after each change; record latency and VRAM.

---

Organized and written by this site for learning purposes; formulas and examples are the site's own; specific versions and measurements are subject to on-site conditions.