AI Infra Quantitative Analysis

About this chapter: before building a cluster, choosing a model, or tuning inference parameters, settle the accounts with order-of-magnitude estimates — what is moved, how much, how many times, through where, and who must wait for it. The method framework follows the open-source book Understanding AI Infra: Quantitative Analysis and System Design (Li Bojie, Apache-2.0); all formulas and examples here are the site's own and reproducible. Measurements link to Chapters 02 (cluster), 03 (network), 11 (training) and 12 (inference).

Quantitative Methods

Inference Resource Accounting

Memory (weights+KV+workspace), compute (prefill/decode), bandwidth (full weight read per token), with reproducible Python functions.

From Model Architecture to Resource Requirements

KV ledger across MHA/GQA/MQA, MoE sparsity's bandwidth ledger, and a quantitative model-selection checklist.

Inference Optimization Boundaries

When continuous batching, KV quantization, prefix caching, and speculative decoding actually pay off — and when they don't.

Training Resource Accounting

Four state copies, All-Reduce scaling with parallelism, and recomputation-for-memory decision rules.