AI Infra Quantitative Analysis
About this chapter: before building a cluster, choosing a model, or tuning inference parameters, settle the accounts with order-of-magnitude estimates — what is moved, how much, how many times, through where, and who must wait for it. The method framework follows the open-source book Understanding AI Infra: Quantitative Analysis and System Design (Li Bojie, Apache-2.0); all formulas and examples here are the site's own and reproducible. Measurements link to Chapters 02 (cluster), 03 (network), 11 (training) and 12 (inference).
🧭 Reading Map
- Prerequisites: none (book starting point)
- Role: Part 0, L0 quantitative methods — do the memory/compute/bandwidth math before building clusters, picking models or tuning inference; revisit this chapter to re-derive any resource decision in later chapters
- Order: cards 1→12 (… → supernode → comm topology → edge-cloud) (reasoning → architecture → optimization boundary → training → accelerator → communication → operator → distributed inference → utilization → supernode → comm topology): reasoning → architecture → optimization boundary → training → accelerator → communication → operator → distributed inference → utilization
- Next: 02. Hardware & Chips — after the math, meet the first brick of the compute base
Quantitative Methods
Inference Resource Accounting
Memory (weights+KV+workspace), compute (prefill/decode), bandwidth (full weight read per token), with reproducible Python functions.
Read →
From Model Architecture to Resource Requirements
KV ledger across MHA/GQA/MQA, MoE sparsity's bandwidth ledger, and a quantitative model-selection checklist.
Read →
Inference Optimization Boundaries
When continuous batching, KV quantization, prefix caching, and speculative decoding actually pay off — and when they don't.
Read →
Training Resource Accounting
Four state copies, All-Reduce scaling with parallelism, and recomputation-for-memory decision rules.
Read →
Accelerator Arithmetic: From Spec Sheet to Deliverable FLOPs
Peak-FLOPs derivation, arithmetic intensity & roofline, why decode is bandwidth-bound, L40S/H-series comparison.
Read →
Communication Ledger: All-Reduce Bandwidth Scaling
Ring all-reduce bandwidth model, busbw vs scale, why gradient all-reduce needs IB, L40S without NVLink ceiling.
Read →
Operator & Runtime Ledger: Roofline, Fusion, FlashAttention I/O
Roofline bottleneck rules, memory movement saved by fusion, FlashAttention's O(N²)→O(N) I/O complexity.
Read →
Distributed Inference Ledger: Where Does State Go
Tensor-parallel memory/comm ledgers, KV placement, prefill/decode separation, scale-out & recovery.
Read →
GPU Utilization Ledger: Why It's Low, Sharing Saves Cards
Sparsity, fragmentation, isolation overhead; how continuous batching lifts utilization and saves cards.
Read →
Supernode Ledger: Capacity, Throughput & Sync Cost
A supernode wraps GPUs into one big GPU: aggregated capacity, in-node vs inter-node bandwidth, sync-ratio<30% sizing.
Read →
Collective Topology Quantified: ring, tree & rail
Bandwidth/latency models of three topologies, when SHARP beats ring, same-rail one-hop vs cross-rail via spine.
Read →
Edge-Cloud Ledger: Local, Edge, or Cloud?
Latency/quality/cost/bandwidth joint decision: local small model vs cloud large model + RTT crossover, buy-vs-rent breakeven.
Read →