Training Resource Accounting: The Memory, Communication, and Recomputation Trade-off

1. Scenario: Why Training Eats More Than Inference

Inference runs a fixed forward pass; training must keep more state per step and synchronize gradients across GPUs. This article extends the three-ledger framework to training — memory, communication, recomputation — and links it to the Chapter 11 llm-lab series (DDP/TP/Allreduce) and Chapter 02 cluster measurements (80-node HPL communication observations).

2. Memory Ledger: Four Copies Stacked

During training, each parameter has four copies:

State	bytes/param (FP16/FP32 mixed)	7B model
Weights (FP16)	2	14 GB
Gradients (FP16)	2	14 GB
Optimizer states (Adam m+v, FP32)	8	56 GB
Activations/intermediates	grows with batch×seq	GBs to tens of GB

Total: 7B FP16 training needs ~84 GB of fixed state alone — vs 14 GB for inference. This is why "training GPU count starts at 2-4× inference." Optimizer states are usually the biggest item (Adam's 8 bytes/param); mixed precision / ZeRO sharding targets it first.

3. Communication Ledger: All-Reduce and Parallelism

Data-parallel (DDP) training does one full-gradient All-Reduce per step:

$$C \approx 2 \times (P-1)/P \times N \times bytes\_grad$$

• P=parallel GPUs, N=params. 7B FP16 gradients are 14 GB, exchanged in full across 8 GPUs each step.
• Key observation (same as the Chapter 02 HPL measurements): communication grows linearly with GPU count; scaling efficiency = compute growth / communication cost growth. With slow interconnects (e.g., TCP only), small models on big clusters can get slower.
• Solution spectrum (measured in llm-lab): gradient compression / delayed sync (DDP improvements) → ZeRO sharding (saves optimizer states, gradient communication drops to shards) → tensor/sequence parallelism (moves communication in-layer, paired with fast scale-up interconnects) → pipeline parallelism (little communication but bubbles). Selection rule: the intersection of the memory and communication ledgers — shard when "state doesn't fit" comes before "communication can't keep up"; split otherwise.

4. Recomputation Ledger: Trading Compute for Memory

Activations are training memory's elastic bulk. Recomputation (activation checkpointing): discard some activations in forward, recompute in backward — saves memory, adds ~1× forward compute (~30-50% total overhead for 50-70% activation memory).

$$\Delta FLOPs \approx +1 \times FLOPs_{fwd}, \quad \Delta memory \approx -k \times memory_{act}$$

When worth it: memory is the hard constraint (batch doesn't fit) and the batch gained outweighs the compute loss; with spare memory, keep it off. Same logic as Chapter 02's HPL "NB blocking": blocking/discarding is about squeezing data movement into what fits on-chip/on-device.

5. Training System Selection Checklist

• [ ] Compute the fixed-state ledger first (weights+gradients+optimizer); it decides GPU count and sharding need; optimizer states are the biggest item
• [ ] Recompute the communication ledger per parallelism: how All-Reduce scales with each doubling, and whether interconnect bandwidth keeps up (measure with ping-pong in Chapter 02 and network chapters)
• [ ] When memory is short, try in order: smaller batch → recomputation → mixed precision/ZeRO sharding → bigger GPUs; decide by the ledger, not gut feel
• [ ] Run communication microbenchmarks (ping-pong/Allreduce) before multi-node training; confirm scaling efficiency isn't negative
• [ ] Compare with the inference ledger: the same cluster's "training profile" and "inference profile" differ; do capacity planning separately

Method note: the memory/communication/recomputation framework follows the open-source book Understanding AI Infra (Li Bojie, Apache-2.0); derivations here are the site's own; measurements link to Chapter 11 llm-lab and Chapter 02 cluster tests.