HBM Memory Architecture and Bandwidth: From HBM3e to Training/Inference Bottlenecks

Note: This is a review article (no first-hand experimental data), based on public technical documentation and architecture analysis.

Experiment Objective

Why, after 2023, do people keep emphasizing "memory bandwidth" rather than "memory capacity" in LLM training? The answer lies in the growth gap between compute and bandwidth: each new GPU generation roughly doubles FP16 compute while HBM bandwidth grows only 40%-60%. On an H100, dense BF16 compute is about 989 TFLOPS with 3.35 TB/s HBM3 bandwidth — for the compute units to stay fed, every byte must sustain an arithmetic intensity of about 295 FLOP/B, yet a large share of Transformer operations (attention, normalization, gradient sync, parameter loading) sits far below that. The goal of this article is a clear bandwidth accounting framework: given a model and cluster configuration, check bandwidth first, compute second, plus an actionable troubleshooting and tuning checklist.

Code Walkthrough

Here is a classic "bandwidth accounting" script. It uses a PyTorch device-to-device copy to measure achievable bandwidth — the first step in deciding whether a configuration is adequate:

import torch, time

def measure_bw(shape, iters=100):
    a = torch.randn(shape, device='cuda', dtype=torch.float16)
    b = torch.empty_like(a)
    # warmup: driver and CUDA context initialization
    b.copy_(a); torch.cuda.synchronize()
    t0 = time.perf_counter()
    for _ in range(iters):
        b.copy_(a)
    torch.cuda.synchronize()
    dt = (time.perf_counter() - t0) / iters
    bytes_moved = 2 * a.numel() * a.element_size()  # read + write
    return bytes_moved / dt / 1e12  # TB/s

# 32GB-class H memory: ~28GB usable
print(measure_bw((6_000_000_000,), iters=50), "TB/s measured copy bandwidth")

Reading it piece by piece:

1. copy_ is a bandwidth probe: one copy reads the whole tensor and writes it back, so bytes moved = 2 * numel * element_size. The measured value approaches 80%-90% of the theoretical number (ECC and memory-controller overhead). This is the "baseline bandwidth" for all later estimates.
2. The warmup is mandatory: the first CUDA call initializes context and stream processors; skipping it would fold first-call latency into the timing and produce a falsely low number.
3. synchronize is the checkpoint: GPU execution is asynchronous; without sync, the loop queues everything and returns instantly — a falsely high number.
4. Use numel * element_size, not nbytes differences: allocation alignment and page sizes inflate nbytes; counting per element is more accurate.

Principles

From GDDR to HBM: why bandwidth is scarce

GDDR (e.g. GDDR6X) raises I/O frequency to pile up bandwidth, at the cost of power and long PCB traces. HBM takes a different route — stacking memory dies vertically and connecting them through a short silicon interposer. Stacking solves capacity (layers × die capacity), short interconnects solve bandwidth (short traces allow wider data buses). HBM3e addresses each pseudo-channel independently, with 8-16 pseudo-channels per stack, greatly increasing memory-access parallelism.

Key public-spec comparisons:

Metric	GDDR6X (RTX 40)	HBM3 (H100)	HBM3e (H200)	HBM3e 12-Hi (B200)
Per-GPU bandwidth	~1 TB/s	3.35 TB/s	4.8 TB/s	8 TB/s
Per-stack capacity	—	16GB	24GB	36GB
Bus width	384-bit	5120-bit	5120-bit	8192-bit

(Public specs, estimated framing.)

Why bandwidth bottlenecks LLMs

Take a 70B model in BF16 (140GB weights):

• Inference, token by token: every generated token must read all weights from HBM once, with arithmetic intensity of only tens of FLOP/B — far below GPU peak. Token throughput ≈ bandwidth / weight bytes: on an H100, 3.35 TB/s ÷ 140GB ≈ 24 token/s (single card, single request). On an H200 at 4.8 TB/s, the same model jumps to ~34 token/s — this is the underlying reason "switching to H200 makes inference faster."
• Attention and RMSNorm in training: these are memory-bound (reading K/V per token, reading statistics), so FlashAttention's core job is turning "read HBM → compute → write HBM" into "read SRAM once → compute → write results only."

Bandwidth accounting framework

For any configuration, check in this order:

1. Weight-load bandwidth: inference throughput ≈ usable bandwidth ÷ per-card weight bytes (pre-KV-cache-hit regime).
2. Activation traffic: training activations ≈ batch × seq_len × hidden × layers × read/write count per layer × bytes, divided per card under tensor parallelism.
3. Communication bandwidth: AllReduce flows over the compute fabric (ROCE), not HBM — but gradient tensors must first be written to HBM before RDMA reads, so weak HBM bandwidth delays communication startup.
4. Compare and judge: if total FLOPs / total HBM bandwidth exceeds per-card compute / per-card bandwidth × cards, the system is bandwidth-bound, and adding cards yields diminishing returns.

Results

Typical estimates using the framework (estimated, for sizing reference):

Scenario	Config	Bandwidth estimate	Conclusion
70B inference (H100)	140GB BF16	~24 token/s	H200 → ~34 token/s; bandwidth buys throughput directly
70B inference (8×H100 TP)	17.5GB/card	weight 1/8 per card	~8× single-card, limited by AllReduce
7B fine-tune (A100 40G)	14GB weights + optimizer state	Adam read/write dominates	Prefer ZeRO/FSDP to cut memory, not more cards

(Estimates from public specs, not measurements.)

Engineering Takeaways

1. Measure bandwidth before discussing compute: before any training/inference system goes live, run measure_bw to establish the achievable-bandwidth baseline and compare it against the vendor spec (80%+ is normal) in the hardware acceptance checklist.
2. Inference sizing should look at HBM bandwidth density: at equal compute, higher-bandwidth cards serve long context and generation better; "GB/s per dollar" is closer to real inference cost than "TFLOPS per dollar."
3. Treat HBM as a scarce resource in design: control activation-checkpointing granularity, whether communication tensors persist in memory, and the KV Cache reserve — every byte of traffic competes for bandwidth.
4. Monitor bandwidth, not just utilization: nvidia-smi dmon -s m fb utilization is only a rough proxy; for precise numbers use DCGM or Nsight Systems memory-workload metrics (the next article covers cluster-level observability).

Next article preview: this one answered "how to read single-card bandwidth"; next we move to cluster view — "GPU Cluster Observability: DCGM Monitoring and Alert Design."