GPU Cluster Capacity Planning: From Business Requirements to Racks & Power Budget

Experiment Goal

"Buy 1000 GPUs first" isn't capacity planning — planning derives from business requirements backwards: how much compute for training/inference, which machine types, how many racks, whether power suffices, and how cost and delivery cadence are sequenced. Wrong planning means overweight racks/power trips at best, capacity shortfall and rework at worst.

This article walks through GPU cluster capacity planning with reproducible calculation methods: requirements → total compute → machine types & racks → power & cooling → cost & cadence. Note: formulas are generic engineering methods; concrete numbers vary with workload and hardware; figures in this article are estimation-standard examples, not fabricated benchmarks.

Section 1: From Business Requirements to Total Compute

Translate "business requirements" into "compute requirements". Estimate the two main loads separately:

Training Load

Training compute (PFLOPS·days) ≈ params × training tokens × 6 / (utilization × card utilization × availability)

• 6 = FLOPs multiple per token (fwd+bwd, ~6× params);
• Utilization: training MFU (Model FLOPs Utilization) is only 30-50% in practice (communication, bubbles, load imbalance eat over half);
• Example: 70B model, 1T tokens → 70e9 × 1e12 × 6 ≈ 4.2e23 FLOPs; assume MFU 40%, H20 FP16 ~148 TFLOPS → ~7100 card·days → 256 cards for ~28 days (estimation standard; varies by machine and parallelism).

Inference Load

Inference compute ≈ peak QPS × avg output tokens × compute per token / per-card throughput

• Inference plans by "throughput × latency", not total compute (Chapter 12 benchmark method);
• Capacity = card count under peak QPS and SLO (TTFT/TPOT) constraints, not average load.

Engineering observation: estimate training by "total card·days" and inference by "peak throughput" separately, then add business peaks and a redundancy factor (e.g. 20%) — mixing the two load types is the most common planning mistake.

Section 2: Machine Types & Rack Layout

Dimension	Considerations	Decision points
GPU model	compute/VRAM/interconnect (Chapter 01 selection)	training prioritizes interconnect (NVLink/IB), inference prioritizes VRAM
Cards per node	8-card SXM vs lower	more cards per node = higher rack density but higher cooling/power demands
Rack layout	4-6 × 8-card nodes per rack	group by Chapter 03 network topology (SU units), same-rail traffic nearby
Switch positions	Leaf/Spine/SuperSpine layers	reserve fiber runs and inter-rack interconnect ports

Engineering observation: rack layout must be designed from the network topology first (Chapter 03 rail principle: same-numbered GPUs converge nearby); re-cabling after delivery is extremely costly.

Section 3: Power & Cooling — Where Planning Most Often Trips

Per-rack power ≈ per-node power × node count + switch power + redundancy margin

• Rated vs actual power: plan by PSU rating and GPU TDP for "full-load ceiling"; calculate running power by typical load — plan power by the ceiling, bill electricity by actuals;
• Redundancy: dual feed + PSU N+1, UPS/diesel capacity by rack peak;
• Cooling: air vs liquid (Chapter 01 liquid cooling) — high-density racks (≥30kW) strain air cooling; liquid becomes necessary. Decide cooling form after computing per-rack average power;
• Floor load: high-density/liquid racks exceed ordinary floor design loads — requires structural assessment. Confirm at planning stage; discovering at delivery is a disaster.

Section 4: Cost & Delivery Cadence

• Cost standard: hardware + power/cooling (the TCO big-ticket) + network + facility + ops; convert to "per card·hour" for business pricing and budgeting;
• Delivery cadence: phased delivery (small-batch acceptance → expansion) beats one-shot — Chapter 01/05 baseline hardening plus the Chapter 05 provisioning pipeline support phasing; schedule by batch with acceptance gates;
• Reserve expansion: reserve 20-30% of power/racks/network ports so expansion doesn't touch infrastructure.

Planning Checklist

• [ ] Training and inference estimated separately (card·days vs peak throughput), summed with redundancy
• [ ] Machine selection aligned with interconnect needs (training: interconnect first)
• [ ] Racks laid out by network topology (same-rail nearby); fiber runs reserved
• [ ] Power planned by full-load ceiling, billed by actuals; redundancy & UPS ready
• [ ] Cooling form chosen by per-rack average power (high density → liquid); floor load assessed
• [ ] Cost standard unified (per card·hour); phased delivery with acceptance gates

Engineering Takeaways

• Planning derives from business backwards: first the demand model (training tokens, inference peak), then total compute, racks, power — never the reverse order.
• Power & floor load are hidden gates: capacity planning often fails at rack power and floor load; confirm at planning stage, not at delivery.
• Phased delivery is part of planning: leave expansion margin, batch cadence, and acceptance gates rather than "all at once".

Next article preview: with capacity planning landing on racks, the storage scheme follows — the next article, Training Data Storage Selection: Object Store, Parallel File Systems & Local NVMe, breaks down the roles and boundaries of three storage classes.