Pretraining Compute & Cost Planning: GPU Estimation & Card-Hour Budget

Experiment Goal

Pretraining is the big "card burner", yet many teams have only a vague sense of "how long this model trains and what it costs". Compute and cost planning turns that feeling into a calculable budget:

• Given model parameters and training data volume, how to estimate card-hours?
• How do MFU and parallel efficiency affect actual demand?
• How much compute goes to training/fine-tuning/inference/experiments, and how to allocate?

This article dissects pretraining compute and cost planning with formulas and methods. Note: formulas are generic engineering methods; concrete numbers vary with model/hardware/cluster; figures are estimation-standard examples, not fabricated benchmarks.

Section 1: Card-Hour Estimation — From Model and Data

Training card-hours ≈ params × training tokens × 6
                    ÷ (per-card compute × MFU × availability)

Example: 70B model, 1T tokens, MFU 40%,
    H20 FP16 ~148 TFLOPS
    card·days ≈ 70e9 × 1e12 × 6 / (148e12 × 0.4) ≈ 7100 card·days
    i.e. 256 cards for ~28 days (estimation standard)

Parameter meaning & pitfalls:

• 6× params: per-token fwd+bwd FLOPs approximation (excluding attention details; fine for estimation; use a FLOPs model formula for precision);
• MFU (Model FLOPs Utilization): actual utilization 30-50% — communication, bubbles, load imbalance, and checkpoint interruptions all eat compute. Estimating at 100% is self-deception;
• Availability factor: clusters aren't 7×24 failure-free — fault recovery, maintenance, and queueing losses count at 80-90% availability;
• Data efficiency: start at the Chinchilla-optimal ratio (~20× params) for training tokens; adjust by experiments and business goals.

Section 2: MFU & Parallel Efficiency — Turning "Paper Compute" into "Actual Compute"

Paper compute (per-card TFLOPS × cards) isn't actual throughput; two corrections:

Correction	Impact	How to improve
Parallel efficiency	TP/PP/EP communication & bubbles (Chapter 09 engine article)	parallel dimension config optimization, comm-compute overlap
Load balance	MoE routing skew (Chapter 10)	load-balancing loss, EP granularity
Checkpoint/fault-tolerance overhead	periodic stalls (Chapter 10 engineering article)	checkpoint frequency & async strategy

Engineering observation: plan with an "experience MFU range (30-50%)" rather than peak compute. The same model differs by 20%+ MFU across parallel configs — measure MFU on a small run first, then scale up the projection; far more accurate than one-shot full-scale estimation.

Section 3: Compute Allocation — Training Isn't Everything

Cluster compute serves multiple loads; the allocation decides "how many cards":

Load	Compute traits	Allocation reference	Notes
Pretraining	large blocks, long, exclusive	60-70%	card-hour big-ticket; exclusive windows
Fine-tuning / alignment	small-medium, batches	10-15%	reuses training-cluster fragments
Inference service	resident, low concurrency high availability	10-20%	Chapter 12 capacity method; dedicated or shared
Experiments / dev	short tasks, peaky	5-10%	queue scheduling absorbs fragments

• Allocation shifts with business phase (pretraining period: training share high; go-live: inference share rises);
• Fragment management: fine-tuning/experiment tasks fill training gaps to avoid idle compute — scheduling strategy (Chapter 06) cooperates.

Section 4: Cost Standards & Budgeting Flow

Card-hour cost ≈ per-card·hour cost × total card-hours
Per-card·hour ≈ (hardware amortization + power + cooling + facility + ops) / (cards × annual available hours)

Budgeting flow:

1. Requirements list: itemize each load's card·day/throughput demand with Chapter 02 planning methods;
2. Total compute: sum + redundancy factor (20-30%), convert to card count & machine types;
3. Cost accounting: per-card·hour × total, billed separately for training/inference/experiments;
4. Budget windows: sequence by batches (Chapter 05 pipeline) and business cadence, each batch with an acceptance gate and cost check;
5. Monitor & review: actual card-hours vs budget (utilization/fault time/queue time); calibrate the next budget with measurements — budget models iterate with measurements, not set once.

Planning Checklist

• [ ] Card-hours estimated with an MFU range (30-50%), never 100% paper compute
• [ ] Availability factor (faults/maintenance/queueing) included
• [ ] Training/fine-tuning/inference/experiment allocation explicit, adjusted by phase
• [ ] Per-card·hour cost standard unified (amortization + power + cooling + ops)
• [ ] Budget sequenced by batch, acceptance gates + cost checks per batch
• [ ] Actual card-hours monitored and reviewed; next budget calibrated with measurements

Engineering Takeaways

• Planning is calculated, not felt: card-hours, MFU, allocation, cost all have formulas and standards — measure MFU small-scale first, then scale the projection;
• Utilization is the first cost lever: a 10% MFU gain saves 10% of card-hours; optimizing parallel config beats cutting budget;
• Budgets must be reviewable: the budget model iterates with measurements — utilization, faults, and queueing all go into the ledger for credibility.

Next article preview: with the compute accounting settled, training enters the long run — the next batch will continue deepening Chapter 12 inference and Chapter 13 applications. Stay tuned.