Scaling Law Ledger: The Loss-Compute-Data Triangle

"Scale is magic" is the industry intuition, but how should you scale — more parameters or more data? That is what Scaling Law answers. This article explains the quantitative relationship among loss, compute, and data, linking to Pretrain Compute Planning.

1. Standard Scaling Law: Loss = Compute + Data + Irreducible Floor

Empirically, model loss L decomposes into three parts:

$$L(N, D) = \frac{A}{N^\alpha} + \frac{B}{D^\beta} + E$$

• N: parameter count, D: training tokens;
• A/N^α: loss from a too-small model (power-law decay in N);
• B/D^β: loss from insufficient data (power-law decay in D);
• E: irreducible loss (data entropy/noise — more compute can't push it to zero).

Takeaway: Scaling Law isn't mysticism — it's "loss = sum of three power laws," which directly says piling parameters on one side is wasteful; feed both sides.

2. The Compute Floor: 6ND

The compute floor for one training run (without repeating data):

$$C \approx 6 \times N \times D \quad (\text{FLOPs})$$

• Forward 2ND + backward 4ND (~6 FLOPs per token per param);
• This is the theoretical floor; real runs multiply by 1.2-2 (activations, recompute, comm);
• It compresses three variables into one equation: given compute C, you can't greed on both parameters and data.

3. Kaplan vs Chinchilla: The Allocation Debate

Conclusion	Claim	Allocation
Kaplan (2020)	with limited compute, add parameters	data grows slower than params
Chinchilla (2022)	grow params & data together, ~20 tokens/param	optimal: D ≈ 20N

Engineering meaning: per Chinchilla, a 70B model optimally needs ≈1.4T tokens; forcing big params with scarce data wastes compute. Today's models mostly train "above Chinchilla" (over-trained), squeezing data value.

4. Inference-Time Scaling: A New Ledger for Compute-to-Quality

Standard Scaling Law covers only training-time compute; since 2024, "thinking models" (RL + long chains) moved compute into inference time:

$$Quality = f(C_{train}, N, D) \quad \xrightarrow{\text{test-time compute}} \quad Quality = f(C_{train}, N, D, C_{inference})$$

• Where inference compute goes: more samples, longer chains, self-correction (see Output Sampling Ledger);
• Key trade-off: the same budget — train a bigger model, or think longer at inference? The latter iterates fast and is cost-controllable — a realistic path for small teams.

5. Reproducible Python

def chinchilla_opt(n_params_b):
    """Chinchilla-optimal data (tokens) and compute floor (FLOPs)"""
    d_opt = 20 * n_params_b * 1e9                # 20 tokens/param
    c_min = 6 * n_params_b * 1e9 * d_opt         # 6ND
    return f"{d_opt/1e12:.1f}T tokens", f"{c_min/1e24:.1f}E FLOPs"

chinchilla_opt(70)
# (1.4T tokens, 5.9E24 FLOPs ≈ 5.9 ZFLOPs)

6. Checklist

1. Before sizing a model, run the Chinchilla ratio: is data ≥ 20 tokens/param?
2. Estimate the compute floor with 6ND, then multiply by 1.2-2 for the real budget (see Compute Planning);
3. With data surplus, over-training (>20 tokens/param) is fine; with data scarcity, shrink the model — don't force it;
4. Evaluate inference-time scaling's cost/benefit against "a bigger model" before deciding;
5. Record (N, D, C, loss) per run to verify your own scaling curve (see Acceptance & Baselines).

---

Organized and written by this site for learning purposes; formulas and examples are the site's own; specific versions and measurements are subject to on-site conditions.