Accelerator Arithmetic: From Spec Sheet to Deliverable FLOPs

The TFLOPS on a spec sheet is a theoretical peak, not what you can actually use. This article breaks a GPU's ledger into three layers: how theoretical FLOPs is derived, how arithmetic intensity decides the bottleneck, and why measured utilization is the only deliverable number. Method framework references the open-source book Understanding AI Infra: Quantitative Analysis and System Design (Bo Jie Li, Apache-2.0); formulas and examples are derived and organized by this site and are reproducible.

1. Layer 1: How Theoretical FLOPs Is Derived

Peak FLOPs of a GPU is decided by three factors:

$$Peak\_FLOPs = cores \times freq \times FLOPs\_per\_cycle$$

• cores: CUDA cores (FP32) or Tensor Cores (FP16/INT8);
• freq: core clock (GHz);
• FLOPs_per_cycle: floating-point ops per core per cycle — 2 for FP32 FMA, 512~1024 for Tensor Core FP16 depending on generation.

Spec ledger for common cards (FP16 Tensor Core theoretical peak):

Card	HBM BW	FP16 Peak	BF16 Peak	Ridge Point
A100 80G	2.0 TB/s	312 TFLOPS	312 TFLOPS	156 FLOPs/Byte
H100 SXM	3.35 TB/s	989 TFLOPS	989 TFLOPS	295 FLOPs/Byte
L40S 48G	864 GB/s	362 TFLOPS	362 TFLOPS	419 FLOPs/Byte

Takeaway: two cards both "FP16" can differ by ~2× due to sparse/dense Tensor Core and boost clocks. Spec numbers are only for order-of-magnitude estimates; delivery depends on Layer 3.

2. Layer 2: Arithmetic Intensity Decides the Bottleneck

Arithmetic intensity (AI) = FLOPs per byte moved:

$$AI = \frac{FLOPs}{Bytes\_moved}$$

Roofline rule: if AI > ridge point (= peak/BW) the op is compute-bound; otherwise bandwidth-bound. From the table:

• L40S ridge ≈ 419 FLOPs/Byte — most LLM ops fall below it and are bandwidth-bound;
• H100 ridge ≈ 295 — relatively more bandwidth headroom, more ops land in the compute-bound region.

This is the quantitative answer to "why big models run slow on L40S": not a lack of compute, but little work possible per byte moved.

3. Layer 3: Decode Is a Bandwidth-Bound Job

In LLM decode, generating one token requires reading all weights once:

$$Bytes\_per\_token \approx N \times bytes\_per\_param, \quad Tokens\_per\_sec = \frac{BW}{Bytes\_per\_token}$$

70B FP16 (140 GB per read) on L40S:

$$\frac{864\ GB/s}{140\ GB} \approx 6.2\ tok/s/card$$

But the "compute ledger" says decode ceiling is 362T / 2×70B ≈ 2585 tok/s — unreachable, because weight-read bandwidth caps it first. Decode is a bandwidth-bound problem; prefill is a compute-bound problem — the two stages need entirely different optimization.

4. Reproducible Python

def gpu_account(bw_gbps, peak_tflops, n_params_b, bytes_per_param=2):
    """Given BW/compute/params, return decode BW ceiling & compute ceiling (tok/s)"""
    bytes_per_token = n_params_b * 1e9 * bytes_per_param          # B
    decode_bw = bw_gbps * 1e9 / bytes_per_token                    # tok/s
    flops_per_token = 2 * n_params_b * 1e9                         # decode 2N FLOPs
    compute_ceiling = peak_tflops * 1e12 / flops_per_token         # tok/s
    return decode_bw, compute_ceiling

gpu_account(bw_gbps=864, peak_tflops=362, n_params_b=70)
# (6.17, 2585.7)  → decode is pinned at ~6 tok/s by bandwidth

5. Measured Practice & Delivery

Use case	Correct basis	Common mistake
Single-card inference QPS	bandwidth ledger (decode) + compute ledger (prefill) separately	derive directly from spec TFLOPS
Training throughput estimate	discount by MFU 40-60%	100% of peak
Card selection	check ridge vs load AI first	look only at TFLOPS

Checklist:

1. For each candidate workload, compute AI and compare with the ridge point to confirm the bottleneck type;
2. Estimate decode throughput with the bandwidth ledger; treat the compute ledger only as an upper bound;
3. Compare measured MFU/utilization with theory; if deviation >30%, debug in the order topology → driver → params → application (see Acceptance & Baselines).

---

Derived and organized by this site; method framework references the open-source book Understanding AI Infra (Apache-2.0). Version numbers and measurements are subject to on-site conditions.