Operator & Runtime Ledger: Roofline, Kernel Fusion, and FlashAttention I/O

Two people run the same model at very different speeds; the gap is usually in "how many times data got moved." This article uses three ledgers to explain operator-level optimization: roofline finds the bottleneck, fusion reduces movement, FlashAttention drops I/O complexity a notch. Method framework references the open-source book Understanding AI Infra (Apache-2.0); formulas and examples are derived by this site and are reproducible.

1. Roofline: Identify the Bottleneck Before Optimizing

Execution time of any op ≥ max(data movement time (bandwidth wall), compute time (compute wall)):

$$T = \max\left(\frac{FLOPs}{Peak}, \frac{Bytes}{BW}\right)$$

Rule: if arithmetic intensity AI > ridge point (ridge = Peak/BW) the op is compute-bound; otherwise bandwidth-bound. L40S ridge ≈ 419 FLOPs/Byte, and most LLM ops (AI often 10~100) fall in the bandwidth-bound region — optimize by moving less, not by stacking compute.

Takeaway: compute AI first, then pick the optimization — this eliminates most "blind optimization." For bandwidth-bound ops, kernel rewriting beats tuning thread counts.

2. Kernel Fusion: Two Round-Trips Saved

Two consecutive kernels each read/write an intermediate tensor:

unfused: A → write intermediate(VRAM) → read back → B      movement = 2×(write+read) × size
fused:   A→B on-chip (registers/SMEM)                        movement = single input/output

Fusion saves intermediate size × 2 writes + 2 reads of movement. For bandwidth-bound ops (most), fusing one intermediate tensor saves 30-50% of execution time. This is the principle behind FlashAttention, fused GELU, fused RMSNorm: put the I/O ledger into the op boundary, not just FLOPs.

3. FlashAttention: I/O Complexity Down a Notch

Standard attention's problem is not FLOPs — it's repeatedly reading/writing the S=P×Q matrix:

Implementation	VRAM I/O complexity	Note
Standard attention	O(N²)	every attention score matrix lands in VRAM and is read back
FlashAttention	O(N)	tiled + online softmax; score matrix never materializes

The longer the sequence, the bigger the gap: at 32K sequence, the standard intermediate matrix is 32K×32K×4 bytes ≈ 4 GB/layer, which FlashAttention avoids entirely. In long-context inference and training, I/O complexity is more fatal than FLOPs complexity.

4. CUDA Graph: Cutting "Instruction Movement" Overhead

For small kernels, the bottleneck may be launch overhead (~µs × thousands of ops). CUDA Graph bakes the whole graph into one submission:

$$T_{graph} \approx T_{compute} + T_{launch\_once}, \quad vs \quad T_{loop} \approx T_{compute} + N \times T_{launch}$$

For decode — many small ops, repeated every step — CUDA Graph saves 20-50% of per-step overhead (see Inference Engine Tuning and why vLLM enables it by default).

5. Reproducible Python

def roofline(peak_tflops, bw_gbps, flops, bytes_moved):
    """Classify compute/bandwidth bound; return predicted time (ms)"""
    t_compute = flops / (peak_tflops * 1e12)          # s
    t_mem = bytes_moved / (bw_gbps * 1e9)             # s
    return max(t_compute, t_mem), "compute" if t_compute > t_mem else "bandwidth"

roofline(peak_tflops=362, bw_gbps=864, flops=2e12, bytes_moved=8e9)
# (9.3e-3, 'bandwidth') → bandwidth-bound, move less first

6. Checklist

1. Compute AI vs ridge before optimizing; confirm bottleneck type;
2. Bandwidth-bound → look for fusable intermediates (each fusion saves 2 writes + 2 reads);
3. Long-sequence workloads: check attention uses FlashAttention-class I/O optimization;
4. Decode-class small-op workloads: enable CUDA Graph;
5. Profile before/after each change and archive results (see Acceptance & Baselines).

---

Derived and organized by this site; method framework references the open-source book Understanding AI Infra (Apache-2.0). Version numbers and measurements are subject to on-site conditions.