GPU's AI Reinvention: From CUDA Cores to Tensor Cores and NVLink
Today "AI compute" is almost synonymous with GPUs — but GPUs were originally designed for graphics rendering. How did they become the engine of large models? The answer lies in two reinventions: Tensor Cores born for matrix math, and NVLink that stitches many cards into one big GPU. This article explains both, linking to the quantitative method in Accelerator Arithmetic.
1. Why GPUs Are Naturally Good at Matrices
Graphics rendering is massively parallel small math: millions of pixels, each doing independent coordinate transforms and color math. This is structurally identical to large-model forward/backward passes — "many identical operations × huge data."
The key difference is throughput-first:
| CPU | GPU | |
|---|---|---|
| Design goal | low latency per thread | high throughput across threads |
| Cores | a dozen complex cores | thousands of simple cores |
| Good at | branches, dependent tasks | data-parallel, compute-dense tasks |
Takeaway: GEMM is ~99% of LLM compute — "massive multiply-add, no branches" — exactly the GPU sweet spot.
2. CUDA Cores vs Tensor Cores: Two Kinds of Compute
- CUDA Cores (FP32): general-purpose programmable cores, 1 FMA (2 FLOPs) per cycle each. They make the GPU "able to compute anything" — the source of flexibility.
- Tensor Cores (matrix units): specialized small matrix multiply (e.g. 4×4), one instruction does a whole matrix multiply-add. Specialization for efficiency — on the same die area, Tensor Core FP16 throughput is 4-10× the FP32 CUDA Core throughput.
| Card | CUDA Core FP32 | Tensor Core FP16 | Ratio |
|---|---|---|---|
| A100 | 19.5 TFLOPS | 312 TFLOPS | 16× |
| H100 | 67 TFLOPS | 989 TFLOPS | 15× |
| L40S | 90.5 TFLOPS | 362 TFLOPS | 4× |
This is the answer to "why spec FLOPs grew 10×": not more cores, but compute mode switching from "general cores doing it the hard way" to "dedicated matrix units."
3. Tensor Core Generations
| Gen | Architecture | Key addition | Meaning |
|---|---|---|---|
| 1st | Volta (2017) | FP16 mixed precision | first inference/training speedup |
| 2nd | Turing (2018) | INT8/INT4 | quantization for inference |
| 3rd | Ampere (2020) | 2× FP16, sparsity | training cost down |
| 4th | Hopper (2022) | FP8, Transformer Engine | FP8 training possible |
| 5th | Blackwell (2024) | FP4, 2nd-gen TE | lower precision, higher throughput |
Takeaway: each generation's "lower precision" doubles throughput — but the lower the precision, the more mixed-precision checkpoints matter. That is the other half of the compute ledger.
4. NVLink & NVSwitch: Stitching Many Cards into One GPU
When one card can't hold the model (see Supernode Ledger), you need high-speed interconnects. NVLink evolution:
| Gen | Per-link BW | Topology | Meaning |
|---|---|---|---|
| NVLink 1 (P100) | 40 GB/s | dual-card direct | cross-card memory consistency |
| NVLink 2 (V100) | 100 GB/s | dual-card direct | training comm speedup |
| NVLink 3 (A100/H100) | 150-200 GB/s | NVSwitch full-mesh | 8 cards any-to-any |
| NVLink 4 (H200/B200) | 300-450 GB/s | 2nd-gen NVSwitch | superchip form factor |
NVSwitch's key property is full-mesh: any two of 8 cards talk at full bandwidth without relay through intermediate cards — the hardware foundation for "full-mesh vs ring/tree" in Collective Topology.
5. One Diagram of GPU's AI Reinvention
Compute Memory Interconnect Software
─────────────────────────────────────────────────────────────
CUDA Cores ──► VRAM/HBM ──► PCIe (inter-node) ──► CUDA/libs
│ │ │
Tensor Cores ──► NVLink/NVSwitch ──► NCCL (collectives)
(matrix-special) (big VRAM+BW) (multi-card one GPU) (comm abstraction)Core logic: Tensor Cores make "one card compute fast enough," HBM makes "data feed fast enough," NVLink/NVSwitch make "cards cooperate fast enough" — missing any one, a GPU is still a "graphics card," not an "AI engine."
6. Checklist
- When selecting cards, check the Tensor Core precision tier (FP16/FP8/FP4), not just FP32;
- Do the math with the matching precision TFLOPS (see Accelerator Arithmetic);
- For multi-card, confirm NVLink topology first: only full-mesh (NVSwitch) lets collectives run at full bandwidth;
- For cards without NVLink (e.g. L40S), plan intra-node comm at PCIe bandwidth (see L40S Without NVLink).
Organized and written by this site for learning purposes; specific specs and measurements are subject to official docs and on-site conditions.