GPU Utilization Ledger: Why It's Low, and How Sharing Saves Cards

GPUs are the most expensive resource in a cluster, yet real utilization often sits at 20-40%. This article breaks "low utilization" into quantifiable ledgers — request sparsity, VRAM fragmentation, isolation overhead — then works out how much continuous batching and sharing lift utilization and how many cards it saves. Method framework references the open-source book Understanding AI Infra (Apache-2.0); formulas and examples are derived by this site and are reproducible.

1. Define Utilization First

$$Utilization = \frac{actual\_useful\_FLOPs}{theoretical\_peak \times time}$$

Note: VRAM occupancy ≠ utilization. A card can be 90% full of tensors while idle-waiting on data, at 15% utilization. Watch compute/bandwidth busyness, not memory.

2. Why Utilization Is Low: Three Ledgers

Ledger 1: request sparsity. Inference requests arrive unevenly; GPUs idle waiting. Night-time utilization can be single digits.

Ledger 2: VRAM fragmentation & bin-packing. Requests differ (models, context lengths); memory is reserved at "worst possible": a 70B service reserves 4 cards of VRAM yet often uses half. Bin-packing granularity waste.

Ledger 3: isolation overhead. Time-slicing has context-switch cost; MIG partitioning loses some compute/bandwidth; multi-tenant cache interference — sharing saves VRAM but costs some performance.

3. Continuous Batching: The Core Ledger That Lifts Utilization

vLLM's core idea is continuous batching: don't wait for a full batch — whoever finishes leaves, a new request enters immediately.

Mode	Idle share	Note
Static batching (run when batch full)	high (tail wait)	slow requests hold the whole batch
Continuous batching	low (finish-and-refill)	reorder requests every step, fill gaps

Quantitative: tail latency shifts from "longest request" to "average request"; throughput can rise 2-3× on the same cards and VRAM. This is the common origin of "same cards, 2× QPS difference."

4. Sharing Options: Isolation vs Overhead

Scheme	Isolation	Overhead	Fits
Whole-card exclusive	strongest	none	training, strict-latency inference
Time-slice (CUDA MPS scheduling)	medium	switching/queueing	low-priority inference, dev
MIG partitioning	hardware-level	compute/BW per share	big-memory, low-compute inference
Multi-tenant + scheduler (vLLM/K8s)	soft isolation	queueing & preemption	production inference clusters

Takeaway: for inference, prefer "continuous batching + scheduler multi-tenancy" before jumping to MIG — MIG's granularity (VRAM multiples) often doesn't align with model VRAM needs, and waste can be worse.

5. Reproducible Python

def utilization_saving(gpus, avg_util, target_util=0.7):
    """Lifting current utilization to target: equivalent cards saved"""
    eff = gpus * avg_util
    needed = eff / target_util
    return gpus - needed, needed

utilization_saving(gpus=16, avg_util=0.3)
# (9.1, 6.9) → 30%→70%: the 16-card job fits in ~7 cards, saving 9

6. Checklist

1. Measure real utilization by compute/bandwidth busyness (not VRAM occupancy) as the improvement baseline;
2. For inference services, enable continuous batching first (vLLM etc.) and observe QPS/utilization — best ROI;
3. Then evaluate MIG/time-slice by "VRAM granularity vs model need"; skip MIG if granularity doesn't align;
4. Do capacity planning by target utilization (function above), not peak requests;
5. Record isolation overhead (P99 latency change) per sharing change; revert to exclusive if unacceptable.

---

Derived and organized by this site; method framework references the open-source book Understanding AI Infra (Apache-2.0). Version numbers and measurements are subject to on-site conditions.