Weight-Quantized Inference: Engineering from FP16 to INT8/INT4/FP8

Note: This is a review article (no first-hand experimental data), based on public practice and quantization literature.

Experiment Objective

Inference quantization is "trading memory and speed for accuracy": INT8 halves memory, INT4 quarters it, but where quantization loss comes from, how to control it, and when it is not worth it — are the recurring decisions of inference engineering. Goal: dissect a complete weight-quantization framework — quantization principles (PTQ/QAT) → calibration data and accuracy evaluation → the INT8/INT4/FP8 selection matrix → the memory/throughput ledger of quantized inference. After reading you can answer: how many bits to quantize to, which method, and how to verify the accuracy loss.

Code Walkthrough

Part 1: PTQ calibration and quantization (INT8 example)

# Post-Training Quantization (PTQ): no retraining; calibration data sets scale factors
from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained("your-model")
# Calibrate: use a small representative set (hundreds) to stat activations, per-layer scales
calib_texts = load_calibration_set(n=512)   # representative, NOT the test set
# Actual quantization libraries (GPTQ/AWQ etc.) do the core here:
#   1) per-layer/per-tensor min/max or percentile → scale
#   2) round weights: q = round(w / scale)
#   3) dequant check: reconstruction error ||w - q*scale|| per layer

Key points:

1. PTQ is the mainstream start: no retraining; a few hundred calibration samples determine quantization parameters; GPTQ/AWQ further minimize per-layer loss via "error minimization" (Hessian-weighted).
2. Calibration data must represent the training distribution: calibrating on the test set or a skewed distribution biases the parameters — calibration-set quality directly decides quantization accuracy.
3. Per-layer reconstruction error is a quick health check: check ||w - q*scale|| layer by layer; locate the large-error layers first (usually a few sensitive layers, keep them high precision).

Part 2: Choosing INT4 and FP8

# INT4 (GPTQ/AWQ 4-bit): memory to 1/4, throughput way up
#   fits: large models memory-constrained, long context, high concurrency (KV share rising)
# FP8 (E4M3): training/inference same precision, small loss, but only half memory
#   fits: new hardware (H100-class native), accuracy-sensitive
# Selection matrix (empirical):
#   FP16 → baseline
#   INT8 → general middle ground (small loss, clear gain)
#   INT4 → memory/concurrency first (when loss acceptable)
#   FP8  → new hardware + accuracy-sensitive

Key points:

• INT4's gain is "concurrency," not just memory: with memory at 1/4, the same card holds more requests/longer contexts (the KV-share problem, ch12's KV article) — quantization decisions link with capacity planning (ch12).
• FP8 is the precision-compression balance: native support on new hardware gives small loss, but only 2× compression (vs INT4's 4×) — accuracy-sensitive → FP8; memory-first → INT4.
• Always measure after quantizing: perplexity/downstream task scores compared (quantized vs FP16); if the drop exceeds a threshold (e.g. ppl +1), not worth it.

Principles

Where quantization loss comes from

Source	Explanation	Mitigation
rounding error	precision lost in weight rounding	higher bits, error-minimizing algorithms (GPTQ)
activation range	activations too wide, wasted quantization steps	per-token/per-channel, better calibration
sensitive layers	a few layers extremely sensitive	keep them high precision (mixed quantization)
outliers	large values widen range, small values lose precision	AWQ activation-aware channel protection

Core insight: quantization loss concentrates in "a few sensitive layers/channels," so "mixed quantization" (most layers INT4, sensitive ones FP16) regains most accuracy for a little memory — don't uniform-quantize; locate sensitive layers first.

PTQ vs Quantization-Aware Training

Method	Needs training	Accuracy	Cost
PTQ (GPTQ/AWQ)	no	medium-high (OK for large models)	low (hours)
QAT	yes (finetune/distill)	high	medium-high
distill to low-precision small model	yes	high (but model smaller)	high

Engineering judgment: large models (70B+) usually suffice with PTQ (parameter redundancy); small/accuracy-sensitive scenarios consider QAT or distillation. Try PTQ first; go QAT only if insufficient.

Memory and throughput ledger (estimated)

Precision	Weight memory (70B)	Relative throughput	Accuracy risk
FP16	~140GB	baseline	none
INT8	~70GB	1.2-1.5×	low
INT4	~35GB	1.5-2.5×	medium (acceptable)
FP8	~70GB	1.2-1.8×	low

(Actual gains vary with KV memory, batch size, hardware — measure.)

Results

(Estimated, not measured)

Option	Memory	Throughput	ppl change (typical)
FP16 baseline	100%	1×	—
INT8 PTQ	~50%	1.2-1.5×	+0.1-0.3
INT4 GPTQ	~25%	1.5-2.5×	+0.3-1.0

Engineering Takeaways

1. PTQ first: large-model PTQ (GPTQ/AWQ) usually suffices; validate in hours; go QAT/distillation only if insufficient.
2. Calibration set is the root of accuracy: representative calibration + per-layer reconstruction checks; half of quantization quality is in the data.
3. Mixed quantization beats uniform: locate sensitive layers (reconstruction error/activation outliers), keep them high precision — a little memory buys most accuracy.
4. Link quantization with capacity planning: INT4's "more concurrency/longer context per card" must feed ch12's capacity formula, not just the memory number.

Next article preview: inference quantization done; back to network foundations — "InfiniBand vs RoCE: Scenarios, Cost, and Operations Compared."