TensorRT-LLM in Practice: From Model Export to FP8 Inference Acceleration

Summary: vLLM suits "out-of-the-box, dynamic batching"; TensorRT-LLM (TRT-LLM) suits "squeeze the last drop of performance from a fixed model". This post walks the full chain: HuggingFace model → TRT-LLM format → FP8 quantization → engine build → inference serving → benchmark comparison. All commands are reproducible.

0. Experiment Goals

• Convert an HF-format model to a TRT-LLM engine and run inference
• Use FP8 quantization to fit a 70B model in a single 80G card (or significantly raise throughput)
• Understand the semantics of trtllm-build core flags (max-batch-size / max-seq-len / dtype)
• Benchmark against vLLM on the same card and model; give a selection recommendation

1. Environment & Format Conversion

# Official image (recommended, avoids environment pitfalls)
docker run --gpus all -it --shm-size=16g \
  nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3

# HF model → TRT-LLM checkpoint (LLaMA-3-8B as example)
python3 convert_checkpoint.py \
  --model_dir meta-llama/Llama-3-8B \
  --output_dir ./trt-ckpt/llama-8b \
  --dtype float16 \
  --tp_size 1

Step-by-step:

• convert_checkpoint.py: rearranges HF weights into the layout TRT-LLM needs (weights, KV cache, MoE routing are pre-partitioned for the TRT graph). Convert once, build engines many times.
• --tp_size 1: tensor parallelism. 8B fits one card → 1; 70B needs --tp_size 2/4/8 (also determines KV cache and attention splitting across GPUs).
• Check GPU driver support first (nvidia-smi); FP8 quantization (below) requires H100 or Ada-generation hardware (H100 is best).

2. FP8 Quantization (The Key Step)

# Option A: convert from already-FP8 weights (if the model publishes an FP8 version)
python3 convert_checkpoint.py \
  --model_dir meta-llama/Llama-3-8B-Instruct-FP8 \
  --output_dir ./trt-ckpt/llama-8b-fp8 \
  --dtype float16 \
  --qformat fp8

# Option B: PTQ post-quantization (when no FP8 weights exist; calibrate on a dataset)
python3 quantize.py \
  --model_dir meta-llama/Llama-3-8B \
  --dtype float16 \
  --qformat fp8 \
  --calib_dataset ./calib-data.jsonl \
  --output_dir ./trt-ckpt/llama-8b-fp8-ptq \
  --calib_size 512

How it works:

• FP8 (E4M3 weights + E4M3 activations / E5M2 in some spots): halves memory for both weights and KV cache; a 70B drops from ~140GB (FP16) to ~70GB—fits one 80G card.
• PTQ calibration: run a few hundred representative samples to profile activation distributions and find optimal FP8 scales. The calibration set must match real traffic (don't reuse training data).
• Per-tensor vs per-channel: TRT-LLM FP8 quantizes per-layer/per-channel by default; accuracy loss is usually under 1% (imperceptible for generation). If the calibration set is biased, generation quality visibly degrades—calibration quality = quantization quality.

3. Building the Engine

trtllm-build \
  --checkpoint_dir ./trt-ckpt/llama-8b-fp8 \
  --output_dir ./engines/llama-8b-fp8 \
  --max_batch_size 64 \
  --max_seq_len 8192 \
  --max_num_tokens 8192 \
  --gemm_plugin auto \
  --kv_cache_dtype fp8 \
  --use_fused_mlp

Step-by-step:

• --max_batch_size 64 / --max_num_tokens 8192: engines are statically planned for maximum dimensions. Want to raise the concurrency cap? Rebuild the engine—this is the biggest difference from vLLM (vLLM dynamic, TRT static).
• --kv_cache_dtype fp8: KV cache in FP8 halves long-context memory.
• --gemm_plugin auto / --use_fused_mlp: fuse GEMMs and MLP into CUDA kernels. These plugin switches are the core of TRT-LLM speedups—keep them on.

⚠️ Build time: ~3-10 min for 8B, up to 30-60 min for 70B. Once build flags (especially max_seq_len) are set, changing them means a full rebuild.

4. Starting the Inference Service

# Option A: trtllm-serve (single-process quick validation)
trtllm-serve --engine_dir ./engines/llama-8b-fp8 \
  --host 0.0.0.0 --port 8001 \
  --max_beam_width 1

# Option B: Triton + TRT-LLM backend (production recommendation: monitoring, batching, dynamic concurrency)
docker run --gpus all --shm-size=16g --rm \
  -p 8001:8001 -v $(pwd)/engines:/engines \
  nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3 \
  tritonserver --model-repository=/engines/triton-repo

Engineering notes:

• trtllm-serve exposes an OpenAI-compatible API (/v1/chat/completions) for quick validation.
• For production, Triton + TRT-LLM backend is the real path: dynamic batching, Prometheus metrics, multi-model management, and Grace queuing.
• Same as vLLM: never expose the service directly to the public internet; consolidate behind a gateway.

5. Benchmark: TRT-LLM vs vLLM (Same Card, Same Model)

# Use /metrics or a custom load script: 32 concurrent, 128 tokens
# Record: throughput (tok/s) | P50 TTFT (ms) | P95 TTFT (ms)

Results (FP8 · single 80G card · 8B model · sample magnitude):

Engine	Throughput (tok/s)	P50 TTFT	Note
vLLM (FP16)	~3.1k	medium	dynamic batching, easy
TRT-LLM (FP16)	~3.6k	medium	static engine, fused kernels
TRT-LLM (FP8)	~4.4k	low	KV cache also FP8, higher concurrency

Relative-magnitude reference only. The conclusion is not "TRT always wins": vLLM fits dynamic workloads and fast iteration; TRT-LLM offers a 10-40% edge for "fixed model, fixed spec, maximum performance" at the cost of build time and spec lock-in.

6. Selection Guide (copy-paste ready)

Scenario	Recommendation	Why
Fast launch / many models / experimentation	vLLM	dynamic batching, no rebuilds
Single flagship model, fixed spec, max throughput	TRT-LLM	fused kernels + FP8
70B+ single-card inference (memory-tight)	TRT-LLM FP8	FP8 weights+KV halves memory
Unified gateway / multi-model entrance	Triton	multi-backend, batching, monitoring

7. Engineering Takeaways

1. TRT-LLM is a "static-spec" engine: changing max_seq_len/batch requires a rebuild.
2. FP8 in three steps: driver support (H100/Ada) → weight conversion or PTQ → build with kv_cache_dtype fp8.
3. Calibration-set quality = quantization quality; calibrate on realistic traffic.
4. Plugin switches are the speed core: keep gemm_plugin/use_fused_mlp on.
5. Production runs on the Triton backend: dynamic batching, monitoring, multi-model management.
6. Complement vLLM rather than compete: vLLM for iteration, TRT-LLM for squeezing the flagship.
7. Version your build flags: put engine build flags in CI config; change flags = new engine version, traceable.

8. Further Reading

• TensorRT-LLM docs
• Triton Inference Server docs
• KB: 12. Inference Engines · vLLM Production Deployment