LLM Inference Benchmarking: Throughput, TTFT & Cost in Practice

Experiment Goal

Before an inference engine (vLLM/TensorRT-LLM etc., see Chapter 12 articles) goes live, "feels fast" isn't enough — production decisions need a unified, reproducible, comparable inference benchmark:

• How to compare engines/quantizations/concurrency settings under the same standard?
• Can "high throughput" and "fast first token" coexist? What's the relationship between metrics?
• How fast is a request "really", measured from both the engine's own tools and external load?

This article uses vLLM's built-in benchmark tools as the main line, dissecting inference benchmarking: metric semantics, tool parameters, concurrency modeling, and cost accounting. Commands are reproducible operations; concrete numbers vary with model/hardware, so methods are described without fabricating benchmarks.

Section 1: Unify the Metric Language — Five Numbers

Metric	Meaning	Who cares
Throughput	output tokens per second, or requests/sec	capacity planning
TTFT (Time To First Token)	latency to first output token	interactive UX (chat/streaming)
TPOT (Time Per Output Token)	interval between output tokens	streaming feel ("typing" sensation)
ITL (Inter-Token Latency)	observed measure of TPOT	streaming UX verification
E2E Latency	full request from start to end	per-request UX / timeout design

Key relationship: as concurrency rises, throughput first increases then decreases (the knee is the capacity ceiling), while TTFT/TPOT degrade — "high throughput" and "fast response" trade off on the same resources. Benchmarks must report both; reporting only one is cheating.

Section 2: vLLM Built-in Benchmarks — Command & Parameter Breakdown

vLLM ships two tools: benchmark_throughput.py (offline, throughput only) and benchmark_serving.py (online, with latency and concurrency modeling).

# Offline throughput: fixed input/output lengths, pure engine ceiling
python3 benchmark_throughput.py \
  --model /models/Qwen2.5-72B-Instruct \
  --backend vllm --num-prompts 2000 \
  --input-len 512 --output-len 256 \
  --max-num-seqs 128

# Online serving benchmark: simulates a real request stream
python3 benchmark_serving.py \
  --model /models/Qwen2.5-72B-Instruct \
  --backend vllm --endpoint /v1/completions \
  --num-prompts 500 \
  --request-rate 20 \          # new requests/sec (Poisson arrival)
  --input-len 512 --output-len 256 \
  --percentile-metrics "ttft,tpot,itl,e2e_latency"

Parameter semantics:

Parameter	Meaning	Tuning notes
--input-len/--output-len	request input/output token lengths	align with real load distribution; long output = pressure on decode
--request-rate	requests arriving per second (0 = full concurrency)	models real arrival; increase to observe throughput knee and latency degradation
--num-prompts	total requests	cover steady state (the first tens of requests are warmup/queuing, not counted)
--percentile-metrics	which metrics' percentiles to output	look at P50/P95/P99, not averages (tails decide UX)

Engineering observation: the question a benchmark must answer is "which load band is my real traffic in" — request length distribution, arrival rate, concurrency peaks drive the parameters. Numbers from default parameters can't guide capacity planning.

Section 3: External Load — Beyond Engine Tools

Engine tools measure the "engine view"; production also needs a "client view" (network, gateway, batching queues all included):

# Use a generic load tool (locust/wrk/custom script) against the real API
# Key: the client records TTFT/TPOT/E2E, not just engine logs

External load covers three blind spots:

1. Network & gateway overhead: the network round-trips and gateway queuing inside TTFT are invisible in engine logs;
2. Real arrival patterns: burst vs steady — model with Poisson/spike arrivals, not uniform rate;
3. End-to-end timeout design: how client timeouts and retry policies affect "perceived latency".

Section 4: Cost Accounting — Turning Performance into Money

Performance metrics ultimately land in cost decisions:

Cost (per million output tokens) ≈ service cost (GPU amortization + energy + ops) / throughput × time

• GPU amortization: by dedicated GPU count and unit price (direct input to capacity planning);
• Throughput-cost knee: beyond the knee, throughput stops growing while latency grows — adding concurrency beyond it burns money for worse UX. Find the knee; set capacity before it;
• Quantization/engine comparison: under the same standard (same model & hardware), compare FP8 quantization and different engines' throughput-latency curves and cost — that's how you answer "is switching engines worth it".

Inference Benchmark Checklist

• [ ] Metrics report both throughput and TTFT/TPOT percentiles (P50/P95/P99), never throughput alone
• [ ] Request lengths/arrival rates model the real load distribution, not defaults
• [ ] Concurrency knee covered (increase request-rate stepwise, watch throughput & latency)
• [ ] External load adds network/gateway/timeout views
• [ ] Cost standard unified (GPU amortization + throughput knee); cross-engine/quantization same-standard
• [ ] Benchmark commands and results archived; rerun after scaling/config changes

Engineering Takeaways

• Metrics are only meaningful in pairs: throughput and latency are two sides of one coin; reporting only one is cheating.
• Load modeling decides decision value: model real distributions (length, arrival, concurrency) so results guide capacity and cost.
• Benchmarks are a regression ruler: rerun the same command after every engine/quantization/GPU change; decisions then have evidence.

Next article preview: with inference measurement ready, we return to the AI×Science main stage — the next article, Weather AI Models: From Paper Architectures to Operational Forecast Services, breaks down the landing path of a representative AI4S field.