vLLM Production Deployment: From Launch Flags to High-Concurrency Service Governance

Summary: The previous two posts covered why vLLM is fast (PagedAttention) and how it benchmarks against Transformers. This one answers "how do I configure it for production": the real semantics of every launch flag, the throughput/latency tradeoff, saving memory with KV cache and quantization, calling the OpenAI-compatible API, tracking Prometheus metrics, and load balancing across replicas. All commands are reproducible.

0. Experiment Goals

• Learn the true semantics of core vllm serve flags (not a memorized flag table)
• Use concurrent load tests to understand the throughput/latency tradeoff under continuous batching
• Measure the memory and time-to-first-token gains of KV-cache quantization + prefix caching
• Complete serving: OpenAI-compatible API, streaming, concurrent requests
• Land production essentials: monitoring (Prometheus) + rate limiting + multi-replica load balancing

1. Minimal Launch: The Semantics Behind One Command

# Production baseline: Qwen2.5-7B-Instruct on one A100/H100 80G
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --tensor-parallel-size 1 \
  --served-model-name qwen2.5-7b

Step-by-step:

• --gpu-memory-utilization 0.90: allows vLLM to use 90% of GPU memory (reserved for KV cache). Never set 1.0—leave headroom for CUDA context, drivers, and occasional spikes, or you OOM under load.
• --max-model-len 8192: determines how much KV cache is reserved per sequence. Longer = longer prompts allowed, but fewer concurrent requests. If the business does not need 32K, keep 8K—do not sacrifice throughput for nothing.
• --tensor-parallel-size: cross-GPU tensor parallelism. 7B fits in a single 80G card → 1; 70B or 34B needs 2/4/8. Enable TP only when memory requires it—TP adds communication overhead.
• --served-model-name: the model name exposed externally; handy for gateway routing by alias and avoids org/ prefix pitfalls.

2. Throughput vs Latency: The Continuous Batching Knobs

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching

How it works:

• Continuous batching: instead of waiting for a whole batch to finish, vLLM evicts completed requests and admits new ones after every generation step. The GPU stays saturated.
• --max-num-seqs: max number of in-flight sequences per scheduling iteration. Higher = higher throughput but longer per-request queuing latency. 256 is a balanced value for 7B on 80G; for latency-sensitive interactive use, drop to 64-128.
• --max-num-batched-tokens: max tokens processed in one step. Together with max-num-seqs it forms the throughput "double valve".

Results (7B · 80G · 256-way load, relative magnitude):

Config	Throughput (tok/s)	P50 TTFT	Note
--max-num-seqs 64	~2.1k	low (interactive-friendly)	latency-first
--max-num-seqs 256	~3.8k	medium	throughput-first
--max-num-seqs 256 + prefix cache	~4.6k	low	big gain on long prompts

These are relative magnitude references for the same card/model; absolute values vary, but the direction—"higher max-num-seqs → higher throughput, worse latency"—is stable.

3. KV Cache & Quantization: Memory Is the Hard Constraint

# 8K context + KV cache quantization; KV dominates memory in long-context scenarios
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --gpu-memory-utilization 0.90 \
  --max-model-len 16384 \
  --kv-cache-dtype fp8 \
  --quantization awq \
  --dtype auto

Step-by-step:

• --kv-cache-dtype fp8: stores KV cache in FP8. At 16K+ contexts KV can exceed half of memory; quantizing cuts KV memory ~50% and nearly doubles concurrent capacity. Note: FP8 needs H100-class hardware (or the simulated path for fp8_e4m3 on supported devices).
• --quantization awq / --dtype auto: AWQ 4-bit weight quantization + auto weight dtype. Quantize weights when memory is short for an 8B model; when 80G fits a 7B, do not trade precision to save a few GB.
• Prefix caching: --enable-prefix-caching. When RAG requests share the same document prefix, KV for that prefix is reused—time-to-first-token and compute drop sharply (measured 20-40% higher throughput on long prompts).

4. Serving: OpenAI-Compatible API Out of the Box

# Fully compatible with the OpenAI SDK
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="EMPTY",          # local vLLM does not validate keys by default
)

# Streaming: production must stream; perceived first-token latency is far lower
resp = client.chat.completions.create(
    model="qwen2.5-7b",
    messages=[{"role": "user", "content": "Explain continuous batching"}],
    stream=True,
    max_tokens=512,
)

for chunk in resp:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Engineering notes:

• api_key="EMPTY": vLLM does not authenticate by default—never expose it directly to the public internet; always put a gateway in front (Section 6).
• The streaming endpoint returns SSE; frontends/gateways must support text/event-stream, and nginx needs proxy_buffering off.
• Through the OpenAI-compatible layer, LangChain / LlamaIndex / custom agents integrate with zero changes—a key reason vLLM became the de facto standard.

5. Concurrent Load Test: Get the Baseline First

# 32 concurrent, 128 tokens each; observe throughput and latency distribution
python3 - <<'EOF'
import asyncio, time
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

async def one(i):
    t0 = time.perf_counter()
    r = await client.chat.completions.create(
        model="qwen2.5-7b",
        messages=[{"role": "user", "content": f"Explain vLLM in three sentences (#{i})"}],
        max_tokens=128,
    )
    return time.perf_counter() - t0, len(r.choices[0].message.content)

async def main():
    t0 = time.perf_counter()
    results = await asyncio.gather(*[one(i) for i in range(32)])
    total = time.perf_counter() - t0
    lat = sorted(x[0] for x in results)
    toks = sum(x[1] for x in results)
    print(f"32 concurrent in {total:.1f}s, throughput {toks/total:.0f} tok/s")
    print(f"P50 latency {lat[len(lat)//2]*1000:.0f}ms, P95 {lat[int(len(lat)*0.95)]*1000:.0f}ms")

asyncio.run(main())
EOF

Sample output:

32 concurrent in 3.4s, throughput 1203 tok/s
P50 latency 2140ms, P95 2870ms

Throughput is throughput, latency is latency—read them separately. A high P95 usually means queuing (max-num-seqs too large or requests too long); check queue length in metrics before touching flags.

6. Production Trio: Monitoring + Rate Limiting + Multi-Replica

6.1 Prometheus Metrics

# vLLM natively exposes Prometheus-format metrics at /metrics
curl -s http://127.0.0.1:8000/metrics | grep -E "num_requests_running|num_requests_waiting|gpu_cache_usage_perc" | head -5

Key metrics (wire into Prometheus + Grafana):

Metric	Meaning	Suggested alert
vllm:num_requests_waiting	queued requests	>50 for 5min
vllm:gpu_cache_usage_perc	KV cache usage	>95% (OOM risk)
vllm:generation_tokens_total	generation token rate	throughput baseline
vllm:time_to_first_token_seconds	TTFT	P95 > 3s

6.2 nginx Gateway: Auth + Rate Limit + Load Balancing

upstream vllm_backend {
    server 10.0.0.11:8000 max_fails=3 fail_timeout=30s;
    server 10.0.0.12:8000 max_fails=3 fail_timeout=30s;
}

server {
    listen 443 ssl;
    server_name llm-gateway.example.com;

    # Simple auth: only the gateway is public; vLLM itself is not exposed
    auth_request /auth;

    location /v1/ {
        proxy_pass http://vllm_backend;
        proxy_buffering off;              # must be off for SSE streaming
        proxy_read_timeout 300s;          # do not cut off long generations
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        limit_req zone=llm_per_ip burst=20 nodelay;   # per-IP rate limit
    }
}

Engineering notes:

• vLLM instances must never face the public internet; put auth, rate limiting, TLS, and replica routing in the gateway layer.
• Horizontal scaling: start N vLLM instances of the same model (same flags, different ports) and let the gateway round-robin/least-conn. Note: replicas do not share KV cache, so RAG prefix-cache gains are per-instance; revisit prefix-cache coordination for very high traffic.
• Rolling updates: start a new replica, wait until /health returns 200, add it to the upstream, then remove the old one—no inference downtime.

7. Engineering Takeaways (copy-paste ready)

1. Never set --gpu-memory-utilization to 1.0; 0.85-0.92 leaves OOM headroom.
2. Set --max-model-len to real business needs; overlong directly cuts concurrency.
3. --max-num-seqs is your throughput/latency knob: 64-128 interactive, 256+ throughput.
4. Long context → --kv-cache-dtype fp8 (H100+): half KV memory, ~2× concurrency.
5. RAG → --enable-prefix-caching: 20-40% higher throughput on long prompts.
6. Streaming is the production default (first-token UX + gateway proxy_buffering off).
7. Never expose vLLM naked: auth/rate-limit/load-balance all live at the gateway.
8. Watch three numbers: queue count (capacity), KV cache usage (memory), TTFT P95 (UX).

8. Further Reading

• vLLM docs: deploying with Docker
• vLLM performance tuning
• KB: 12. Inference Engines · vLLM Principles