Private LLM Serving Architecture: From GPU Selection to Production High Availability

Scenario: An enterprise wants to deploy its own LLM Q&A / writing / coding services on the intranet — data stays in-domain, customizable, cost-controllable. Between "buy a few GPUs" and "serve thousands of users reliably" lies a full engineering chain. This article lays out a practical private LLM serving architecture, connecting the mechanism principles in 12 · Inference Engines with the application side of Production-Grade Knowledge-Base Q&A.

1. Scenario and Decision Framework

Private deployment has three typical constraints:

Constraint	Problem	Decision impact
Data compliance	data cannot leave the domain	private inference mandatory; public-cloud APIs forbidden
Performance experience	internal staff/customers need fluency	throughput and latency have SLAs
Cost boundary	limited compute budget	balance model size, concurrency, hardware

Core decision order: first decide "how many users, what response requirement," then "model scale," and only then "which GPUs" — doing it in reverse is a guaranteed pitfall.

2. GPU Selection: Compute Memory First, Then FLOPs

2.1 Memory Estimation (Inference Side)

Inference memory ≈ model weights + KV Cache + activations/temp buffers. Quick formulas:

weights memory ≈ parameter count × bytes
  FP16/BF16: params × 2 bytes
  INT8: params × 1 byte
  INT4: params × 0.5 byte
KV Cache ≈ 2 × layers × KV heads × head dim × seq len × batch size × 2 bytes

Example (7B model, FP16, 4096 context):

Item	Calc	Memory
Weights	7B × 2B	~14 GB
KV Cache (per concurrency)	grows with seq & concurrency	~0.5–2 GB
Single-GPU floor total		~16 GB (4090 runs, but concurrency limited)

Quick conclusion: 7B on a single 24G card (4090/A10) runs but with limited concurrency; 70B starts at 4×A100-80G (FP16) or 2×A100 (INT8/quantized); 100B-class needs multi-GPU TP (tensor parallelism — see 11 · Tensor Parallelism).

2.2 Inference GPUs vs Training GPUs

Scenario	Recommended	Why
Pure inference, high concurrency	inference GPUs (L20/L40S/4090/dedicated)	best memory-bandwidth price/performance
Mixed training + inference	A100/H100	training needs NVLink and high FLOPs
Budget-sensitive	consumer large-memory cards + quantization	trade throughput for cost

Engineering note: private setups commonly separate "training GPUs for training, inference GPUs for serving"; mixing the two causes mutual contention — scheduling reference: 07 · Slurm Heterogeneous Scheduling.

3. vLLM Deployment: Three Core Mechanisms

vLLM is the de facto engine for private inference (mechanism deep-dive: 12 · vLLM Principles). Production deployment hinges on three mechanisms:

3.1 PagedAttention: On-Demand Memory Paging

Traditional inference pre-allocates the max KV Cache memory (60–80% wasted); vLLM allocates KV blocks on demand like OS paging — dramatically better memory utilization → the same card serves far more concurrency. This is the key to "same cards, double concurrency."

3.2 Continuous Batching

Each completed request immediately admits a new one, instead of waiting for the whole batch — throughput (token/s) improves by ~2× directly. Deployment notes:

• Tune max batch size (max_num_seqs) against memory and latency requirements;
• Large models (70B+) have volatile memory under continuous batching; leave headroom with gpu_memory_utilization.

3.3 Quantization: Trade Precision for Concurrency

Option	Memory	Quality impact	Use when
FP16/BF16	2B/param	lossless	quality first
INT8 (W8A8)	1B/param	slight	balanced
INT4 (AWQ/GPTQ)	0.5B/param	acceptable	concurrency first

Experience: a 7B INT4 model serves dozens of concurrent requests on a 24G card; for quality-sensitive scenarios, benchmark pre/post-quantization with an eval set first (reuse the evaluation approach from KB Q&A Production Stack).

4. Production HA Architecture

Single-instance demo → production service, add four layers:

                     ┌── GPU Node 1 (vLLM replica A)
users ──> LB/gateway ─┼── GPU Node 2 (vLLM replica B)
   (auth/rate-limit/routing) ├── GPU Node 3 (vLLM replica C)
                     └── (rolling scale out/in)

1. Multi-replica + load balancing: at least 2 replicas, gateway distributes by weight; replicas are stateless (KV Cache lives on-card; session stickiness optional);
2. Canary release: new model/engine first gets 10% traffic; promote only after metrics pass — treat model upgrades like software releases;
3. Failover: card failure → health check eviction → replica self-heal; critical services need GPU nodes across physical machines;
4. Queue & rate limiting: when peak exceeds capacity, queue instead of failing, return expected wait time.

5. Observability and Capacity Planning

5.1 Core Metrics

Layer	Metric	Alert threshold reference
Service	QPS, time-to-first-token (TTFT), per-token latency (TPOT)	TTFT < 1–2 s
Resource	GPU utilization, memory usage, KV Cache water level	memory > 90% warns
Business	queue length, error rate, timeout rate	error rate > 1% alerts

5.2 Capacity Planning Formula

needed concurrency ≈ peak QPS × average response seconds
per-card concurrency capacity ≈ f(memory, model, quantization, context length)  ← measure by load test
cards ≈ ceil(needed concurrency / per-card capacity)

Engineering note: capacity must come from load tests, not estimation — hammer a single card with a realistic mixed-length prompt workload, derive the "concurrency-latency" curve, then apply a redundancy factor (usually 1.5–2×).

6. Pitfalls and Tuning (Field Experience)

1. "It runs" ≠ "it serves": single-request inference success is only the start; once concurrency rises, memory fragmentation, KV Cache churn and queueing latency all surface. Load-test before launch — the most common private-deployment failure point.

2. Context length is a hidden cost bomb: the larger max_model_len, the higher KV Cache usage and the lower concurrency. Set it to real business needs (most scenarios: 4K–8K), don't max it out by default.

3. Lock model and engine versions together: vLLM upgrades can change operator behavior / quantization compatibility; pin the combo and regression-test before upgrading — don't chase the latest.

4. Test together with the RAG chain: private services almost always connect to a knowledge base (see KB Q&A Production Stack); end-to-end latency = retrieval + rerank + generation; include the retrieval chain in capacity planning.

5. Monitoring must be replayable: record per-request (model, quantization, concurrency window, latency distribution); when something breaks, you can answer "what was saturated then."

7. Quantified Results and Retrospective

Metric	Reference baseline (7B model, single 24G card, vLLM)
Per-card concurrency	dozens (hundreds with continuous batching + INT4)
Throughput	thousands of token/s (measure on your own)
TTFT	hundreds of ms (small batches)
Availability target	99.9% (multi-replica + health checks)

Note: figures are engineering-order-of-magnitude estimates; actuals follow your own load tests — the load-test data is itself a deliverable.

Takeaway: private LLM serving = memory computed + vLLM mechanisms understood + four HA layers + load-test-driven capacity. Selection and architecture settle in the first two days; the remaining 80% of effort goes to load testing, tuning and monitoring.

Engineering Takeaways

• Private serving is the common base for all "AI-compute-side" applications: KB Q&A, writing assistants and coding assistants all run on top of it — make the service rock-solid first, then talk applications;
• Relationship to Chapter 12: Chapter 12 explains engine principles and performance comparison; this article covers production architecture — mechanics explain "why it's fast," this article explains "how to keep it stable";
• Next reading: Scientific Agent Deployment — on top of a stable inference service, upgrade single-turn Q&A into multi-step scientific workflows.