13. Inference Engines

Chapter focus:

🧭 Reading Map

• Prerequisites: 12. Distributed Training — inference plans tie directly to the trained model's characteristics
• Role: closing chapter of Part 4, L4 — vLLM/TensorRT-LLM principles and production, long context and speculative decoding
• Order: engine principles (vLLM → comparisons) → production deployment → frontier (long context/speculative)
• Next: 14. Industry Apps — turn inference capability into business

Articles

Long-Context Inference: From RoPE Extrapolation to Efficient KV Cache Serving

Reproducible configs and scripts dissecting long-context inference: RoPE extrapolation (PI/NTK/YaRN), the KV cache memory math, chunked prefill and efficient serving strategies, with engineering takeaways.

阅读文章 →

TensorRT-LLM in Practice: From Model Export to FP8 Inference Acceleration

Reproducible commands walking the TensorRT-LLM production chain: HF-to-TRT-LLM conversion, FP8 quantization, engine build flags, inference serving, and benchmark comparison, with engineering takeaways.

阅读文章 →

Transformers in Distributed Deployment: Performance Comparison with vLLM

In 2017, Google introduced the Transformer architecture in "Attention Is All You Need", replacing recurrent networks with multi-head self-attention me

阅读文章 →

Deep Dive into vLLM: PagedAttention & High-Performance Inference

A deep analysis of vLLM's core technology PagedAttention, revealing memory management optimization mechanisms, with a complete hands-on guide.

阅读文章 →

vLLM Production Deployment: From Launch Flags to High-Concurrency Service Governance

Reproducible commands dissecting vLLM production deployment: launch flag semantics, throughput/latency tuning, KV cache and quantization, OpenAI-compatible serving, Prometheus monitoring and multi-replica load balancing, with engineering takeaways.

阅读文章 →

LLM Inference Benchmarking: Throughput, TTFT & Cost in Practice

metric semantics, concurrency modeling, external load, cost.

阅读文章 →

Speculative Decoding: Principles, Engineering & Benefit Boundaries

small-model draft + large-model verification, consistency checks, acceptance-rate judgment.

阅读文章 →

Output Sampling Ledger: Greedy, Beam, top-k/p

Four strategy tiers, temperature & top-p calibration, where sampling sits in cost, speculative-decoding consistency.

阅读文章 →

KV Cache Compression: GQA, Quantization & Paging

GQA/MQA sharing, KV INT8/INT4, PagedAttention & prefix sharing — 95% memory saving combined.

阅读文章 →

MoE 推理优化账：路由、负载均衡与专家并行

稀疏激活/稠密驻留的显存账、路由失衡代价、All-to-All 通信账与 prefetch/卸载优化。

Read →

Multimodal Inference Ledger: Visual Tokens & Cost

Visual tokenization (one image = 1K tokens), encoder+projection, KV/compute amplification, engineering checklist.

阅读文章 →