LLM Inference and Agents: From One Request to the Agent Loop

The same model can be either a chatbot or an agent that calls tools and recovers from surprises — the difference is not the weights but the system around them: how serving turns one request into tokens, how prompting changes behaviour without training, and how tool calls let the model decide what it sees next. These are reading notes on the Yale Introduction to Machine Learning lecture "LLM Inference and Agents", linked to this site's inference chapter.

1. Why Agents: Letting the Model Decide What It Sees

1.1 Chatbot vs. Agent: the Same Weights, Arranged Two Ways

	Chatbot (instruct model)	Agent
Data flow	prompt → model → answer; the user writes the next prompt	prompt → model → tool call → observation → back to the model
Who decides what to see next	the user	the model (calls a tool and reads the result)
Handling surprises	no	yes: the model decides what data it sees next

Takeaway: the weights are identical; only the arrangement differs. In agent mode the model decides what data it will see next, which is exactly what lets it recover from a surprise.

1.2 A Concrete Shape: OpenClaw

The lecture uses OpenClaw as an example: WhatsApp / Telegram / Discord send instructions to OpenClaw on the user's machine (a model + a loop + installed skills), which can run terminal code, read and search files, and send emails and messages — all with the user's own permissions.

OpenClaw connects an LLM backend to the external world. Architecture as described in Bitdefender, Technical Advisory: OpenClaw Exploitation in Enterprise Networks, 2026.

1.3 Why Enterprises All Build Agents: the Data Flywheel

Anthropic's revenue curve after Claude Code is the most direct illustration (the company's own disclosures):

Milestone	Date
Research preview	Feb 2025
General availability	May 2025
$1B run-rate revenue	Nov 2025 (announced 2025-12-03)
Over $2.5B run-rate revenue	Feb 2026 (announced 2026-02-12)

The mechanism underneath is a data flywheel: deploy the agent on the company's own tasks → it does the work and every run leaves a trace → grade each trace by whether the task worked → train on the runs that worked → deploy again. Each turn makes the next agent better at the company's own tasks.

1.4 What an Agent Is Made Of

Following Lilian Weng, LLM Powered Autonomous Agents (2023), an agent is roughly a served model + tools + actions + chain of thought (this lecture's focus), surrounded by memory and planning (reflection, subgoals, self-critique).

Key point: an agent is not a new model — it is "the same model + tools + a loop + a prompting system". The weights are fixed throughout; nothing in this lecture involves training.

2. Model Serving: What One Request Contains

2.1 What Serving Looks Like

Many users, prompts of very different lengths, and one copy of the weights held in memory across several GPUs. The server collects whatever has arrived into a batch, and the whole batch moves through the weights together — throughput comes from sharing a forward pass.

2.2 One Request = the Conversation So Far

A request carries the whole conversation, tagged by role, plus decoding settings:

{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the weather in New Haven?"},
    {"role": "assistant", "tool_calls": [
      {"name": "get_weather", "arguments": {"city": "New Haven"}}
    ]},
    {"role": "tool", "name": "get_weather", "content": "41F, clear"},
    {"role": "user", "content": "Should I bring a jacket?"}
  ],
  "temperature": 0.2, "max_tokens": 300
}

• system carries the developer's instructions, user the authenticated person's request, assistant earlier model output, and tool what a program returned — a tool result is a message like any other;
• all four arrive as tokens in one string, so nothing in the format enforces the system message;
• the server holds no state between calls: "the model remembers" is a property of the client, which resends the history.

2.3 The Three Steps on the Server

1. Apply the chat template: flatten the message list into one token string (string handling);
2. Prefill and decode: prefill the string, then decode until the stop token (the only step that runs the network);
3. Parse: recover the generated tokens into one message (string handling).

Engineering note: step 2 is where the cost is; steps 1 and 3 are what make the model usable as a chatbot.

2.4 The Chat Template: Roles Are a Convention, Not a Network Feature

The template flattens the roles and marks turns with special tokens (e.g. <|im_start|> / <|im_end|>):

<|im_start|>system
You are a course assistant.<|im_end|>
<|im_start|>user
How many meetings are left?<|im_end|>
<|im_start|>assistant
Nine.<|im_end|>
<|im_start|>user
What is meeting 25 about?<|im_end|>
<|im_start|>assistant

• the roles are a convention of the template, not a feature of the network; the server flattens, and the model only ever sees the flattened string;
• post-training taught the model to stop at the end marker, which is how the server knows a reply has finished;
• decoding produces <|im_start|>assistant 41°F and clear.<|im_end|>; keeping what sits between the markers recovers {"role": "assistant", "content": "41°F and clear."} — the format alone carries the message.

3. Prefill and Decode: the Same Weights, Two Cost Models

• Prefill: run the whole prompt through the network once, all token positions in parallel. The causal mask lets every position be computed at the same time, so a long prompt is one wide matrix multiply; a next-token distribution is computed at every position and discarded, and only the distribution at the last position starts decoding;
• Decode: produce tokens one at a time; each needs a full forward pass and cannot start until the previous token is chosen.

Cost baseline: a 2000-token prompt costs one pass; a 500-token answer costs 500. Output length is what users wait for.

Engineering note: the two phases share weights but stress hardware very differently (prefill is compute-bound wide matmul; decode is memory-bound per-token passes), so large deployments run them on separate machines — search "PD disaggregation".

4. The Three Big Ideas: KV Cache, Paged Attention, Continuous Batching

Decode is the expensive half, and three system-level ideas carry almost all of the practical gain — they change what serving costs and leave the output distribution untouched (source: Kwon et al. 2023, vLLM, arXiv:2309.06180).

4.1 The KV Cache: Store What You Already Computed

Attention at position $t$ compares the query $q_t$ against the key of every earlier position and averages those positions' values:

$$\text{Attn}(q_t) = \sum_{s \le t} \alpha_{ts}\, v_s, \qquad \alpha_{ts} \propto \exp\{q_t^\top k_s / \sqrt{d}\}$$

The pairs $(k_s, v_s)$ do not depend on $t$: computed once, read back at every later step — that is the KV cache.

• Memory: $2 \times (\text{layers}) \times (\text{width}) \times (\text{bytes}) = 128\ \text{KiB per token}$;
• For Llama-3 8B: ~1 GiB per 8k-token conversation, against 16 GiB of weights — the cache is what limits how many users fit at once;
• What it saves: without a cache, every decode step recomputes the keys and values of the whole context; storing them makes each step add only one new pair, and the cost of a step stops growing with the conversation.

4.2 Paged Attention: Virtual-Memory-Style Allocation

The problem with static allocation: each request reserves its maximum length (say 2048 tokens) as one contiguous run, while a typical reply runs a few hundred tokens — most of the run is reserved and never written, and being contiguous, the free part cannot be handed to anyone else.

PagedAttention moves to non-contiguous allocation: each request still sees one contiguous cache, but a block table maps its blocks onto fixed-size slots anywhere in a shared pool — the same indirection virtual memory uses. Slots are claimed only as they are filled; nothing reserves space for a length it may never reach.

4.3 Continuous Batching: Let Finished Requests Free Their Slots

Decode reads every weight in the model to produce one token, so a single request wastes almost all of that read — prompts of different lengths are prefilled, then the batch advances in lockstep: one forward pass per step, one new token for every request in it.

Static batching fixes the membership at the start, so slots sit idle once requests stop at different steps; continuous batching lets a finished request leave and a waiting one take its slot.

4.4 The Three Together

Technique	Problem it solves	Effect
KV cache	decode recomputes the whole context every step	step cost stops growing with context
Paged attention	static reservations waste memory; free parts unusable	memory usage tracks actual usage
Continuous batching	fixed batch membership leaves slots idle	throughput stays near a full batch

5. Sampling: Why We Do Not Want the Tail

A model's next-token distribution is heavy-tailed: a few plausible continuations, and tens of thousands of tokens each carrying a little mass (about 30% of the mass in the illustrative example). Faithful sampling draws from that tail every so often, and the model then conditions on what it drew — everything after continues from a sentence it would never have written.

Hence truncation at serving time:

• top-k: keep the k most probable tokens, then renormalize (k = 5 → 5 tokens kept);
• top-p: keep the smallest set whose cumulative probability reaches p, then renormalize (p = 0.90 → 25 tokens kept);
• temperature: rescale the logits first — a fixed count (top-k) and a fixed mass (top-p) need not agree, so they must be calibrated together.

Takeaway: for chat/generation default to top-p (p≈0.9) with temperature (T≈0.7–1.0); use greedy for deterministic tasks. Sampling itself adds almost no compute — what costs is output length and "sample k times, take the best" (i.e. inference compute ×k). See this site's Output Sampling Ledger.

6. Prompting: Changing Behaviour without Training

The weights are fixed and the request is one token string, so everything an application controls is what is in that string. This lecture sits inside the chain-of-thought box: the text a model writes before it answers.

The lecture's own line: "Prompting is an art." The same weights answer well or badly depending on how the request is phrased; practitioners collect phrases that happen to work — "take a deep breath", "let's solve this step by step", "you are an expert" — and swap them in and out.

Engineering note: prompts are part of the product — version them, accumulate winning phrases, A/B them. No weights change, but online answer quality does (see LLM Serving SRE).

7. Engineering Takeaways and a Delivery Checklist

1. Decide whether to build an agent: ask whether the task needs the model to read external results inside a loop, and whether you accept the model deciding what it sees next. The value is recovering from surprises; the cost is multiple serving calls and longer end-to-end latency per loop;
2. Stateless serving is a rule: the client resends history and tool results; a tool result is just another message, so the format supports it natively;
3. Throughput comes from shared forward passes: the fuller the batch, the cheaper each request; budget the KV cache (on the order of 128 KiB/token) before promising concurrency limits (see KV Cache Compression Ledger);
4. Memory optimization order: KV cache (mandatory) → a serving framework with PagedAttention / continuous batching (e.g. vLLM) → sampling calibration;
5. Watch prefill/decode hardware mismatch: consider PD disaggregation for high concurrency or very long prompts; TTFT and throughput are dominated by prefill and decode respectively (Inference Benchmarking);
6. Productize prompting: keep a versioned prompt library and A/B it in production, alongside model serving, under SRE governance.

8. Reading Links and Sources

• Related articles: vLLM Principles · KV Cache Compression Ledger · Output Sampling Ledger · Long-Context Inference · vLLM Production Deployment · LLM Serving SRE · Scientific Agent Deployment
• This article is based on: the Yale Introduction to Machine Learning lecture "LLM Inference and Agents" (reposted by the WeChat account "论智谈芯" on 2026-11-01); the agent architecture figure follows Lilian Weng, LLM Powered Autonomous Agents (2023, lilianweng.github.io); PagedAttention and continuous batching are from Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv:2309.06180; Anthropic revenue figures are from its official announcements of 3 December 2025 and 12 February 2026. These are study notes with no first-hand experimental data; formulas and figures are restated from the above sources, and the current versions and measurements at your site are authoritative.

---

These notes were prepared by this site from the public Yale lecture materials for learning purposes; formulas and examples are restatements of the lecture content. Actual versions and measurements may vary on your site.