Multimodal Inference Ledger: Visual Tokens, Sequence & Cost

Multimodal inference (vision, video, speech) isn't just "an extra encoder" — tokenizing visual inputs explodes sequence length and KV cost. This article settles the tokenization ledger, KV ledger, and engineering optimizations for multimodal inference, linking to KV Cache Compression Ledger.

1. Tokenization Ledger: How Many Tokens Is One Image?

Input	Typical tokens	Note
1K chars of text	~1K tokens	baseline
one 224² image	256 tokens (14² patches)	one token per patch
one hi-res image	1K+ tokens (slicing/hi-res encoding)	higher res = pricier
1-minute video (8fps)	~12K+ tokens	frames × patches

Takeaway: visual tokenization is the cost amplifier — a hi-res image ≈ a thousand tokens, one minute of video ≈ ten thousand; KV and prefill compute scale together (see Inference Resource Accounting).

2. Architecture Ledger: Encoder + Projection + LLM

image/video ── visual encoder (ViT/custom) ── projection (MLP/Q-Former) ── LLM
speech ────── audio encoder ──────────────── projection ──────────────── LLM

• Visual encoder: pixels → patch features (ViT family);
• Projection: map visual features into the LLM's token-embedding space (MLP or Q-Former compression);
• LLM: shared weights with text, performs cross-modal reasoning.

Takeaway: the projection is the first "compression" layer — Q-Former-style schemes compress visual tokens with a few learned queries, an architecture-level lever on sequence length.

3. Cost Ledger: Three Amplifiers

Cost item	Amplifier	Optimization
Prefill compute	many visual tokens, attention grows quadratically	visual token downsampling/merging
KV memory	visual tokens stay resident	quantize/cache visual KV separately
Visual recompute	re-encoding same image per request	image-token cache reuse

Takeaway: multimodal optimization differs from text — compress visual tokens first, then KV, finally image-level caching (same image with different questions encodes once).

4. Engineering Checklist

1. Input level: lower resolution/frame rate per task — only as clear as needed;
2. Architecture level: Q-Former-style projections to shrink visual token counts;
3. Inference level: manage visual KV separately from text KV; quantize or expire visual segments;
4. Service level: cache image/video encoding results by content hash; reuse for identical visual input;
5. Evaluation level: baseline "single-image QA" and "video QA" token & latency (see Acceptance & Baselines).

5. Reproducible Python: Multimodal Cost Estimate

def mm_cost(n_img, tokens_per_img, n_text_tokens, kv_bytes_per_token):
    """Multimodal request KV memory (GB): visual tokens dominate"""
    vis = n_img * tokens_per_img
    tot = vis + n_text_tokens
    return round(tot * kv_bytes_per_token / 1e9, 3), vis

mm_cost(n_img=5, tokens_per_img=512, n_text_tokens=500, kv_bytes_per_token=0.128)
# total KV 0.358 GB, visual 0.328 GB (92%) → visual tokens dominate

6. Checklist

1. Quantify the "single-image/video QA" token ledger before launch;
2. Grade visual inputs by task; don't default to max resolution;
3. Compress visual tokens via projection + inference-time downsampling/merging;
4. Quantize visual KV separately; cache encoding results by content;
5. Monitor KV share of multimodal requests; squeeze the visual side when over threshold (see llm-serving-sre).

---

Organized and written by this site for learning purposes; formulas and examples are the site's own; specific versions and measurements are subject to on-site conditions.