11. Pre-trained Models

Chapter focus: Pre-trained models — Transformer and MoE/MLA architectures, data pipelines and engineering.

Transformer Architecture

• Self-Attention Mechanism
• Multi-Head Attention
• Feed Forward Network (FFN)

Positional Encoding

• Absolute Positional Encoding (Sinusoidal)
• Rotary Positional Embedding (RoPE)

Training Objectives

• Masked Language Modeling (MLM)
• Causal Language Modeling (CLM)

🧭 Reading Map

• Prerequisites: 10. Heterogeneous Computing — heterogeneous base ready, pre-training builds on it
• Role: opening of Part 4, L4 — pre-training architectures (MoE/MLA) and data/compute/engineering pipelines
• Order: data pipeline & compute planning (do the math first) → model architecture (DeepSeek MoE/MLA) → training engineering
• Next: 12. Distributed Training — with a model in hand, train efficiently on thousands of GPUs

Articles

DeepSeek Architecture: MLA & Extreme MoE Optimization

A deep dive into DeepSeek-V3/R1 architecture innovations, explaining how MLA compresses KV Cache and how Fine-Grained MoE achieves load balancing.

Read article →

LLM Pretraining Data Pipeline: From Raw Corpus to High-Quality Training Set

Runnable engineering scripts dissecting the pretraining data pipeline: corpus cleaning, quality filtering, deduplication, mixing ratios, and tokenizer training, with a pipeline diagram and engineering takeaways.

Read article →

Pretraining Engineering: Checkpoints, Fault Tolerance, and Training Stability

Reproducible configs dissecting pretraining engineering: checkpoint strategy (period/async/sharding), node-fault recovery, loss-anomaly triage, and training-stability guardrails, with engineering takeaways.

Read article →

RTP Hybrid Architecture: DeepSeek & Next-Gen Paradigm

In-depth analysis of the Retention-Transformer Pipeline (RTP) hybrid architecture, revealing its performance advantages in training and inference.

Read article →

MoE Training Engineering: Expert Parallelism & Load Balancing in Practice

EP placement, load-balancing loss, token dropping, All-to-All overhead.

Read article →

Pretraining Compute & Cost Planning: GPU Estimation & Card-Hour Budget

card-hour estimation, MFU corrections, allocation, budget flow.

Read article →

Scaling Law Ledger: Loss-Compute-Data

Kaplan vs Chinchilla allocation, the 6ND compute floor, inference-time scaling, with reproducible Python.

Read article →