Post-Training Alignment: The Compute and Engineering of SFT, RLHF/DPO

Note: This is a review article (no first-hand experimental data), based on public alignment methods and training practice.

Experiment Objective

Pretraining builds a "model that predicts"; alignment turns it into an "assistant that obeys": SFT teaches format, RLHF teaches preferences, DPO is a simpler preference alternative. Goal: dissect the full engineering of post-training alignment — SFT/RLHF/DPO principles and pipelines → three-stage compute ratios → preference data and stability engineering → compute linkage with pretraining. After reading you can answer: how much compute alignment needs, RLHF or DPO, how to build data, and how to keep stability.

Code Walkthrough

Part 1: The core pipelines of the three alignment methods

# Alignment trio (illustrative flows):
# 1) SFT (supervised finetuning): standard finetune on human-written instruction-answer pairs
#    data: (instruction, answer) pairs, loss = standard cross-entropy
#    effect: teaches "format/style/obedience" (model knows how to answer)
# 2) RLHF (RL from human feedback): 4 steps
#    a) SFT model → b) train a reward model (human ranking labels)
#    c) RL training (PPO: policy + reward + reference models)
#    d) aligned model output
# 3) DPO (direct preference optimization): skip reward training
#    directly optimize policy on preference pairs (chosen, rejected) — no reward model
def align_stage(model, method, data):
    if method == "sft":   return sft_train(model, data.qa_pairs)
    if method == "rlhf":  return ppo_train(model, reward_model, ref_model)
    if method == "dpo":   return dpo_train(model, data.preference_pairs)

Key points:

1. SFT is the foundation: RLHF/DPO both start from the SFT model — SFT teaches "how to answer," preference methods teach "which answer is better"; RLHF without SFT performs poorly.
2. RLHF is heavy: reward training + PPO four models (policy/reward/reference/value), high memory and tuning cost; DPO is light: preference pairs optimize directly, no reward model.
3. Data shapes differ: SFT uses instruction-answer pairs, preference methods use (chosen, rejected) pairs — data shape decides the method.

Part 2: Compute ratios (estimated)

# Alignment compute share (relative to pretraining, typical empirical):
#   Pretraining 1× (baseline)
#   SFT    ~0.5-2%   (small data, few epochs)
#   RLHF/DPO ~1-5% (multi-round sampling + training)
# Data-volume heuristics:
#   SFT: tens of thousands to hundreds of thousands of instruction pairs
#   preference: hundreds of thousands of (chosen, rejected) pairs

Key points:

• Alignment compute is far smaller than pretraining: SFT + preference usually under 5% of pretraining (estimated) — the compute bottleneck stays in pretraining; alignment is "small compute for large value" (linking ch10's compute-cost formula).
• RLHF costs more than DPO in sampling: PPO samples the policy and scores with the reward model each round, low sample efficiency; DPO trains directly on one preference pair — tight budget → DPO.
• Data quality > data quantity: preference-pair consistency (inter-annotator agreement) decides alignment quality — audit consistency before scaling.

Principles

Method comparison

Dimension	SFT	RLHF (PPO)	DPO
goal	teach format/obedience	maximize human preference	optimize preference directly
components	1 model	4 models (policy/reward/ref/value)	2 models (policy/ref)
data	instruction-answer pairs	ranking labels	preference pairs
compute	low	high	medium
stability	stable	hard (reward hacking/collapse)	fairly stable

Core insight: the evolution is "getting simpler" — SFT teaches format, RLHF teaches preference but is engineering-heavy, DPO optimizes directly on preference pairs without the reward pipeline. Selection by budget and stability: tight budget/small team → DPO; pursuit of extreme alignment → RLHF (willing to pay tuning cost).

Stability engineering points

1. Reward hacking: the policy exploits reward-model loopholes (answering off-topic but scoring high) — needs KL constraints + reward-model iteration.
2. Preference data pollution: inconsistent/mistaken preference pairs teach the model badly — clean data + consistency audit first.
3. Forgetting and degradation: alignment can degrade general ability — mix in pretraining data, control alignment-data share.

Results

(Estimated, not measured)

Option	Compute (vs pretraining)	Alignment quality	Engineering complexity
SFT only	~1%	medium (format good, preference weak)	low
SFT + DPO	~2-4%	high	medium
SFT + RLHF	~3-6%	highest	high (hard to tune)

Engineering Takeaways

1. SFT is the foundation: SFT teaches format/obedience first, then preference methods — preference without SFT performs poorly.
2. Tight budget → DPO: DPO skips reward training and PPO sampling; compute/tuning cost far below RLHF; RLHF only for extreme alignment.
3. Data quality first: preference consistency audit before scaling — polluted data teaches the model badly; half of alignment quality is in the data.
4. Prevent forgetting and hacking: mix pretraining data against degradation, KL constraints against reward hacking — stability engineering equals accuracy in importance.

Next article preview: alignment fills the gap; the training side (ch10/11) is now thick; next could pivot to SEO wrap-up (submit the 432-url sitemap to GSC) or deepen ch04/08/13. Your call.