Post-Training Engineering Ledger: SFT, LoRA, and RL Memory & Cost

Pretraining produces a model that "can talk"; post-training (SFT, RLHF/DPO) makes it "obedient" and aligned with human preferences. Post-training's engineering ledger differs from pretraining: less data, faster iteration, but still large memory demands. This article settles the SFT/LoRA and three-RL-route memory and cost ledgers.

1. Post-Training Pipeline

Pretrain → SFT (supervised fine-tuning) → RL (PPO/DPO/GRPO) → Eval → Serve
   │                 │                          │              │
  base model      learn format/style         learn alignment   quality gate

Each stage uses the same infrastructure (training framework, memory budget, parallelism), but memory and cost structures differ.

2. SFT: Full Fine-Tune vs LoRA

	Full fine-tune	LoRA
Trained params	all	only low-rank adapters (<1%)
Memory	full (weights+grads+optimizer)	weights frozen, saves grads/optimizer
Extra storage	one full model per task	MB-level adapter per task
Fits	big data, max quality	multi-task, medium data, budget

Quantified: 70B full fine-tune needs ~300GB+ memory (four states, see ZeRO Ledger); LoRA needs about 1/3-1/2 (frozen weights save grads+optimizer).

Takeaway: LoRA isn't "second best" — it's the default in the multi-task era: one base model + many small adapters, zero-cost task switching.

3. Three RL Routes

Route	Needs	Memory/cost	Character
PPO	policy+ref+reward+critic (4 models)	highest (4×)	stable, classic
DPO	policy+ref (2 models), no reward model	medium (2×)	direct preference optimization
GRPO	policy+ref, no critic	medium-low	group sampling replaces critic, R1-family standard

Quantified: PPO's 4-model overhead makes post-training memory approach pretraining levels; DPO drops the reward model+critic, halving memory; GRPO uses relative advantages within a group, saving the critic further.

Takeaway: start with DPO/GRPO; go to PPO only when quality is insufficient — same alignment goal, half the cost.

4. Evaluation: Don't Let the Gate Be a Formality

Dimension	Tool/benchmark	Use
General capability	OpenCompass, MMLU	regression of overall ability
Alignment quality	human/preference labels	did SFT/RL pass
Deployment metrics	latency/throughput/cost	pre-launch acceptance
Regression monitoring	continuous post-launch sampling	prevent degradation

Takeaway: every post-training version must pass three gates — "capability doesn't regress + alignment passes + deployment metrics OK"; evaluation is part of the engineering loop (see llm-serving-sre).

5. Reproducible Python: Fine-Tune Memory Comparison

def finetune_mem(n_params_b, method, world, fp8=False):
    """Per-card memory for full vs LoRA (GB, ZeRO-3 basis, ignoring activations)"""
    w = n_params_b * 1e9 * (1 if fp8 else 2) / 1e9
    g = w; o = n_params_b * 1e9 * 4 / 1e9 * 2
    if method == "full":
        return round((w + g + o) / world, 1)
    # LoRA: weights frozen (reside, but no grad/optimizer copies)
    return round((w + 0.05 * g + 0.05 * o) / world + 0.5, 1)  # + adapters

finetune_mem(n_params_b=70, method="lora", world=32)
# ≈ 4.9 GB/card vs full ≈ 26.3 GB/card → LoRA saves 80%+

6. Checklist

1. Default to LoRA for multi-task/medium data; full fine-tune only for single-task max quality;
2. Start RL with DPO/GRPO; move to PPO only if quality is insufficient (the 2× cost must be justified);
3. Run three gates per version: capability benchmark (OpenCompass-class) + preference eval + deployment metrics;
4. Reuse the ZeRO Ledger function for memory budgets before training;
5. Record memory/duration/quality per experiment and build baselines (see Acceptance & Baselines).

---

Organized and written by this site for learning purposes; specific versions and measurements are subject to on-site conditions.