LLM Pretraining Data Pipeline: From Raw Corpus to High-Quality Training Set

Summary: 70% of pretraining quality comes from data. This post walks every stage of the data pipeline with runnable scripts and quantitative metrics: cleaning, quality filtering, deduplication, mixing ratios, tokenizer training. Architecture (MLA/MoE) sets the model's ceiling; data decides whether it reaches it.

0. Experiment Goals

• Turn "raw corpus" into a "training-ready token stream" with a runnable Python flow
• Master the five key stages: cleaning → quality filtering → deduplication → mixing → tokenizer
• Understand each stage's quantitative metrics (dedup rate, quality score, mix ratio, tokenizer efficiency)
• Extract several iron rules for pretraining data engineering

1. Stage 1: Cleaning (Removing Impurities)

import re

def clean_text(text: str) -> str:
    """Basic cleaning: control chars, extra whitespace, noise punctuation."""
    # 1) Remove Unicode control characters (zero-width, BOM, private-use)
    text = "".join(ch for ch in text if not (
        ord(ch) < 32 and ch not in "\n\t" or 0xE000 <= ord(ch) <= 0xF8FF
    ))
    # 2) Collapse consecutive whitespace (keep newlines)
    text = re.sub(r"[ \t]+", " ", text)
    # 3) Collapse repeated punctuation (marketing noise: "!!!!!!", "????")
    text = re.sub(r"([!?。！？]){3,}", r"\1\1", text)
    # 4) Optional: mask URLs and emails (keep URLs if training code corpora)
    # text = re.sub(r"https?://\S+", "<URL>", text)
    return text.strip()

How it works:

• Cleaning targets signal-to-noise ratio, not "cosmetics": control characters make tokenizers emit anomalous tokens; repeated punctuation marks webpage noise.
• Order matters: strip invisible characters first, then whitespace, then punctuation—reversing it leaves control characters hidden inside whitespace.
• Cleaning must be auditable: every rule maps to a counter log ("removed N zero-width chars") so mis-killed corpus types can be located quickly after launch.

2. Stage 2: Quality Filtering (Classifier Scoring)

# Lightweight heuristic pre-filter (no model needed), score 0-1
def quality_heuristic(text: str) -> float:
    n = len(text)
    if n < 200:
        return 0.0                      # too short: fragments/nonsense
    # Punctuation density: too low means not natural language
    punct = sum(1 for ch in text if ch in "，。！？；：,.!?;:")
    punct_ratio = punct / n
    if punct_ratio < 0.02:
        return 0.1
    # Duplicate-line ratio: high repetition (ads, copy-paste comments) is low quality
    lines = [l.strip() for l in text.split("\n") if l.strip()]
    dup_ratio = 1 - len(set(lines)) / max(len(lines), 1)
    if dup_ratio > 0.5:
        return 0.15
    return min(1.0, 0.3 + punct_ratio * 20)   # empirical weighting

# Production pipeline: rule pre-filter (cheap) → model refine (expensive, candidates only)
# Recommended: fastText / BERT classifier scores candidates; keep top-k

Results (example):

Corpus type	Rule pre-filter pass	After model refine	Note
Wikipedia	95%	92%	high-quality baseline
News crawl	78%	71%	ads/reposts noise
Forums	55%	41%	needs aggressive filtering
Code repos	88%	85%	separate code pipeline

Key: two-stage filtering. Rules block 99% of junk first (fast, cheap); model scoring runs only on survivors (expensive, accurate). Running a full-model classifier on everything is a classic compute waste.

3. Stage 3: Deduplication (Exact + Approximate)

import hashlib
from collections import defaultdict

def minhash_dedup(texts, n_grams=5, n_hashes=64):
    """MinHash + LSH approximate dedup: the standard for millions of docs."""
    buckets = defaultdict(list)
    for idx, text in enumerate(texts):
        # Slice doc into 5-grams; hash each; doc signature = min-hash set
        grams = [text[i:i+n_grams] for i in range(0, max(1, len(text)-n_grams+1), n_grams)]
        sig = tuple(min(hashlib.md5(g.encode()).hexdigest() for g in grams)
                    for _ in range(n_hashes))
        buckets[sig].append(idx)
    return [indices for indices in buckets.values() if len(indices) > 1]

# Exact dedup (fallback for small data)
seen = set()
unique = []
for doc in texts:
    h = hashlib.md5(doc.encode()).hexdigest()
    if h not in seen:
        seen.add(h)
        unique.append(doc)

How it works:

• Exact dedup (full-text MD5) only removes byte-identical copies; "pseudo-different" docs from webpage templating need approximate dedup.
• MinHash + LSH is the industry standard: split docs into shingles (n-grams), use a set of min-hashes to approximate similarity, cluster pairs above a threshold, then verify exactly.
• Cheaper still: suffix-array dedup (Google-style) for cross-document substring dedup at extreme scale, which also prevents test-set leakage into the training set—this directly affects benchmark credibility.

4. Stage 4: Mixing Ratios (The Data Recipe)

# Example recipe (by token share):
DATA_MIX = {
    "wikipedia":       0.35,   # high-quality general knowledge
    "books":           0.15,   # long-form coherence
    "code":            0.15,   # code and reasoning
    "web":             0.25,   # open-world knowledge
    "math_science":    0.10,   # math and scientific reasoning
}

How it works:

• The mix ratio is the most underestimated "hyperparameter": too much general web → "broad but shallow"; too much code/math → missing common sense. Industry experience: code 10-20%, math 5-15% are common ranges; settle exact values with small-scale ablations.
• Sample by token, not by document—source documents differ wildly in length; document-level sampling over-samples long-document types unintentionally.
• Late training often applies mixing-ratio annealing: gradually raise high-quality sources (books/math/code)—the "sprint phase" of training.

5. Stage 5: Tokenizer Training & Checks

# Train a BPE/Unigram tokenizer (Unigram + char fallback recommended for Chinese)
from tokenizers import Tokenizer, models, trainers, pre_tokenizers

tok = Tokenizer(models.Unigram())
tok.pre_tokenizer = pre_tokenizers.Metaspace()

trainer = trainers.UnigramTrainer(
    vocab_size=65536,
    special_tokens=["<unk>", "<s>", "</s>", "<pad>"],
    unk_token="<unk>",
    show_progress=True,
)
tok.train(files=["corpus_zh.txt", "corpus_code.txt"], trainer=trainer)
tok.save("tokenizer.json")

# Three mandatory checks:
# 1) Chinese single-char fallback rate: <2% normal; higher = corpus too small or vocab too small
# 2) Avg tokens per char: better compression = better language modeling
# 3) Code corpus: inspect token shapes for comments/indentation—anomalies mean pre-tokenizer issues

Engineering notes:

• The tokenizer sets the "vocabulary budget": 32K-64K vocab is common for Chinese; too small → poor Chinese compression (expensive long sequences); too large → wasted embedding parameters.
• The pre-tokenizer hugely impacts code corpora: token shapes for indentation, comments, and string literals materially affect code-completion quality.
• Train the tokenizer on the same corpus family: directly reusing Llama's tokenizer for a Chinese model yields naturally worse Chinese token efficiency—the reason "Chinese vocab share" is a recurring topic for Chinese models.

6. End-to-End Orchestration (One Command)

# Pipeline: raw/ → clean/ → filter/ → dedup/ → mix/ → tokens/
python3 pipeline.py \
  --input raw/ \
  --output tokens/ \
  --clean --filter --dedup --mix \
  --mix-config data_mix.json \
  --tokenizer tokenizer.json \
  --shards 256            # parallel shards; horizontally scalable via Spark/Ray

Engineering notes:

• Sharding + idempotency: every stage writes immutable shards (shard-0001.parquet); a rerun only reprocesses unfinished shards. Pretraining data is TB-scale; full reruns are unacceptable.
• Data versioning: each dataset carries a version and a hash of processing parameters; training logs record "which data version was used" for reproducibility—the basics of reproducible training.
• Closed-loop quality monitoring: every stage emits a stats report (token count, quality-score distribution, dedup rate). When training loss misbehaves, check the data report first.

7. Engineering Takeaways

1. Cleaning is auditable: every rule has a counter log; mis-kills are locatable.
2. Two-stage filtering: rules block 99% of junk; the model only scores candidates.
3. Approximate dedup is mandatory: MD5 catches only exact copies; MinHash+LSH catches templated pseudo-duplicates; mind test-set leakage.
4. Mix by token: sample at token granularity; anneal ratios in the late stage.
5. Tokenizer trained on same-family corpus: check single-char fallback and compression for Chinese.
6. Shard-idempotent + data versioning: TB-scale data resumes from checkpoints; training is reproducible.
7. Quality reports go into training logs: on anomalous loss, check data first, then the model.

8. Further Reading

• HuggingFace datasets processing docs
• RedPajama data recipe research
• KB: 10. Pre-trained Models · DeepSeek-MoE/MLA Architecture