Scientific Data Pipelines: Engineering from Experimental Data to Training Sets

Note: This is a review article (no first-hand experimental data), based on public practice and data engineering.

Experiment Objective

The bottleneck of AI for Science is often not the model but the data: experimental data formats are wildly heterogeneous (vendor-private .csv/.h5/logs), units are inconsistent, missing values abound — and they must align with simulation output, be labeled, and versioned. Goal: a scientific data pipeline framework — ingestion and parsing → cleaning and alignment → featurization/labeling → versioned datasets — covering three sources (instrument data, simulation output, public datasets), with concrete directory conventions and quality gates.

Code Walkthrough

Part 1: Heterogeneous ingestion and a unified schema

# Goal: unify three sources into one parquet table (aligned by experiment ID)
import pandas as pd
from pathlib import Path

def ingest_experiment(src: Path) -> pd.DataFrame:
    # Dispatch parsers by suffix: instrument csv / vendor h5 / logs
    if src.suffix == '.csv':
        df = pd.read_csv(src, comment='#')          # skip comment headers
    elif src.suffix == '.h5':
        df = pd.read_hdf(src)                        # vendor HDF5 export
    else:
        df = parse_log(src)                          # custom log parser
    # Unify column names and units (critical!): raw columns → standard schema
    df = df.rename(columns={'Temp [C]': 'temperature_c', '时间': 't_s'})
    return df.assign(source=src.stem)                # keep provenance

Key points:

1. Dispatch by suffix is step one: the most common problem in science is not "no data" but "data in N formats" — route parsers by file type, then unify column names.
2. Unit and column-name standardization is the core action: Temp [C] → temperature_c, 时间 → t_s — without unified units, downstream modeling is full of hidden bugs (Fahrenheit/Celsius, seconds/milliseconds).
3. The source column preserves provenance: every sample knows its original file — the basis of traceback, dedup, and permission auditing.

Part 2: Cleaning and quality gates

def clean_and_gate(df: pd.DataFrame) -> pd.DataFrame:
    n0 = len(df)
    df = df.dropna(subset=['t_s', 'temperature_c'])       # drop if key cols missing
    df = df[(df['temperature_c'] > -50) & (df['temperature_c'] < 200)]  # physical range
    df = df.drop_duplicates(subset=['t_s', 'source'])     # dedup by time+source
    n1 = len(df)
    print(f"[gate] {n0} -> {n1} rows ({n0-n1} dropped)")
    assert (n1 / n0) > 0.8, "cleaning rate too high; check source quality"     # gate
    return df

Key points:

• Gates are not optional: missing values, physical violations, and duplicates each have rules; the assert gate alarms when the cleaning rate is abnormally high (>20% dropped), preventing dirty data from being silently treated as normal.
• Physical-range filtering is scientific-data-specific: temperature -50~200°C, pressure non-negative — domain-knowledge physical constraints beat statistical outlier detection.
• Dedup must carry the source: two experiments may share a value at the same time; dedup on ['t_s','source'] avoids deleting valid rows.

Principles

Alignment strategies for the three sources

Source	Traits	Alignment issue	Handling
Instrument/experiment	private formats, irregular sampling	timestamp alignment	resample to a unified grid
Simulation output	grid/step set by solver	space inconsistent with experiment	interpolate/map to experimental coords
Public datasets	tidy but different conventions	unit, definition differences	remap to unified schema

Core method: convert all sources to "a unified space-time grid on the standard schema," then align by experiment ID. Do not preserve each format's native structure — unified grid + source column is the general reversible form.

Versioning training sets and simulation inputs

Scientific datasets need versioning like code: a dataset/ tree organized by "version + change note," with checksums:

datasets/
  v2026-10-baseline/    # baseline: cleaning rules v1, 100k samples
  v2026-10-pharma/      # pharma extension: +20k proprietary data
  MANIFEST.json         # version manifest: sources, cleaning rules, counts, checksums

Key points:

• Bind versions to code versions: training scripts record the dataset version (dataset_v=v2026-10-pharma); reproducibility requires "code version + data version both locked."
• Version the cleaning rules too: one line changed in cleaning logic changes the entire dataset's semantics — version the cleaning scripts with the dataset, more rigorous than recording files alone.
• The manifest is the traceability carrier: each version declares "source files, cleaning rules, sample counts, checksums" — audit and reproduction both rely on it.

Connecting to the AI4S chain

The data pipeline is AI4S's foundation: generative models (the drug-discovery article) need "clean + diverse" training data; surrogate models (the HPC-surrogate article) need "simulation-experiment aligned" paired data; reproducible workflows (ch14) require lockable data versions. If the pipeline is bad, every downstream model stands on sand — data-pipeline engineering has the highest ROI in AI4S.

Results

(Estimated, not measured)

Pipeline stage	Time share	Common failure
Ingestion/parsing	30%	heterogeneous formats, per-parser adaptation
Cleaning/alignment	40%	unit inconsistency, timestamp misalignment
Featurization/labeling	20%	labeling drift, inconsistent manual labels
Versioning	10%	forgetting to record versions, unversioned cleaning rules

Engineering Takeaways

1. Unify the schema before discussing quality: column/unit standardization comes first; without it everything downstream is hidden bugs.
2. Quality gates via assert, not promises: automate the three gate tables (cleaning rate, missing rate, physical range); alarm on threshold — "data quality is measured, not felt."
3. Source column and manifest as double insurance: every row has a source, every version has a manifest; data incidents (bad data entering training) trace back to root.
4. Data pipeline before model selection: spending 40% of an AI4S project's effort on the data pipeline is reasonable — models can be swapped; dirty data cannot be fixed by swapping models.

Next article preview: with the data foundation in place, the AI4S engineering chain (JAX → pharma → data) forms a small closed loop in this chapter — next steps could deepen chapter 14's AI×Science, or pivot to site SEO wrap-up. Your call.