Science Computing Embraces AI: Surrogate Models in HPC Workflows

Scenario: A CFD solve takes 2 hours per case, and design optimization needs to sweep 2,000 parameter combinations — the pure-HPC path would take 4,000 hours. This is the most typical interface of "AI computing × HPC reinforcing each other": HPC generates high-quality data, AI turns expensive solves into cheap predictions.

1. Scenario and Pain Points

A traditional scientific-computing workflow (simulation-driven design) looks like this:

parameter space ──> submit simulation jobs one by one (Slurm queue + solve) ──> collect results ──> manual analysis ──> next round of parameters

Three fatal bottlenecks:

Bottleneck	Symptom	Cost
Expensive single solve	CFD/structural/multiphysics: hours to days per case	parameter sweeps basically infeasible
Queue amplifies cost	shared cluster + large jobs queue, wall-clock doubles	iteration cycles measured in weeks
Curse of dimensionality	each extra parameter dimension explodes the sweep	design space only partially explored

The surrogate model solution: use an AI model to learn "input parameters → simulation results", compressing each evaluation from 2 hours to milliseconds, turning a thousand-case sweep into a single model inference pass:

HPC sampled simulation (generate training data) ──> train surrogate ──> high-speed surrogate evaluation (thousand-case sweep in seconds) ──> verify key points back on HPC

2. Overall Architecture

A production-grade "AI-accelerated simulation" workflow has four subsystems:

┌─────────────────────────────────────────────────────────────┐
│  ① Data pipeline (HPC side)                                     │
│  parameter sampling → Slurm batch simulation → result extraction │
│  → data versioning → featurization                              │
├─────────────────────────────────────────────────────────────┤
│  ② Model training (AI-compute side)                             │
│  train/val split → model selection → error monitoring →          │
│  uncertainty estimation                                         │
├─────────────────────────────────────────────────────────────┤
│  ③ Inference service                                            │
│  package model as service/library → embed in optimization loop   │
│  → re-verify key points on HPC                                   │
├─────────────────────────────────────────────────────────────┤
│  ④ Closed-loop iteration                                        │
│  surrogate error exceeds threshold → active sampling →           │
│  incremental training → update deployment                        │
└─────────────────────────────────────────────────────────────┘

Key design principle: the surrogate is not a "solver replacement" but a high-speed filter in the optimization loop — coarsely scan the parameter space, and only send a handful of promising candidates back to the expensive solver for precise verification.

3. Key Implementation

3.1 Sampling Strategy: More HPC Data Is Not Always Better

The quality ceiling of a surrogate is set by sampling. Three mainstream strategies:

Strategy	Idea	When to use
Latin Hypercube (LHS)	stratified uniform sampling per dimension	unknown parameter space, baseline
Space-filling (Sobol/Hammersley)	low-discrepancy sequences, more uniform coverage	low-to-mid dimensionality (≤20)
Active Learning	train a draft, then use uncertainty to pick "points the model is least sure about"	expensive simulation, constrained budget

Engineering note: active learning typically cuts sampling volume by 50–70% for the same accuracy, but requires the "train → infer → pick points → resimulate" loop to be automated — which is exactly what Slurm batch scripts plus scheduler APIs can chain together.

3.2 Model Selection: From Regression to Neural Operators

Model	Capability boundary	Data needs	Typical use
Gaussian Process (GP)	small data, built-in uncertainty	hundreds	low-dim design optimization
Random Forest / XGBoost	mid data, tabular features	thousands	scalar metric prediction (lifetime, efficiency)
Deep Neural Network (DNN)	large data, strong nonlinearity	tens of thousands	complex response surfaces
Neural Operators (FNO/DeepONet)	function-to-function, spatial structure	tens of thousands+	field prediction (flow field, temperature field)

A common mistake: conflating "predict a scalar" (e.g., total drag coefficient) with "predict an entire field" (e.g., flow distribution). Scalar regression can use tabular models; field prediction needs neural operators or spatial-structured networks — an order of magnitude more engineering complexity. Theory details: 14 · AI-CFD.

3.3 Error Control: A Surrogate Must "Know What It Doesn't Know"

Scientific computing cannot tolerate hallucination. Productionization needs three lines of defense:

1. Held-out validation set: 20% of sampled data never enters training; report RMSE / relative error;
2. Uncertainty estimation: GP gives confidence intervals; DNNs approximate with ensembles or MC-Dropout;
3. Re-verification: key surrogate-driven decisions (optimization optimum, dangerous operating points) must be re-solved on HPC; error above threshold triggers resampling.

4. Pitfalls and Tuning (Field Experience)

1. Extrapolation failure is the sneakiest trap: a surrogate is accurate inside the training distribution; once the optimizer pushes parameters outside the sampled range, predictions can be completely off. Mitigation: constrain the optimization range to the sampling range, and make re-verification a hard process step.

2. Simulation noise gets "memorized": rerunning the same parameters gives slightly different results due to numerical noise. Don't chase 100% training fit — treat noise as aleatoric uncertainty and control overfitting with early stopping.

3. Version data, not folders: 2,000 simulations = 2,000 result files; three iteration rounds and it's chaos. Use metadata-aware versioning (parameters, solver version, mesh scale) such as DVC — otherwise "model got worse after swapping data" becomes untraceable.

4. Stagger HPC and AI-compute resources: sampling simulation eats CPU/GPU compute nodes; training eats GPU. Off-peak scheduling (simulation on idle night nodes, training on daytime GPU pools) significantly cuts cost — see 06 · Scheduling & Cloud-Native and 07 · Slurm Heterogeneous Scheduling.

5. Quantified Results and Retrospective

A standard retrospective template (hypothetical CFD design-optimization scenario):

Metric	Pure HPC	HPC + Surrogate
Single evaluation	~2 hours	~10 ms
Sweep of 2,000 parameter sets	~4,000 hours (longer with queueing)	minutes of inference + dozens of re-verifications
Total simulations for equal design accuracy	2,000	200–400 (active sampling + re-verification)
Iteration cycle	weeks	2–3 days

Note: figures above are engineering-order-of-magnitude estimates for building intuition; actual gains depend on dimensionality, per-solve cost and required accuracy. Validate with your own minimal closed loop first, then review with the same table.

Takeaway: the surrogate moves "sweep-optimize" workflows from week-scale to day-scale, at the cost of adding a model training/maintenance stage — only worth it when single simulations are expensive enough and sweep volume large enough; otherwise brute-force enumeration is simpler.

Engineering Takeaways

• The most pragmatic entry point for "AI computing × HPC reinforcing each other": HPC produces data, AI accelerates, HPC re-verifies — three links chained together, the core idea of the AI-HPC platform.
• The surrogate is the technical base of the Software Factory "one problem, one software" methodology: every scientific problem can be packaged into a "sample → train → serve → re-verify" software loop.
• Next reading: the full end-to-end case in Battery Lifetime Prediction: From Scientific Problem to Software Delivery.