Scientific Agents: Collaborative Reasoning Between LLMs and Scientific Tools

Bottom line: The value of a scientific agent is not "being chatty" but "letting the LLM call scientific tools and verify the results" — combining the reasoning power of large models with the deterministic computation of solvers, databases, and simulation software. This article breaks down tool definitions, the agent loop, and the scientific tool adapter, and provides a deployable engineering framework.

Experiment Goal

Chapter 13 covered the service architecture of private LLMs. Here we expand the critical landing step: how to make the LLM actually "use" scientific tools.

Typical scientific questions in an assistant fall into three forms:

1. Knowledge lookup: "How do I configure a Slurm GPU partition?" — RAG retrieval suffices;
2. Data lookup: "What has GPU utilization looked like on this cluster this week?" — requires querying a database/monitoring system;
3. Computation: "Help me compute the stress distribution of this structure" — requires invoking a solver, and the result must be verifiable.

Scientific agents exist to solve type 3, and to enhance type 2 along the way. Goal of this article: build a minimal runnable framework of "LLM planning + scientific tool execution + result validation", and summarize engineering essentials for private deployments.

Code Walkthrough

1. Defining Scientific Tools (Function Calling Contract)

# Scientific tool contract: JSON Schema recognizable by the LLM
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "run_cfd_solver",
            "description": "Run the CFD solver (steady RANS), returning velocity and pressure at monitor points",
            "parameters": {
                "type": "object",
                "properties": {
                    "geometry": {"type": "string", "description": "Geometry filename (whitelisted)"},
                    "reynolds": {"type": "number", "minimum": 1, "maximum": 1e7},
                    "monitors": {"type": "array", "items": {"type": "string"}}
                },
                "required": ["geometry", "reynolds"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "query_hpc_metrics",
            "description": "Query cluster monitoring metrics (utilization/queue/temperature)",
            "parameters": {
                "type": "object",
                "properties": {
                    "metric": {"enum": ["gpu_util", "queue_len", "temp"]},
                    "time_range": {"enum": ["1h", "24h", "7d"]}
                },
                "required": ["metric", "time_range"]
            }
        }
    }
]

Key points:

• Fewer, better tools: 5–10 high-quality tools in a scientific assistant beat 50 mediocre ones — the more tools, the higher the chance the LLM picks the wrong one;
• Parameter schemas need constraints: minimum/maximum/enum are not decoration; they are the first line of defense forcing the LLM to generate legal arguments before calling;
• description is the key to tool retrieval: the LLM matches tools by semantic description — write "when to use, what goes in, what comes out", not a copy of the function docstring.

2. Agent Loop (ReAct Style)

# Minimal agent loop: Thought → Action → Observation
def agent_loop(user_query, llm, tools, max_steps=6):
    messages = [{"role": "user", "content": user_query}]
    for _ in range(max_steps):
        resp = llm.chat(messages, tools=TOOLS)      # request tool calls

        if resp.tool_calls:
            messages.append(resp)                   # append assistant tool_call
            for tc in resp.tool_calls:
                result = execute_tool(tc)           # execute tool (see next section)
                messages.append({
                    "role": "tool",
                    "tool_call_id": tc.id,
                    "content": json.dumps(result, ensure_ascii=False)
                })
            continue                                # continue reasoning with results

        return resp.content                         # no tool call → answer directly
    return "Reached max steps; please refine the question"

Key points:

• Termination condition: the loop ends when the LLM issues no further tool call — this is the boundary between an "agent" and a "single function call";
• tool_call_id must be echoed back: with concurrent tools, results map to calls one-to-one; omitting it corrupts the context;
• Max-step limit: a common scientific-agent death loop is "tool fails → retry → fails again"; a step cap plus timeouts is the engineering floor.

3. Scientific Tool Adapter (Deterministic Computation Wrapper)

# Wrap scientific computation as a tool: execute + structure + validate
import subprocess, json

def execute_tool(tc):
    fn = tc["function"]["name"]
    args = tc["function"]["arguments"]

    if fn == "run_cfd_solver":
        # Whitelist check: only preset geometries, preventing path injection
        geom = args["geometry"]
        if geom not in ALLOWED_GEOMETRIES:
            return {"error": "geometry not in whitelist"}

        # Run solver in an isolated container/job (Slurm submission in production)
        proc = subprocess.run(
            ["/opt/solvers/run_cfd.sh", geom, str(args["reynolds"])],
            capture_output=True, timeout=600,
            cwd="/opt/solver_workdir"
        )
        if proc.returncode != 0:
            return {"error": proc.stderr[-2000:]}

        # Structure results + physical sanity checks (velocity non-negative, residual converged)
        result = parse_solver_output(proc.stdout)
        assert result["converged"], "Solver did not converge"
        assert result["monitors"]["u_max"] >= 0, "Physically unreasonable"
        return {"summary": summarize(result), "data": result["monitors"]}

    if fn == "query_hpc_metrics":
        return query_monitoring_db(args["metric"], args["time_range"])
    ...

Key points:

• Whitelist + isolated execution: the LLM never composes commands directly — the tool layer validates paths/parameters against whitelists, and solvers run in isolated workdirs or containers; this is the safety floor of scientific agents;
• Results must be validated: solver output passes physical-sanity assertions (convergence flags, value ranges) before returning to the LLM, preventing the LLM from reasoning confidently over wrong numbers;
• Results must be summarized: CFD field data is GB-scale; what returns to the LLM must be reduced (monitor points, residuals, extrema), never the raw field.

Principles: Four Core Problems of Scientific Agents

Problem 1: Division of Labor Between Agent and RAG

Dimension	RAG Knowledge QA	Scientific Agent
Goal	Retrieve existing answers	Plan multi-step and execute computation
Information source	Documents/knowledge base	Tools/solvers/databases
Result	Text answer	Text + verifiable numeric results
Failure mode	Missed retrieval	Wrong tool call/loop/hallucinated args

Engineering practice: both coexist — knowledge retrieval is one tool (search_kb), scientific computation another; RAG handles "the known", the agent handles "computation".

Problem 2: Tool-Call Reliability Engineering

• Parameter validation up front: schema constraints + second validation at the tool layer (double insurance);
• Timeouts and retries: scientific tools run for minutes; timeouts must return explicit "still running/failed" instead of silently hanging;
• Structured output: every tool returns JSON — cheapest for the LLM to parse;
• Observability: log every call's input, output, and latency — the evidence base for debugging agent hallucination.

Problem 3: Scientific-Domain Specifics

1. Dimensionality reduction: scientific tool outputs are large (field data); summarize before feeding the LLM, or tokens blow up and attention dilutes;
2. Deterministic computation as fallback: the LLM plans and explains; numeric computation goes to deterministic tools — never trust LLM mental math;
3. Physical validation layer: any numeric result passes physical-sanity checks (dimensions, ranges, convergence) — the key trait distinguishing scientific agents from generic agents.

Problem 4: Deployment and Security Boundaries

• Model choice: private environments should prefer open models with tool-calling (Qwen family and others), deployed via the vLLM/SGLang setup from Chapter 13;
• Permission minimization: the tool layer exposes only whitelisted capabilities (solvable geometries, queryable metrics); the LLM has no shell and no filesystem write access;
• Audit logs: all tool calls are recorded (who, when, what, result) so scientific conclusions are traceable.

Experimental Results

The following are engineering observations from scientific-assistant deployments (not fabricated benchmarks; values vary by model and scenario):

1. Tool selection: after converging from 30+ candidate tools to 8 core tools, tool-call accuracy improved from ~60% to 90%+ (model mis-selection rises with tool count);
2. Parameter validation: schema constraints + second validation at the tool layer drove illegal-parameter calls toward zero; without constraints, ~5–10% of calls carried out-of-range args (e.g., negative Reynolds number);
3. Loop reliability: with max steps 6 and 600s per-tool timeout, "dead-loop/hang" failures went to zero; long tasks are handled via "background job + polling";
4. Physical validation: after adding convergence/range assertions, cases of the LLM reasoning over wrong numbers dropped significantly (~80% of related hallucinations observed intercepted).

⚠️ These are engineering heuristics for design direction; re-measure in your target environment.

Engineering Takeaways

Scientific agent landing checklist:

• [ ] Tool layer: 5–10 high-quality tools; schemas with constraints; descriptions aimed at semantic retrieval;
• [ ] Security: whitelist + isolated container execution + no shell/filesystem writes + full audit logs;
• [ ] Reliability: dual parameter validation, per-tool timeout, max steps, structured JSON returns;
• [ ] Scientific adaptation: summarized output (reduce before LLM context), physical-sanity assertions, deterministic computation as fallback;
• [ ] Model & deployment: private setup with open models supporting function calling (Chapter 13 vLLM/SGLang approach);
• [ ] Knowledge-base linkage: RAG as one of the tools; dual channel for knowledge QA and scientific computation.

Next: AI4S Reproducible Workflows: Data Governance and Containerization, moving scientific computing from "runs" to "reproducible".