Experiment Audit

Most experiment-tracking tools show you numbers. experiment-audit checks whether your claim about those numbers actually holds up — missing evidence, out-of-scope comparisons, contradictions with earlier results, and confidence that isn't just assumed.

Install

pip install experiment-audit

Requires Python 3.11+. See Quick start below, or jump straight to Claude Code integration.

Quick start (30 seconds)

from experiment_audit.reasoning import (
    ScientificReasoningPipeline, ScientificReport,
    Claim, ClaimCategory, Scope,
)

claim = Claim(
    id="c1",
    subject="model-x",
    statement="model-x achieves 95% accuracy on CIFAR-10",
    category=ClaimCategory.PERFORMANCE,
    scope=Scope(dataset="cifar-10"),
)

pipeline = ScientificReasoningPipeline()
context = pipeline.build_initial_context(claims=[claim], evidence=[])
report = ScientificReport.from_pipeline_report(pipeline.execute(context))

print(report.to_markdown())

Claim → Evidence → Reasoning → Scientific Report. Every finding in the output traces back to specific evidence — the engine doesn't assert anything it can't point to.

Prefer the command line?

experiment-audit reasoning schema > claims.json   # see the expected input shape
experiment-audit reasoning run --input claims.json --format markdown

Claude Code Compatible

The reasoning discipline behind this project — how to phrase findings, weigh contradictory evidence, and write structured reviewer-style feedback — ships as a Claude Code skill, with the eight MCP audit tools available automatically wherever WANDB_API_KEY is set.

_{◆ Claude Code — run from inside this repo}

/plugin marketplace add ./dev/experiment-audit-plugin
/plugin install experiment-audit@experiment-audit

It triggers automatically on prompts like:

• "Is this ablation confounded?"
• "Why did my loss crash?"
• "Review this paper's results claim."
• "Compare these ablation studies."
• "Write reviewer feedback on this."

See Quick start: the MCP server below for manual MCP setup, or the plugin's own README for full details.

Why experiment-audit?

Traditional experiment trackers are good at one thing: displaying metrics. They will happily tell you a run's final accuracy, loss curve, or config diff. What none of them do is check whether the sentence you're about to write about those numbers is actually supported by them.

experiment-audit treats a result the way a careful reviewer would before publication:

• Is there evidence behind this claim at all, or is it an assumption that snuck in?
• Does the evidence actually match what's being claimed — same dataset, same protocol, same scope — or is it being stretched to cover more than it proves?
• Does anything else you've measured contradict it?
• Is the confidence in the write-up proportional to the evidence, or borrowed from how confident the result felt?

This matters most for reproducibility, ablations, and the kind of paper-writing claims that are easy to overstate under deadline pressure — the exact places research claim verification tends to break down silently.

Who is this for

ML engineers	sanity-check a result before it ships in a report or a PR description
AI researchers	catch confounded ablations and out-of-scope comparisons before submission
Graduate students	get reviewer-style feedback on a results section before your advisor does
Research labs	a shared, deterministic check for scientific claims across a team's experiments
Academic / open-source projects	structured, evidence-traced scientific reports instead of ad hoc write-ups

What the reasoning engine does

Given a set of claims ("model-x achieves 95% accuracy on CIFAR-10") and the evidence backing them (metrics, configs, logs, prior runs), the engine runs six rules in sequence and produces a ScientificReport:

#	Rule	What it checks
1	Missing evidence	Does this claim have any supporting evidence trace at all?
2	Scope	Does the evidence actually match the claim's stated scope (same dataset, same hardware, same evaluation protocol)?
3	Contradiction	Does any other claim or evidence item conflict with this one?
4	Confidence	A computed score, not a guess — based on evidence quality, quantity, contradictions found, and what's missing.
5	Judgment	A verdict (supported / partially supported / unsupported) with the reasoning behind it.
6	Recommendation	What to do about it — gather more evidence, narrow the claim's scope, retract it.

Every finding traces back to specific evidence. Nothing in the report is an unsupported assertion — that would rather defeat the point.

This is one of two reasoning pipelines in the package. The second, lower-level pipeline (ScientificReasoningEngine — Evidence → Observations → Hypotheses → Confidence → Judgment → Recommendation) is a more generic, extensible framework for injecting custom hypothesis and confidence logic. Most people should start with the six-rule pipeline above. See src/experiment_audit/reasoning/__init__.py for both.

Features

Reasoning engine (the core)

• Claim and evidence modeling (Claim, EvidenceItem, Scope) with structured categories
• Six-rule scientific reasoning pipeline, run end-to-end or rule-by-rule
• Contradiction detection across claims and evidence
• Confidence scoring driven by evidence quality/quantity, not a fixed heuristic
• Structured ScientificReport — Markdown, JSON, or plain text
• Zero network calls; runs entirely on data you provide

Interfaces around the engine

• Python API — experiment_audit.reasoning, for embedding the pipeline in your own tooling
• CLI — experiment-audit reasoning run|schema
• Claude Code skill — the reasoning discipline as an installable skill, with worked examples
• MCP server — eight tools for auditing Weights & Biases runs directly from an agent
• Weights & Biases backend — read-only run/sweep/metric access behind the MCP tools

Quick start: the MCP server (W&B audit tools)

The original W&B experiment-audit tools are still here, unchanged, as an MCP integration. Set a read-only W&B API key:

export WANDB_API_KEY="your-read-only-key"
export WANDB_ENTITY="your-team-or-username"   # optional

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "experiment-audit": {
      "command": "experiment-audit-mcp",
      "env": {
        "WANDB_API_KEY": "your-read-only-key"
      }
    }
  }
}

Claude Code:

_{◆ Claude Code}

claude mcp add -e WANDB_API_KEY=your-read-only-key experiment-audit -- experiment-audit-mcp

-e must come before the server name, not after — putting it after the name has been a source of "Invalid environment variable format" errors in some Claude Code versions.

Then ask your agent something like:

"Did I mess up my memory-ablation run? Compare mamfac-baseline and mamfac-no-memory in the mamfac project and check whether the only real difference is use_memory."

The agent calls audit_ablation, which returns a verdict (clean / confounded / uncertain), a confidence level, and the full config diff it based that verdict on.

Full tool reference (all eight tools, exact schemas, methodology) is in docs/design-spec-v1.md and docs/audit-methods.md — unchanged from the v1.0.0 release.

Architecture

experiment_audit/
├── reasoning/                 # the Scientific Reasoning Engine
│   ├── claims.py               # Claim, ClaimSet, Scope
│   ├── evidence.py              # Evidence, EvidenceItem (shared by both pipelines)
│   ├── contradictions.py        # Contradiction, ContradictionSet
│   ├── scientific_rules/        # the six concrete rules
│   │   ├── missing_evidence_rule.py
│   │   ├── scope_rule.py
│   │   ├── contradiction_rule.py
│   │   ├── confidence_rule.py
│   │   ├── judgment_rule.py
│   │   └── recommendation_rule.py
│   ├── rules.py                 # RuleContext, ScientificRule base
│   ├── pipeline.py              # ScientificReasoningPipeline: runs the six rules in order
│   ├── scientific_report.py     # ScientificReport: to_markdown/to_json/to_text
│   ├── observations.py          # generic pipeline: pattern detection over Evidence
│   ├── hypotheses.py             # generic pipeline: candidate explanations
│   ├── confidence.py             # generic pipeline: confidence scoring
│   ├── judgment.py                # generic pipeline: verdict rendering
│   ├── recommendation.py          # generic pipeline: recommendations
│   └── engine.py                  # ScientificReasoningEngine: the generic pipeline's orchestrator
├── cli.py                     # `experiment-audit reasoning run|schema`
├── models.py                  # RunRef, Run, MetricPoint, MetricHistory, Sweep, Page[T]
├── errors.py                  # ToolError + the frozen error_type taxonomy
├── server.py                  # FastMCP entrypoint; registers the 8 W&B audit tools
├── backends/
│   ├── base.py                # ExperimentBackend ABC, BackendCapability
│   ├── fake_backend.py        # in-memory test double
│   └── wandb_backend.py       # real W&B implementation
└── analysis/                  # the W&B audit tools' pure heuristics
    ├── comparison.py
    ├── divergence.py
    ├── confound.py
    └── sensitivity.py

The reasoning engine and the MCP/W&B layer are independent — the reasoning engine takes Claims and EvidenceItems directly and has no dependency on W&B, FastMCP, or any backend. Feeding W&B run data into the reasoning engine as claims/evidence (rather than hand-constructing them, as the quick-start example above does) is on the roadmap.

For the reasoning engine's design rationale, see research/07_reasoning_engine/ (reasoning-engine.md, reasoning-rules.md, confidence-system.md, evidence-model.md, scientific-reviewer.md). For the MCP/W&B layer's frozen contract, see docs/design-spec-v1.md.

Data handling

• Data never leaves your machine except calls to your own W&B endpoint (MCP layer only — the reasoning engine itself makes no network calls at all).
• Credentials are read once from environment variables, validated fail-fast on server start, and never logged.
• Use a read-only W&B API key — this server has no write path.

Known gaps (honest status)

• No built-in adapter converts a W&B run directly into Claims/EvidenceItems yet — you construct them yourself (CLI schema or Python), or write your own extraction step. This is the top roadmap item.
• The generic pipeline (ScientificReasoningEngine) defaults its rule-engine stage to a no-op unless you inject one — it's an extensibility point, not a second complete pipeline.
• 274 tests pass (pytest tests/ -q); this is real coverage of the pipeline's mechanics, not a substitute for domain review of the six rules' thresholds by someone in your research area.
• The MCP/W&B layer is W&B-only for now (MLflow is prototyped at the interface level, not implemented), and audit_sweep's correlation ranking only detects linear relationships.

Full detail, including what's blocked purely by this build environment's lack of live credentials, is in docs/design-spec-v1.md and the CHANGELOG.

Development

git clone https://github.com/SreeDharshan-GJ/experiment-audit.git
cd experiment-audit
pip install -e ".[dev]"
pytest tests/ -q        # 274 tests
ruff check src/ tests/  # lint

The reasoning engine's tests need no network access or credentials at all — they run entirely on in-memory Claim/Evidence fixtures. The MCP/W&B layer's tests run against FakeBackend, an in-memory test double that can inject every adversarial state named in the design spec.

Contributing

Contributions are welcome — please read CONTRIBUTING.md first.

The MCP/W&B layer's v1 design (docs/design-spec-v1.md) is frozen: changes to its tool schemas, model fields, or backend interface need an explicit, logged design decision, not a silent PR. The reasoning engine's six rules and their thresholds are newer and more open to discussion — if you're proposing a change to rule logic (as opposed to wiring), explain the reasoning-quality tradeoff you're making, not just the code change.

Roadmap

• Near-term — a W&B-run-to-claims/evidence adapter, so the MCP audit tools can hand their findings directly to the reasoning engine instead of requiring hand-built Claim/EvidenceItem objects.
• v2 — MLflow backend for the MCP layer, a versioned API compatibility matrix, first public case study from a real project.
• v3 — RL-specific pathology signals, proper multi-seed statistical tests, Optuna/Ray Tune sweep support, open to external audit_* and reasoning-rule contributions.

Citing this project

If experiment-audit was useful in your research or workflow, a citation or a link back is genuinely appreciated:

@software{experiment_audit,
  author  = {Sree Dharshan G J},
  title   = {experiment-audit: A Scientific Research Reasoning Engine for ML Experiments},
  year    = {2026},
  url     = {https://github.com/SreeDharshan-GJ/experiment-audit}
}

Author

Built and maintained by Sree Dharshan G J.

GitHub

If this project is useful to you, a star on the repo is the easiest way to support it and helps others find it.

License

MIT — see LICENSE.