Verify

A benchmarked hallucination detector. Give it a SOURCE (the source of truth) and a DRAFT; it returns the specific claims the source doesn't support, each with {claim, reason, source_fact_checked, category, severity}.

Benchmarked: 106 labeled cases across 4 domains + an adversarial red-team tranche. Over 3 independent runs: precision 97.9% [96.6, 99.1], recall 99.1%, F1 98.5% (95% CIs) — with every residual failure adjudicated and documented. See BENCHMARK.md.

from mcp_verify import build_default_client, verify

client = build_default_client()  # reads ANTHROPIC_API_KEY
report = verify(client, source="...source of truth...", draft="...text to check...")
for f in report.failures:
    print(f.severity, "|", f.claim, "—", f.reason)

# Reliability mode: run N times, keep only majority-confirmed claims;
# unstable ones land in report.uncertain instead of report.failures.
report = verify(client, source="...", draft="...", consistency=3)

Run as MCP server

One tool, verify(source, draft, consistency=1), over stdio:

pip install mcp-verify   # or: pip install -e . from a checkout
claude mcp add verify -- mcp-verify

Two modes, auto-selected (MCP_VERIFY_MODE=api|sampling overrides):

• sampling (default, no key): the server asks the HOST's model to do the verification via MCP sampling — no ANTHROPIC_API_KEY, no extra cost.
• api: if ANTHROPIC_API_KEY is set in the server's environment, it calls the pinned benchmark model directly — the exact path BENCHMARK.md measures. claude mcp add verify -e ANTHROPIC_API_KEY=sk-... -- mcp-verify

Both return the same JSON: {"passed", "failures": [...], "uncertain": [...], "mode"}.

The benchmark

The eval suite (106 labeled cases across 4 domains plus a red-team tranche, a 15-type hallucination taxonomy, and a confusion matrix) travels with the detector. See BENCHMARK.md.

# Offline checks (no API key, no cost):
bash check.sh                      # lint + full test suite
python -m eval.analysis --sample   # failure analysis on a sample report

# Live eval (needs ANTHROPIC_API_KEY):
python -m eval.run                 # full suite
python -m eval.run --smoke         # 5 representative cases
python -m eval.run --judge         # LLM-judge matcher (meaning, not tokens)
python -m eval.ablation            # model/thinking/token-budget ablation