Claimcheck

MIT licensed · An MCP server that settles a claim about code instead of asserting it.

Built for the Alexa+ track of the Amazon Build, Ship, Shape hackathon as a self-hosted MCP server over Streamable HTTP.

The problem it addresses

An assistant answering questions about code can say "this is fixed" or "these tests cover it" without either being true. The failure is not a crash — it is a confident sentence nobody checked.

The specific trap: a test that passes proves nothing unless it can also fail. A test that passes both with and without the change it supposedly guards is decoration, and it is the most common way a "verified" claim turns out to be empty.

claimcheck gives a model a way to run that second experiment.

Tools

verify_guard(fixed_code, broken_code, test_code)

Runs the test twice — against the code with the fix and against the same code without it — and reports which of five things is true:

verdict	meaning
KILLED	passes with the fix, fails without it → a real guard
DECORATIVE	passes both ways → cannot detect the bug it claims to cover
BROKEN_TEST	fails even with the fix → the claim or the test is wrong
NO_ASSERTIONS	the test has no assert, so passing is meaningless
REJECTED	input refused (see reason)

check_claim_shape(claim)

Turns a vague claim into something measurable, or reports that this server cannot settle it. It deliberately recognises a closed set of shapes — a tool that claims to check anything checks nothing. "3x faster" returns checkable_here: false with the reason: a timed comparison needs a baseline and run-to-run variance measured before any difference is believed, and this server does not measure time.

evidence_digest(label, measurement)

Hashes a measurement so a later answer can be checked against it, because a summary of a result drifts from the result.

Verified end to end

Every number below was produced by running it, over real HTTP, not by local function calls.

protocol negotiated : 2025-11-25          # the spec version the track requires
server_info.name    : claimcheck
list_tools          : verify_guard, check_claim_shape, evidence_digest

verify_guard(assert clamp(7) == 7)   -> DECORATIVE   with_fix=True  without_fix=True
verify_guard(assert clamp(-5) == 0)  -> KILLED       with_fix=True  without_fix=False

The SDK's LATEST_PROTOCOL_VERSION is 2026-07-28; the negotiated version on the wire is 2025-11-25, which is what the track specifies.

All five verdict branches were exercised: KILLED, DECORATIVE, BROKEN_TEST, NO_ASSERTIONS, REJECTED.

Run it

Install it:

pip install .                        # or: pip install dist/claimcheck-0.1.0-py3-none-any.whl
claimcheck streamable-http           # serves on http://127.0.0.1:8000/mcp
claimcheck stdio                     # for a local MCP client

Or run it straight from the checkout, which takes the identical code path:

pip install mcp
python server.py streamable-http

Connect with any MCP client:

import asyncio, json
from mcp.client.streamable_http import streamable_http_client
from mcp.client.session import ClientSession

async def main():
    async with streamable_http_client("http://127.0.0.1:8000/mcp") as (r, w):
        async with ClientSession(r, w) as s:
            await s.initialize()
            res = await s.call_tool("verify_guard", {
                "fixed_code":  "def clamp(n):\n    if n < 0:\n        return 0\n    return n\n",
                "broken_code": "def clamp(n):\n    return n\n",
                "test_code":   "assert clamp(-5) == 0",
            })
            print(json.loads(res.content[0].text)["verdict"])   # KILLED

asyncio.run(main())

Safety boundary

This is not a sandbox and does not pretend to be one. It runs snippets you hand it, so it refuses the constructs that would let a "test" reach outside the snippet — at which point the pass/fail signal would no longer be about the code under test:

• static AST rejection of os, sys, subprocess, socket, shutil, pathlib, requests, http imports;
• rejection of eval, exec, compile, __import__, open calls;
• 20,000-character source limit, 4,000-character output cap;
• no shell execution, no network access.

Run it against code you are willing to execute.

Architecture

MCP client (Alexa+ / any MCP host)
        │  Streamable HTTP, protocol 2025-11-25
        ▼
┌─────────────────────────────────────────┐
│ MCPServer("claimcheck")                 │
│                                         │
│  verify_guard ──┐                       │
│  check_claim_shape ─┼─→ _reject_unsafe  │  static AST gate
│  evidence_digest ───┘        │          │
│                              ▼          │
│                          _run(source)   │  fresh namespace,
│                              │          │  stdout/stderr captured
│                              ▼          │
│                    RunResult{ok, error, │
│                      assertions_run}    │
└─────────────────────────────────────────┘
        │
        ▼
  two runs compared → verdict

The two-run comparison is the whole design. One run tells you a test passed; only the second tells you whether that meant anything.

Licence

MIT — see LICENSE.