ReceiptGuard

An autopilot agent that can't lie about what its tools did.

Global AI Hackathon Series with Qwen Cloud — Track 4: Autopilot Agent

A qwen3.7-max agent resolves real support tickets end-to-end (refunds, warranty replacements). Watch what happens without a guard: it confidently reports "the Standing Desk has 8 units in stock, so I shipped a free replacement and emailed the tracking number" — when it never checked inventory, never shipped anything, and the order is 40 days past the warranty window. The tool was never called; the agent fabricated the result. This is the #1 blocker to trusting autonomous agents in production, and every recall/RAG approach misses it: they check whether the agent retrieved the right fact, not what it claims its tools did.

ReceiptGuard makes fabrication structurally impossible. Every tool call routes through a gateway that returns an unforgeable HMAC-signed receipt; every claim in the agent's answer is typed by epistemic source and cross-checked against those receipts. A claim no real tool produced is caught and the agent is forced to re-ground before the action commits — and when policy forbids auto-acting (e.g. outside the refund window) the agent escalates to a human instead. Every decision lands in a tamper-evident, hash-chained audit log.

What one fabrication costs

A fabricated "refund issued / replacement shipped" is a direct cash loss plus a compliance event. A first-order exposure model — every input is a knob you can turn:

monthly exposure ≈ monthly tickets × share touching money × fabrication rate × avg refund

Illustrative: 50,000 tickets/mo × 10% refund/replacement × 1% fabricated × $79 avg refund (our demo order) ≈ $3,950/mo in phantom payouts — before re-contact cost, chargebacks, or the compliance event. And the legal floor is already set: in Moffatt v. Air Canada, 2024 BCCRT 149 a tribunal held the airline liable for one fabricated support answer — its chatbot invented a refund policy — awarding the customer ~CAD $812 and ruling that a company owns what its agent says. (Illustrative model; the only hard number here is the case.)

Why this isn't "just a guardrail"

A guardrail scores how confident the model sounds and filters on a threshold. ReceiptGuard demands cryptographic proof that each tool actually ran: claims are typed by epistemic source and checked against an HMAC-signed receipt the agent cannot forge — and unbacked claims aren't suppressed, they trigger a budgeted agent loop that re-runs the real tools and re-grounds the answer before it ships. Proof-of-execution, not vibe-checking.

The money shot — inject a fabrication (red-team toggle) and watch the baseline ship 3 lies while ReceiptGuard blocks each unbacked claim and re-grounds against signed receipts, with qwen3.7-max reasoning shown live:

Proven: 100% fabrication-detection at 0% false-positives, sub-millisecond — while a real qwen-flash judge with no receipts catches 0% of the same fabrications (and is ~1000× slower). An ablation shows the receipt value-check is load-bearing; claim-typing fails safe (100% tool_derived recall). Runs live on Qwen3.7-Max + Qwen-Flash via Alibaba Model Studio. (Full table below.)

vs the field: ZeroClaw / Fetch.ai AEVS prove a tool ran but never check the agent's prose against the receipt (ZeroClaw's own docs: receipts "don't constrain text output"). ReceiptGuard does the part they skip — types every claim, cross-checks each against its receipt, recovers or escalates, then resolves the ticket.

---

The mechanism

ticket ─▶ Autopilot agent (qwen3.7-max) ──tools──▶ ToolGateway ──▶ HMAC-signed receipts
                  │                                                      │
                  ▼ draft answer                                         │
        Claim extractor (qwen-flash)  ── atomic claims, typed ──┐        │
                                                                ▼        ▼
                                              Cross-check claims vs receipts
                                                                │
                                        ┌───────────────────────┤
                                        ▼ all backed             ▼ unbacked / contradicted
                                     PROCEED            Tiered recovery (GSAR): regenerate / replan
                                                          (qwen3.7-max thinking adjudicates,
                                                           reasoning_content = audit reason)
                                        └──────────┬────────────┘
                                                   ▼
                              SHA-256 hash-chained append-only audit ledger

1. Receipts — every tool call routes through ToolGateway, which emits HMAC(secret, {tool, args_hash, output_hash, snippet, ts}). The agent never sees the secret, so it cannot forge a receipt for a tool it never called.
2. Typed claims — qwen-flash splits the answer into atomic claims, each tagged tool_derived | inference | absence | opinion (NabaOS pramana taxonomy, arXiv 2603.10060).
3. Cross-check — each tool-derived claim is matched to the receipt ledger: missing receipt → unbacked; value/count mismatch → contradicted; "nothing found" with a non-empty receipt → false absence.
4. Tiered recovery — groundedness score → proceed / regenerate / replan under an explicit compute budget (GSAR, arXiv 2604.23366). qwen3.7-max thinking-mode adjudicates contested claims; its reasoning_content is stored as the human-readable justification.
5. Audit — every decision is appended to a SHA-256 hash-chained ledger; tampering with any row breaks the chain (EU AI Act Art. 12 record-keeping).

Prior art & what's actually new here

The HMAC tool-receipt is not our invention and we don't claim it — it's a 2026 pattern shipped by ZeroClaw and Fetch.ai AEVS, and formalized in NabaOS and GSAR. Each stops at a different layer and stops there:

• ZeroClaw / AEVS prove what a tool executed — a signed receipt you can look up — but never check the agent's prose against it. ZeroClaw's docs: receipts "don't constrain text output"; AEVS verifies "what an agent executed, not why" and is "purely a post-execution recording system."
• MemTrust (arXiv 2601.07004) secures the memory, not the truthfulness of the answer.
• NabaOS / GSAR do claim-typing and graduated recovery — as a framework and a benchmark, not a running autopilot that takes an action.

ReceiptGuard's contribution is the closed loop, deployed: unforgeable receipt (floor) → qwen-flash epistemic claim-typing cross-checked against those receipts → GSAR budgeted regenerate/replan on unbacked claims → a policy gate that auto-resolves the refund/warranty or escalates to a human → SHA-256 hash-chained ledger → exposed as an MCP server → running on Qwen + Alibaba Function Compute, with an on-the-record ablation showing the verifier fails safe. We win on the composition and the deployment, not the primitive — and we say so out loud.

"Isn't this just a guardrail?" A guardrail scores how confident the model sounds and filters on a threshold — vibe-checking. ReceiptGuard demands cryptographic proof each tool actually ran: claims are checked against a receipt the agent cannot forge, and unbacked claims aren't suppressed — they trigger a real tool re-run before the action commits. Proof-of-execution, not prompt-based filtering.

---

Quickstart

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# runs fully OFFLINE in deterministic MOCK mode (no API key needed)
PYTHONPATH=src python -m receiptguard.cli run                          # baseline vs guarded (refund)
PYTHONPATH=src python -m receiptguard.cli run warranty_replacement     # 2nd scenario: escalates to human
PYTHONPATH=src python -m receiptguard.cli run refund_damaged --adversarial  # inject a lie -> watch it get caught
PYTHONPATH=src python -m receiptguard.cli serve                        # demo UI at http://localhost:8000
PYTHONPATH=src python -m receiptguard.cli bench 200                     # detection benchmark + ablation + chart
PYTHONPATH=src python tests/test_receiptguard.py                       # test suite (20 tests)

Going live on Qwen / Alibaba Cloud

Copy .env.example → .env, set DASHSCOPE_API_KEY (Alibaba Model Studio). With a key, the agent (qwen3.7-max, thinking mode), claim typer (qwen-flash), and the LLM-judge baseline call real Qwen via the OpenAI-compatible DashScope endpoint. No key → mock mode, everything still runs.

• Alibaba-usage proof file: src/receiptguard/llm/qwen.py
• MCP server (Function Compute, SSE): python -m receiptguard.mcp.server (needs pip install mcp)

---

Why it scores

Criterion	How
Innovation 30% (tiebreaker)	the composition, not the primitive: receipts + epistemic claim-typing + GSAR tiered recovery + auto-act/escalate policy gate + audit ledger, served over MCP; reasoning_content as audit artifact; two-speed qwen-flash/qwen3.7-max fleet under a thinking_budget
Technical Depth 30%	deterministic verifier (no LLM in the hot path), modular adapters, hash-chained ledger, reproducible benchmark + ablations
Problem Value 25%	fabricated tool results are the #1 blocker to autonomous agents; a first-order model puts a 1% fabrication rate at ~$4k/mo in phantom refunds (dollar model above), and Moffatt v. Air Canada (2024 BCCRT 149) already made a company pay for one fabricated support answer; ReceiptGuard is a drop-in MCP gateway any Qwen agent installs
Presentation 15%	live split-screen demo: watch the agent get caught lying and self-correct, on the record

Benchmark

receiptguard bench builds an adversarial set (fabricated reference, value mismatch, false absence) and compares ReceiptGuard against an ablation and a receipt-free judge. The judge row below is a real qwen-flash call on Alibaba Model Studio (run with DASHSCOPE_API_KEY):

System	Detection	False-positive	Overhead
ReceiptGuard	100%	0%	0.007 ms/claim
ReceiptGuard (no value-check) — ablation	66.7%	0%	0.004 ms/claim
qwen-flash judge (no receipts)	0.0%	0%	~1040 ms/claim

Reading it: the ablation (cross-check tool presence but skip numeric matching) misses every value/count mismatch — proving the receipt value-check is load-bearing, not decoration. The real qwen-flash judge catches 0% of the fabrications — because they are plausible, and a model with no receipts structurally cannot tell a real tool result from an invented one — while costing ~1000× more latency per claim. ReceiptGuard hits 100%/0% deterministically at sub-millisecond cost, because it checks cryptographic receipts, not plausibility. That contrast (real LLM judge at 0%) is the whole thesis. Methodology + the offline heuristic fallback: eval/BENCHMARK_METHODOLOGY.md.

Honesty note: the judge row is a real qwen-flash run (n=24); the set is self-authored and the point it proves is structural — a receipt-free checker cannot verify a plausible fabrication, so it lands at 0% while ReceiptGuard's deterministic receipt check lands at 100%. Re-run at larger n with python eval/benchmark.py <N>.

Extraction-typing eval (receiptguard bench covers the verifier; this covers the claim typer): PYTHONPATH=src python eval/extraction_eval.py scores extract_claims on 15 labeled drafts. Offline: 80% overall typing, 100% tool_derived recall — and every error is conservative (inference→tool_derived), so a mis-type means the verifier checks more claims, never fewer. With the fail-closed gateway, no fabrication slips through a typing error.

Layout

src/receiptguard/  gateway/(receipts,tools) claims/ verify/ recovery/ audit/ agent/ mcp/ api/ llm/
eval/              adversarial benchmark + chart
tests/             pytest invariants
static/            demo UI

License: MIT.