Multi Judge Consensus

English · 简体中文

Multi-Judge Consensus (MJC)

A cross-vendor model committee that reviews agent-generated content before it ships.

MJC assembles multiple LLMs from independent vendors (DeepSeek, Zhipu GLM, Alibaba Qwen, and others) into a review committee. Each judge produces a structured opinion; a purely rule-based arbiter votes, and divergent opinions trigger a bounded cross-debate. The result is an architecture-level defense against hallucination: one model's blind spot rarely overlaps with another's.

Zero third-party dependencies (Python stdlib only). Bring your own API keys. Runs fully locally.

---

What it does

• Structured multi-judge review — verdicts (pass / revise / reject / need_human), confidence, and itemized issues (factual_error, logical_error, hallucination, style) in machine-readable JSON.
• Rule-based arbitration — no LLM decides the final verdict. Majority vote; ties or high-confidence minority objections trigger cross-debate (≤ 2 rounds, stops on consensus).
• Deterministic verifier — date-span, percentage-base, and explicit-sum errors are checked by code, not by an LLM: zero cost, zero latency, no false positives by design.
• Cost-tiered routing — screen (1 call) → cheap pair → flagship committee. Escalation is monotonic; identical content re-reviews are free (cache).
• Administration console — a local Web UI for API keys, model catalog, tier presets, trust scores, live review streams, and per-issue disposition records.
• Adapter surface — MCP server, subprocess CLI with gate exit codes, HTTP API, and optional OpenClaw integration.

How it works

content
  ├─ ⓪ verifier      deterministic checks (dates / percentages / sums) — 0 LLM cost
  ├─ ① cache         identical content → previous verdict, 0 calls
  ├─ ② screen        cheap judge (glm-4-flash); pass with conf ≥ threshold → done
  ├─ ③ committee     3 models review independently and in parallel
  ├─ ④ arbiter       pure rule vote: ≥2/3 pass → pass; disagreement → debate
  ├─ ⑤ debate ≤2 rds each judge sees the others' opinions, may change verdict
  ├─ ⑥ factcheck     non-pass only: factual issues re-checked by independent, non-complainant judges
  └─ ⑦ verdict       result + token usage + cost estimate, fully logged

Zero-hallucination design (v0.12.1)

Detect → arbitrate → repair → re-check — engineered so the system never "fixes" a fact into a different error:

• Fact arbitration (mjc/factcheck.py) — before any factual fix is attempted, the claim is independently re-checked by non-complainant, cross-vendor models (confirmed / refuted / unknown). Only confirmed replacements may touch specific facts; refuted means keep the original; unknown allows softening only.
• Falsifier pass (mjc/falsifier.py) — an adversarial red-team reviewer (cross-vendor by default) hunts for the weakest claims (premises, facts, absolutes); its challenges go through the same arbitration, and only confirmed challenges can escalate a committee pass to revise — no false-positive escalations by design.
• External knowledge (opt-in) (mjc/knowledge.py) — arbitration can pull web evidence snippets (Bing/Sogou/Baidu HTML backends, or a custom command backend; off by default, cached & rate-limited) so refutation/confirmation is grounded in retrievable sources rather than model memory alone. v0.11.0 reality check (measured 2026-09-13): the scraped backends are effectively dead — Sogou returns HTTP 403, Baidu returns its anti-bot page, Bing works but has poor recall for specific entities, and overseas endpoints are unreachable from this host. Each backend now records a health status (ok / blocked / unparsed / error) that is surfaced by evidence_check, so "we got blocked" is no longer reported as "no evidence exists"; probe with python3 -m mjc.cli knowledge-probe. For dependable evidence, wire a real search API through the cmd backend (MJC_KNOWLEDGE_CMD).
• Dual-producer repair (mjc/repair.py) — two vendors revise independently under the same rules. v0.7.2 value-level consensus: cross-model wording differs naturally, so "near-identical text" was too strict (blocked every real fix in A/B pilot C5). Agreements are now judged on numeric value sets — equal sets (both dropped the old value) or a true-subset revision (fewer claims) may be applied; text-only rewrites and gutted "no-info" revisions are never auto-applied. Disagreement keeps the original (no single-model error injection).
• Value-operation safety gate (v0.8.0) — after pilot C6 showed a both-producers-agree block can still gut a correct value when the committee false-rejects, only net value→value replacements are allowed: pure drops (losing values) and pure additions are blocked outright; replacements can additionally require blind resample support (settings repair.resample_gate, off by default — see mjc/resample.py, self-consistency line) before adoption.
• Knowledge evidence gate (v0.9.0) — when blind resampling is inconclusive (or the resample gate is off), a value replacement can still be cleared by external retrieval evidence: an opt-in gate (settings repair.evidence_gate, off by default) searches the question only (≤80 chars) and requires ≥ min_snippets snippets that contain the new value and not the old one. A resample conflict can never be overridden by evidence; retrieval errors block conservatively. Both gates off keeps the previous default behavior exactly.
• Backend-merge retrieval (v0.9.1) — evidence search now accumulates and dedupes snippets across the configured backends (merge_backends in repair.evidence_gate, on by default): stops at ≥3 unique snippets or 3 backends tried, retries a backend once on an empty result, and continues past failures — removing the single-backend flakiness reproduced in the csqa-07 demo (merge_backends: false = old path).
• Evidence independence + query hygiene (v0.11.0) — two validity bugs found by auditing the csqa-07 evidence demo, both fixed: (1) the query used to be task + proposed value, so searching "2009" trivially returned pages containing "2009" — the gate was self-certifying (support measured keyword echo, not independent corroboration). The query now contains the question only, and any old/new value literal is stripped defensively. (2) the query used to include the prompt's instruction wrapper ("请用一句话以内回答下面的问题："), which scraped back dictionary pages for the character "请" (6/6 snippets) — task_question() now strips the wrapper first. Also: producer failures are surfaced (producer_errors + note) instead of degrading to a bare "fewer than 2 revisions" message.
• Revision rules (mjc/revision.py) — the repairer must never introduce new specific facts; when in doubt, hedge or soften instead of substituting a guess (reviewer suggestions are leads, not truth).
• Gate default = full committee — deliverable gates skip the cheap screen by default (--screen to opt back in).

Lightweighting (production-oriented; experiment defaults unchanged)

Measured cost per question (2026-09-13, experiment arm C9f, api_calls): clean content 4 calls, ordinary non-pass 6-18, full chain (committee + arbitration + falsifier + two repair rounds + re-review) 32. The heavy tail sits on the non-pass path:

Lever	Setting	Measured / expected	Trade-off
Browser retrieval backend	knowledge.backends=["browser"]	evidence wait 300s -> 2.5s (no anti-bot, no cooldown)	needs Chrome on the host
Batch arbitration	factcheck.batch=true	arbitration block issues x arbiters -> 2 calls (8 -> 2 at 4 issues)	one call must judge several issues; slightly coarser
Screening	screen_enabled=true + screen_model	clean content passes in 1 call (skips the 3-seat committee)	miss risk bounded by screen_conf
Economy tier	current.tier="eco"	3 models -> 2 models	detection rate needs re-testing
Single repair round	caller passes --max-rev 1	saves a whole round (committee + falsifier + arbitration)	one less chance to converge

Every default stays as-is (batch=false, experiment arms pass no_screen=True, evidence gates are opt-in) so A/B comparisons remain valid; the table above is the recommended production recipe.

Tier presets (one-click in the admin console):

Tier	Committee	Strategy
Economy	2x budget models	cheapest, screen-first
Standard	2 budget + 1 flagship	default (benchmark-verified)
Strict	flagship only, no screen	maximum rigor

Measured results

Adversarial benchmark v1 — 21 samples (cross-document contradictions, temporal hallucinations, numerical traps, instruction deviation, plus clean controls), run with real API calls, 2026-09-06:

Pipeline	Defect pass-through	False kills on clean content
No review (shipped as-is)	100% (18/18)	—
Single-model self-check (glm-4-plus)	11.1% (2/18 missed)	—
MJC full pipeline	0% (18/18 caught)	0/3

Full-set recall 1.0 · precision 1.0 · F1 1.0. Per-category: 5/5, 5/5, 5/5, 3/3. Layering: 1 deterministic catch by the verifier (0 LLM cost), 17 by committee + debate.

Reproduce: python3 -m mjc.cli bench --set v1-full (≈ ¥0.6 in API spend). History is appended to logs/bench-history.jsonl for regression tracking.

Code-review case study (2026-09-13)

A 518-line / 14,044-char Python module (an order-settlement engine, compiles clean) was written with 12 defects deliberately injected inside MJC's declared scope — docstring-vs-implementation mismatches (4), numeric/date errors (4), logical contradictions such as dead or unreachable branches (4) — plus 4 correct-but-suspicious decoys (an inclusive >= boundary, a documented discount-stacking order, ROUND_HALF_UP quantization, a negative-value clamp) to measure false positives. Compile/runtime errors were excluded on purpose: those belong to the test suite, not to MJC.

Arm	API calls	Latency	Cost	Defects found	False positives	Output usable
A Deterministic verifier (no API)	0	0.0s	0	0/12	0	yes
B Single-model self-check (glm-4-plus)	1	88s	~0.21	4/12	n/a	no
C MJC full pipeline	7	346s	0.29192	6/12	0	yes

• Zero false positives. All 6 issues MJC raised were verified against the sealed ground truth, and none of the 4 decoys was reported.
• The single-model arm did not produce usable output. Its response hit the output-token ceiling, leaving the JSON array unterminated and unparseable; from item 11 onward it also fell into a repetition loop — roughly only 22 of the 114 recoverable objects are distinct — the remaining 92 restate an existing complaint (the most frequent one appears 13 times, so the 114 objects are 5.2x the distinct content).
• Recall by category (MJC / single model): doc-vs-impl 2/4 vs 1/4, numeric 2/4 vs 2/4, logic 2/4 vs 1/4.
• Cost per defect found: 0.049 for MJC vs ~0.053 for the single-model arm (which was unusable). 59,678 tokens in total (51,545 prompt / 8,133 completion).
• Where MJC is weak (stated plainly): the 6 misses all lack a direct textual contradiction — they require business-semantics reasoning (tax computed on the pre-discount subtotal, a refund path that can exceed the amount paid, an inconsistent coupon threshold, an inverted top-N sort, a missing "unopened" check, a delivery estimate whose comment and constant disagree). MJC is a pre-delivery proofreader, not a replacement for unit tests. The 12 defects are constructed, not a production distribution, and the sample is one module.

Getting started

Step-by-step walkthrough for first-time users (key setup, first review, web console, agent integrations, cost table, troubleshooting): QUICKSTART.md

git clone https://github.com/ElonAug7/multi-judge-consensus.git
cd multi-judge-consensus

python3 -m mjc.cli setup      # ① enter API keys interactively (skippable; stored locally, 0600)
python3 -m mjc.cli webui      # ② admin console at http://127.0.0.1:8123

# ③ review a piece of content
python3 -m mjc.cli judge-only \
  --task "Summarize tomorrow's weather in Beijing" \
  --output "Beijing will have a heavy storm tomorrow"

Requirements: Python ≥ 3.9. No pip install needed. Without any keys you can still run the verifier, the UI, and the offline test suites; with one vendor key the committee shrinks automatically (recommended: two or more vendors).

Keys are read from environment variables (MJC_DEEPSEEK_KEY, MJC_GLM_KEY, MJC_<PROVIDER>_KEY) or keys.local.json (gitignored, chmod 600). See settings.example.json for the configuration template.

CLI

Command	Purpose
setup	interactive first-run key configuration + connectivity probes
doctor	health check: keys, tier, trust scores, cache, cumulative usage
judge-only --task --output	review given text with the committee
review --task	generate → review → rewrite loop (≤ 3 rejections)
auto --task --content [--kind code]	single-shot review with memory context, JSON output
gate --stage design|code|deliver --task --content	stage gate; exit codes pass=0 / revise=2 / reject=3
bench [--set quick|v1-full] [--ablation 0,1,2]	red-team benchmark with history
dispose --in <json>	record per-issue dispositions (adopted / rejected with reason)
savings [--days N] [--json]	savings ledger: tokens / cost / wall-clock avoided by each mechanism
knowledge-probe	probe every retrieval backend and report health (ok / blocked / unparsed / error)
switch on|off|status	pause/resume the auto-review hook (0 cost, fully silent; manual commands unaffected)
webui	local admin console (127.0.0.1:8123)
mcp	MCP stdio server

Integrating with other agents

MCP (Claude Desktop/Code, Cursor, Windsurf, Cline, …):

python3 -m mjc.mcp
# Claude Code:  claude mcp add mjc -- python3 -m mjc.mcp

The review(task, output) tool returns {verdict, votes, issues, api_calls, tokens, cost_yuan}.

Subprocess CLI — one-line JSON plus exit-code gate semantics; usable from any language or CI.

HTTP — POST /api/review with optional token auth (Dify/Coze/n8n custom tools, cross-machine).

OpenClaw — optional native integration: transcript scanning, coding stage gates, live review stream.

Architecture

mjc/
├── providers.py     vendor registry (deepseek/glm/qwen/dashscope/doubao/kimi; add keys to enable)
├── judge.py         single reviewer: structured JSON, memory injection, same-vendor fallback
├── arbiter.py       vote/debate arbitration; per-opinion live events
├── pipeline.py      verifier → screen → committee; fault-tolerant degradation
├── verifier.py      deterministic checks (dates/percentages/sums), zero false positives by design
├── bench.py         adversarial benchmark runner (recall/precision/F1 + history)
├── webui.py         admin console (review / live tasks / settings)
├── mcp_server.py    MCP stdio server
└── settings.py      runtime config (provider registry, model catalog, tier presets)
tests/               18 offline suites (0 API calls; key-dependent cases skip gracefully)
samples/bench-v1.json   adversarial benchmark corpus

Development

python3 tests/test_phase3.py     # pipeline/cache/degradation
python3 tests/test_verifier.py   # deterministic verifier
python3 tests/test_mcp.py        # MCP protocol
# …18 suites total, all offline. GitHub Actions runs them on Python 3.9/3.11/3.12.

Security & notes

• Keys live only in environment variables or a local keys.local.json (0600, gitignored).
• Runtime config, logs, and review records stay local and are never committed.
• Cost figures are estimates from per-vendor price tables (¥/1K tokens), marked as approximations; token counts come from API usage fields.
• The benchmark measures worst-case interception on a constructed corpus, not a natural production distribution.

License

GPL-3.0. Local invocation only; keys are the user's own.