GoldenMatch web workbench — pair drilldown with NL prose

_{Pair drilldown in the web workbench: cluster members, field-level diff, and a one-line NL explanation per pair. pip install goldenmatch[web] then goldenmatch serve-ui <project>. More screenshots →}

# Dedupe a CSV in 30 seconds — zero config, writes <timestamp>_golden.csv.
# Add --tui to review interactively, --output-all for every artifact.
pip install goldenmatch && goldenmatch dedupe customers.csv

# From Python — zero-config, returns golden records
python -c "import goldenmatch as gm; gm.dedupe('customers.csv').golden.write_csv('deduped.csv')"

npm install goldenmatch     # TypeScript / edge runtimes
pip install golden-suite    # the WHOLE suite (Check + Flow + Match + Analysis + Pipe + InferMap) + native

Unreleased — Embeddings are first-class on Fellegi-Sunter matchkeys. embedding and record_embedding field scorers now train (EM) and score end-to-end on the probabilistic path via the vectorized matrix — previously they raised Unknown scorer on both training and scoring. They are matrix-only, so a matchkey carrying one always runs vectorized, and the TUI now routes FS through the same native/vectorized selector.

v3.3.0 — 3.3.0 — negative evidence on Fellegi-Sunter matchkeys. negative_evidence now works on type: probabilistic matchkeys as EM-learned __ne__ dimensions (no labels needed; penalty_bits as a fixed override), and the Splink migration upgrade pass gains a fan-out lever — a risk-gated NE suggestion plus cluster-guard tuning from your reference clusters. goldenmatch-native 0.1.15 scores NE in the Rust kernels (FS_SUPPORTS_NE; older wheels keep the pure-Python fallback automatically).

v3.1.0 — 3.1.0 — polars is optional (and the polars-free install is the fast configuration). The engine is Arrow-native end to end with the Rust fused kernels on the hot paths (a zero-polars CI gate proves a full dedupe with polars imports blocked); pip install 'goldenmatch[polars]' is a compatibility extra (classic lane, kernel-absent golden replay, cell-quality weighting), byte-identical to 3.0.x.

---

Why a suite?

Each tool stands alone, but they compose into a single pipeline:

flowchart LR
    raw([raw rows])
    golden([golden records])

    subgraph orchestration ["GoldenPipe orchestrates"]
        direction LR
        infermap[InferMap]
        goldencheck[GoldenCheck]
        goldenflow[GoldenFlow]
        goldenmatch[GoldenMatch]
        infermap --> goldencheck --> goldenflow --> goldenmatch
    end

    raw --> infermap
    goldenmatch --> golden

Step	Role
InferMap	schema mapping — auto-aligns columns across heterogeneous sources
GoldenCheck	profile + validate — encoding, format, anomaly detection
GoldenFlow	standardize + transform — phone, date, address, categorical normalization
GoldenMatch	dedupe + cluster + survivorship — fuzzy / exact / probabilistic / LLM
GoldenAnalysis	analysis + reporting — one exportable report over any stage, plus cross-run regression detection
GoldenPipe	orchestrator — declarative YAML pipeline wiring the steps

What sets it apart:

• Zero-config that beats hand-tuned. 96.4% F1 on DBLP-ACM out of the box; the opt-in Fellegi-Sunter engine beats expert-tuned Splink head-to-head on every dataset Splink scores (historical_50k pairwise F1 0.778 vs 0.757, cluster B³ 0.844 vs 0.789; one shared evaluator, reproducible bake-off). Every step self-verifies (preflight + postflight) and returns an inspectable report instead of failing silently.
• A healing loop, not a one-shot. Zero-config gets you most of the way; the healer attaches ranked, self-verified config tweaks and closes the gap to expert-tuned without you being the expert. ↓ details
• Durable identity. Learning Memory persists corrections across runs (re-anchored across row reorders); the Identity Graph gives stable entity_ids that survive re-runs, an append-only event log, and create / absorb / merge / split semantics on CLI, REST, MCP, and SQL.
• Privacy-preserving record linkage — match across organizations without sharing raw data (PPRL, 92.4% F1 on FEBRL4).
• AI-native by design — every package ships an MCP server, a REST API, and an A2A agent surface (70+ MCP tools across the suite), all exposing the same JSON telemetry shape across web, TUI, CLI, Postgres, DuckDB, and MCP.
• Polyglot parity, edge-safe, optional native speed. The full suite ships on npm alongside PyPI; Python and TypeScript track the same outputs to 4-decimal precision. The TS cores are dependency-free and node:*-free (browsers, Cloudflare Workers, Vercel Edge, Deno); an opt-in WebAssembly backend (await enableWasm()) swaps in the same pyo3-free Rust kernels the Python wheels and SQL UDFs use, with pure-TS as the byte-identical default.
• SQL-native at parity — the same functions run inside PostgreSQL (pgrx) and DuckDB: dedupe / match / score / auto-config + telemetry / identity graph, profiling, evaluate, Fellegi-Sunter scoring, and GoldenFlow transforms.
• Production paths — Postgres sync, daemon mode, lineage tracking, review queues, dbt integration, GitHub Actions.

---

The healing loop

GoldenMatch's core workflow is a loop, not a one-shot:

1. Zero-config first pass — dedupe_df(df) runs with no rules and no training data; auto-config picks a defensible config and you get good results immediately.
2. You get the config it chose — on result.config: inspectable, diffable, versionable. Never a black box.
3. The healer suggests tweaks — every run checks a free signal and, when there's headroom, attaches ranked, explainable, self-verified edits to result.suggestions. Each is kept only if it doesn't worsen an unsupervised health proxy, so a tweak never makes results worse.
4. You apply them — dedupe_df(df, heal=True) applies and re-runs in one call (returning the healed result.config + a result.heal_trail); or take the wheel with apply_suggestion.
5. Results improve. Repeat — until the healer goes quiet.

Wired into the default pipeline on every surface — Python (suggest=True / heal=True / review_config), CLI (--suggest / --heal), MCP & A2A, REST, web, TUI, and the edge-safe TypeScript port via WebAssembly (enableSuggestWasm()). Needs goldenmatch[native]; degrades gracefully without it (attaches nothing, never errors). Kill-switch GOLDENMATCH_SUGGEST_ON_DEDUPE=0. Full details: config-suggestions.

---

The Suite

Package	Lang	What it does	Install
golden-suite	Python	One-line meta-install: the whole suite + native acceleration, defaulted to the perf-optimized config.	pip install golden-suite
GoldenMatch	Python · TS	Zero-config entity resolution. Fuzzy + exact + probabilistic + LLM. Headline package.	pip install goldenmatch · npm i goldenmatch
GoldenCheck	Python · TS	Data-quality scanning: encoding, Unicode, format validation, anomaly detection.	pip install goldencheck · npm i goldencheck
GoldenFlow	Python · TS	Transforms & standardizers: phone, date, address, categorical normalization.	pip install goldenflow · npm i goldenflow
GoldenPipe	Python · TS	Orchestrator wiring Check → Flow → Match → Identity → Analysis into one declarative pipeline.	pip install goldenpipe · npm i goldenpipe
InferMap	Python · TS	Schema mapping — auto-aligns columns across heterogeneous sources.	pip install infermap · npm i infermap
GoldenAnalysis	Python · TS	Cross-cutting analysis & reporting — any stage's artifacts (or a raw DataFrame) → a unified exportable AnalysisReport; optional Rust / WASM kernels.	pip install goldenanalysis · npm i goldenanalysis
goldenmatch-extensions	Rust	Postgres extension (pgrx) + DuckDB UDFs. SQL-native fuzzy matching.	source build
dbt-goldensuite	dbt · Python	dbt package — dedupe + match materializations (incl. zero-config FS), an ER build gate, quality tests, transforms, identity-graph reads.	packages.yml (git subdir)
goldencheck-action	YAML	GitHub Action — fail PRs that introduce data-quality regressions.	Marketplace

The deepest docs live in packages/python/goldenmatch/README.md (~1,300 lines: full feature list, CLI, architecture, benchmarks).

Knowledge graphs

Entity resolution is the stage most GraphRAG pipelines do badly — duplicate surface forms of the same entity scatter across documents. Two packages put GoldenMatch's resolution there:

Package	What it does	Status
goldenmatch-kg	Drop-in GoldenMatch resolution as the ER stage of existing KG frameworks (neo4j-graphrag, LlamaIndex PropertyGraphIndex, Graphiti). One framework-agnostic resolve_entities core + per-framework adapters. Lift measured by ER-KG-Bench, not asserted.	in-repo · first PyPI release pending
goldengraph	Build-your-own-KG from text — text → LLM extraction → GoldenMatch resolution → a durable bi-temporal store. Engine (store / query / community detection) is pyo3-free Rust; ER is the differentiator.	in-repo · first PyPI release pending

---

Real-world pipelines

Reproducible end-to-end pipelines running GoldenMatch on public data at scale, each with measured headline numbers vs baselines:

• shell-company-network — investigative ER across ICIJ Offshore Leaks + OpenSanctions + GLEIF + UK PSC + disqualified-directors. −62.5% analyst-hours to triage vs single-source baselines; +133% adversarial perturbation recovery.
• vuln-attribution — cross-database ER on 6.1M OSS vulnerability records across 40 sources. 6,126,895 records → 847,475 canonical vulns in ~5 minutes on a single 64GB runner via the full suite.
• sanctions-reconciliation — cross-list coverage on 85 public sanctions lists across 50+ jurisdictions, plus 10-year OFAC SDN history and PEP/crypto cross-analysis. A coverage-gap benchmark for any screening vendor.

---

Choose your path

I want to...	Go here
Deduplicate a CSV right now	goldenmatch quick start
Match records from PDFs / images (unstructured input)	document ingest
Use from Claude Desktop / Code	goldenmatch — MCP
Edit rules in a browser, label pairs, compare runs	goldenmatch — Web UI
Build AI agents that deduplicate	ER Agent / A2A wiki
Profile data quality before matching	goldencheck
Standardize messy fields (phone, date, address)	goldenflow
Run the full pipeline declaratively	goldenpipe
Map columns across schemas	infermap
Analyze + report across stages and runs	goldenanalysis
Write TypeScript / Node / Edge (optional WASM)	packages/typescript/goldenmatch
Match in Postgres / DuckDB SQL	packages/rust/extensions
Add data-quality gates to dbt	dbt-goldensuite
Block bad data in GitHub PRs	goldencheck-action
Run as Airflow DAGs	examples/airflow/ — 13 drop-in DAGs
Run from a single MCP container	goldensuite-mcp

---

Quick examples

Python — dedupe in 30 seconds

import goldenmatch as gm

result = gm.dedupe("customers.csv")               # zero-config
print(result)                                     # DedupeResult(records=5000, clusters=847, match_rate=12.0%)
result.golden.write_csv("deduped.csv")

result = gm.dedupe("customers.csv",               # or be explicit
    exact=["email"], fuzzy={"name": 0.85, "zip": 0.95},
    blocking=["zip"], threshold=0.85)

TypeScript — edge-safe core

import { dedupe } from "goldenmatch";

const result = dedupe(rows, { fuzzy: { name: 0.85 }, blocking: ["zip"], threshold: 0.85 });
console.log(result.stats);   // { totalRecords, totalClusters, matchRate, ... }

Runs in browsers, Vercel Edge, Cloudflare Workers, Deno — and optionally swaps in the Rust score-core kernel via await enableWasm(). ~940 tests, strict TypeScript.

Composed pipeline

import goldenpipe as gp

pipeline = gp.Pipeline.from_yaml("pipeline.yaml")   # check → flow → match
result = pipeline.run("customers.csv")
result.report.write_html("report.html")

Web workbench — pip install 'goldenmatch[web]' then goldenmatch serve-ui my-project (opens http://localhost:5050): edit rules with live validation, preview against a sampled slice, label pairs (mirrored into Learning Memory), compare runs, sweep parameters.

More: examples/ has runnable demos — Python (quickstart, full pipeline, customer 360, PPRL, review, MCP client) · TypeScript (quickstart, Vercel Edge, MCP client) · Airflow (production-shaped DAGs).

---

Install

The whole suite, configured for speed — the golden-suite meta-package pulls in every package plus the native (Rust) kernels, pinned to compatible versions and defaulted to the perf-optimized config (native paths on, no env vars). The native wheels are hard dependencies on purpose: a platform without a wheel fails loudly rather than silently running the slow pure-Python path.

pip install golden-suite
golden-suite doctor        # verify every package + native kernel is importable and healthy
golden-suite optimize      # repair / re-enable the perf-optimized config

pip install golden-suite[mcp]     # + the aggregator MCP server (every tool, one endpoint)
pip install golden-suite[agent]   # + GoldenPipe serving surfaces (A2A + REST + TUI)
pip install golden-suite[all]     # everything

Just GoldenMatch — ships fat optional extras so you only pay for what you use (native acceleration is already default on common platforms):

pip install goldenmatch                    # core (CSV in, CSV out) + native
pip install goldenmatch[documents]         # + PDF/image ingest (run on unstructured input)
pip install goldenmatch[embeddings]        # + sentence-transformers, FAISS
pip install goldenmatch[llm]               # + Claude / OpenAI for LLM boost
pip install goldenmatch[duckdb]            # + DuckDB out-of-core backend
pip install goldenmatch[ray]               # + Ray distributed backend (50M+ rows)
pip install goldenmatch[postgres]          # + Postgres sync  (also: [snowflake] [bigquery] [databricks] [salesforce])
pip install goldenmatch[quality]           # + GoldenCheck    (also: [transform] for GoldenFlow)
pip install goldenmatch[mcp]               # + MCP server     (also: [agent] A2A, [web] browser workbench)

goldenmatch setup    # interactive wizard: GPU, API keys, database

Sister packages compose: pip install goldenpipe[full] brings in Check + Flow + Match together.

---

Deploy

Remote MCP (nothing to install)

Hosted on Smithery — connect any MCP client:

{ "mcpServers": { "goldenmatch": { "url": "https://goldenmatch-mcp-production.up.railway.app/mcp/" } } }

70+ MCP tools across the suite: deduplicate, match, explain, review, link privately, configure, scan quality, transform, synthesize golden records, analyze trends and regressions, manage Learning Memory.

Containers

Every package ships as a multi-arch image (linux/amd64 + arm64) on GHCR — pull anonymously:

docker run -p 8300:8300 ghcr.io/benseverndev-oss/goldensuite-mcp:latest   # one container, every tool
docker run -p 8200:8200 ghcr.io/benseverndev-oss/goldenmatch-mcp:latest   # per-package (also: goldencheck/goldenflow/goldenpipe/infermap -mcp)
docker run -e POSTGRES_PASSWORD=secret ghcr.io/benseverndev-oss/goldenmatch-extensions:latest   # Postgres + extension

Tags: :latest (current main), :main-<sha7> (every push, immutable), :vX.Y.Z / :vX.Y (on release). See goldensuite-mcp for the aggregator's tool-collision behaviour.

Airflow

13 drop-in DAGs at examples/airflow/ (TaskFlow API, Airflow 2.7+ / 3.x; tunable knobs, idempotent retries, marker-protected against double-processing), grouped by lifecycle stage:

Group	DAGs
Core pipeline	daily_dedupe, incremental_match, warehouse_native (Snowflake), customer_360, identity_graph
Privacy	pprl_linkage (two-party PPRL)
Onboarding & monitoring	schema_align_and_load, schema_drift_alarm, quality_gate
Feedback loop	review_worker, active_learning
Operationalize	reverse_etl (Salesforce/HubSpot), backfill

---

Benchmarks & scale

Published GoldenMatch numbers (DQbench composite 91.04, DBLP-ACM 0.9641 F1, Febrl3 0.9443 F1, NCVR 0.9719 F1) map back to a single committed runner, scripts/run_benchmarks.py. See docs/reproducing-benchmarks.md for per-number commands, dataset URLs, expected output with tolerance, and a one-click reproduction snippet. The same runner powers the weekly benchmarks.yml workflow.

Scale envelope (docs/scale-envelope.md) — per-backend ranges (Polars in-memory < 500K, DuckDB out-of-core 500K–50M, Ray distributed ≥ 50M), block-size failure modes, candidate-pair math, and a decision tree for picking a backend.

Verified at the top end: a full 100M-row dedupe on a 5-node Ray cluster (e2-standard-16, 80 CPU) in 9.2 min (554 s), 20,000,000 golden records recovered exactly, driver peak 0.36 GB RSS. The default distributed path is recall-complete (blocking-key shuffle scoring + distributed randomized-contraction WCC), so duplicates merge correctly no matter how the input is partitioned, and it stays driver-collect-free end to end (#844). A faster per-partition path (GOLDENMATCH_DISTRIBUTED_BLOCK_SHUFFLE=0, ~213 s on a 4-worker run) suits inputs where duplicates already co-locate within partitions. Recipe: configs/distributed-100m.yaml.

---

Repository layout

goldenmatch/
├── packages/
│   ├── python/        goldenmatch · goldencheck · goldenflow · goldenpipe · infermap · goldenanalysis
│   │                  goldensuite-mcp (aggregator) · golden-suite (meta)
│   ├── typescript/    full TS ports (edge-safe cores + WASM) · goldencheck-types
│   ├── rust/extensions/  Postgres pgrx + DuckDB UDFs (own Cargo workspace)
│   ├── dbt/goldensuite/  dbt materializations, tests, macros
│   └── actions/goldencheck/  GitHub Action
├── examples/          python · typescript · airflow (drop-in DAGs)
├── docs/superpowers/  design specs and implementation plans
├── justfile · pyproject.toml (uv workspace) · pnpm-workspace.yaml (Turborepo) · .github/workflows/ci.yml

• Cargo — no root workspace. packages/rust/extensions/ is itself a Cargo workspace (the postgres crate is excluded for pgrx build requirements); Cargo commands run from inside it.
• TypeScript — one pnpm workspace. packages/typescript/* form a single pnpm + Turborepo workspace; .npmrc pins node-linker=hoisted for a flat node_modules (avoids Windows symlink issues).

just install   # uv sync + per-package npm install + cargo fetch
just test      # all languages   ·   just lint   ·   just build

---

Contributing

• Feature work on feature/<name> branches; merge via squash PR. Titles: feat: / fix: / docs:.
• Tests must pass on all three languages where the change applies; the parity harness in packages/typescript/goldenmatch/tests/parity/ enforces 4-decimal Python ↔ TypeScript scorer parity.
• See docs/superpowers/specs/ for design rationale.

TypeScript dev setup (pnpm + Turborepo) — from the repo root:

corepack enable                               # one-time, picks up [email protected] from package.json
pnpm install
pnpm turbo run build test typecheck lint      # full pipeline (cached after first run)

Windows: enable Developer Mode (Settings → For Developers) so pnpm install can create symlinks; if corepack enable needs admin, npm i -g [email protected] is equivalent.

---

_{This repo was formed on 2026-05-01 by folding 8 sibling repos into goldenmatch via git filter-repo (full history preserved) — design · plan. Built by Ben Severn. MIT — see LICENSE.}