Compliance Evidence Agent

evidence-gate

That badge gates five things, not just "the tests pass": gap detection is exact against the planted deficiencies and the existence-only arm still finds none of them; every tamper class is caught by its expected detector with no no-op tampers; every credential revocation produces zero silent gaps while the existence-only arm produces some; the independent auditor accepts every audit-ready control and still separates artifact quality from control effectiveness; and removing any one of the twenty guards makes a specific test fail.

An agent that collects SOC 2 evidence for 15 controls out of four company systems over scoped MCP tools, stores every artifact with a hash-chained provenance record, and then refuses to call a control done when the evidence is stale, partial, misaimed, or unattested. It finds all six planted deficiencies with the right reason for each. The same evidence through an existence-only checker reports 15 of 15 green — including through a credential revocation that blinded it completely.

---

Why this project exists

Every quarter, somebody assembles hundreds of evidence artifacts for an audit. It is repetitive, multi-system, correctness-critical work — exactly what agents are for. The risk is also exactly what agents are bad at: a collection pipeline that fails quietly produces a dashboard that is green and wrong, and nobody finds out until an auditor does.

So this is built around the three ways that happens, and each one is measured:

1. Collecting evidence today does not make it current. An artifact carries effective_at — the day the review was signed, the scan completed, the restore ran — separately from collected_at. The corpus contains a real, signed, population-complete access review attestation from May 2025 that the collector happily fetches today. Freshness is measured from the first field. A tracker keyed on the second reports it green forever.

2. Most compliance evidence is an absence claim, and absence inverts badly. "No user lacks MFA" and "I can no longer see users" are the same HTTP 200. Directory and ticketing APIs apply permissions as a row filter, so a revoked scope returns fewer rows rather than a 403. Every endpoint that can do that has a count endpoint on a different scope, and no collector may emit a clean artifact from zero observations.

3. Complete evidence of a failing control is not a pass. A vulnerability scan that ran on time and honestly reported four criticals past SLA is perfect evidence and a broken control. Tools that track collection report it green; evidence completeness and control effectiveness are kept apart here and reported separately.

Status

Built and measured. Every number below comes from a committed script running against the deterministic system stand-ins, reproducible with the commands in Reproduce.

Evidence	Result
Test suite	172 passed
Controls, systems	15 controls across 12 TSC criteria, 4 systems, 4 needing human sign-off
Completeness — this checker	9 / 15 audit-ready, 6 gaps across 5 distinct reasons
Completeness — existence-only checker, same evidence	15 / 15 audit-ready, 0 gaps
Gap detection vs the 6 planted deficiencies	precision 1.00, recall 1.00, and 6/6 with the correct reason
Gap detection, existence-only arm	recall 0.00 — all six missed
Provenance over the whole ledger	75 artifacts, 75 ledger entries, 5 runs, chain intact
Tamper detection	8/8 caught by the expected detector, 0 no-op tampers
Credential revocation, 4 scenarios	7 controls damaged, 0 silent gaps; existence-only arm: 10
Independent auditor on artifact quality	11 accepted / 4 rejected; all 9 audit-ready controls accepted
The separation that matters	the 2 effectiveness failures have artifact quality the auditor accepts
MCP surface	4 servers, 19 tools, one credential each; artifacts byte-identical to the in-process path
Mutation check: break each guard	20/20 mutants caught by a specific test

The headline: same evidence, two checkers

Fifteen controls, one collection run, two ways of asking "is this done".

control   criteria  system     effective    age    coverage  this checker            existence-only
CC6.1-a   CC6.1     idp        2026-06-30     1d   62/62     READY                   READY
CC6.1-b   CC6.1     idp        2026-06-30     1d     7/7     READY                   READY
CC6.2     CC6.2     idp        2026-06-24     7d     9/9     READY                   READY
CC6.3     CC6.3     idp        2025-05-12   415d     1/1     GAP stale               READY   <-
CC6.6     CC6.6     cloud      2026-06-30     1d   14/14     READY                   READY
CC6.7     CC6.7     cloud      2026-06-30     1d   40/52     GAP population_incomplete READY <-
CC6.8     CC6.8     idp        2026-06-30     1d   62/62     READY                   READY
CC7.1     CC7.1     ci         2026-06-27     4d     1/1     GAP control_ineffective READY   <-
CC7.2     CC7.2     cloud      2026-06-30     1d     1/1     READY                   READY
CC7.3     CC7.3     ticketing  2026-06-27     4d   11/11     READY                   READY
CC7.4     CC7.4     ticketing  2026-05-19    43d     1/1     GAP unsigned            READY   <-
CC8.1-a   CC8.1     ci         2026-06-30     1d   47/47     GAP control_ineffective READY   <-
CC8.1-b   CC8.1     ci         2026-06-30     1d     1/1     READY                   READY
A1.2-a    A1.2      cloud      2026-06-10    21d     1/1     GAP wrong_scope         READY   <-
A1.2-b    A1.2      cloud      2026-06-30     1d     6/6     READY                   READY

The existence-only arm is not a straw man. It is what an evidence tracker built around collection status does, and all six deficiencies have an artifact that exists, hashes correctly, and was collected without error.

The six planted deficiencies, and why each is subtle

Each one is a different failure and produces a different gap reason, because "you have a gap" is not actionable and the six get fixed in six different places.

Control	Reason	What was planted	Why existence-only misses it
CC6.3	stale	The IdP's most recent access review is AR-2025-Q2, signed 2025-05-12. This quarter's review never happened, so the newest thing on file is 415 days old.	Collection succeeded today. Freshness has to come from signed_at, not from when the agent looked.
CC6.7	population_incomplete	prod-1 holds 52 data stores. The inventory scan is capped at 40 rows and says so. All 40 are encrypted.	"Zero unencrypted stores" is true and holds only for the 77% that was examined. Only the authoritative store count exposes it.
CC7.1	control_ineffective	The scan ran 2026-06-27 and reported four critical findings open past their 30-day SLA.	Nothing about the evidence is deficient. It is complete, current, and in scope — and the control is failing.
CC7.4	unsigned	The IR tabletop record is complete and recent, and no human has attested that it genuinely tested the plan.	Whether a control needs human judgment is a property of the control, not the artifact.
CC8.1-a	control_ineffective	All 47 changes merged to main carry an approval. Three were approved only by their own author.	"Every change was reviewed" is literally true and the approval rate is 100%. The control is about independence.
A1.2-a	wrong_scope	The control applies to prod-1, which has no restore test in the period. The evidence mapping points the collector at dev-1, which has a clean, successful, recent one.	Present, fresh, complete, and it says the restore succeeded. Only the scope in its provenance disagrees.

A1.2-a deserves a note: the misconfiguration is left in controls/soc2.yaml, marked as deliberate. Somebody wiring a collector at the wrong account is one of the most common things that actually happens on an engagement, and it is the failure provenance exists to catch.

Credential revocation: the silent kind

Four scenarios, one per system, split across the two ways a real API reacts to a narrowed credential. A silent gap is a control a checker still calls audit-ready while its evidence is degraded — computed from the artifacts, not from the checker's own verdict, so neither arm is grading itself.

Scenario	API behaviour	Damaged	This checker	Existence-only
idp-users-read-revoked	filter — 200 with an empty roster	CC6.1-a	0 silent	2 silent
cloud-config-read-revoked	deny — 403	CC6.6, CC6.7	0 silent	2 silent
ci-token-expired	deny — 403	CC7.1, CC8.1-a, CC8.1-b	0 silent	4 silent
ticketing-incidents-narrowed	filter — 4 of 11 incidents, no error	CC7.3	0 silent	2 silent

The two deny scenarios are the easy half; any tool notices a 403. The filter ones are the point:

GAP CC6.1-a (collection_failed): observed 0 of 62 expected records — an empty result
cannot support a compliance claim, whether the population is truly clean or the
credential can no longer see it

That artifact's content, read on its own, says users_examined: 0, users_without_mfa: [], exception_count: 0. There is nothing in the payload to notice. The only thing that catches it is knowing the workforce is 62, from a directory:stats endpoint on a different scope. Five of the fifteen controls have that kind of independent population cross-check, and they are exactly the endpoints that can silently return fewer rows — that is the design rule, and a test pins which scopes are the silent ones so an endpoint cannot quietly switch to raising and make these scenarios pass on the easy path.

A failed collection still produces a ledgered artifact with status='failed' and the reason. "We could not collect this" is evidence an auditor needs, and logging the error and continuing is what loses it.

Provenance: hashed, chained, and attacked

Each artifact binds its content with sha256. An append-only ledger binds the artifact id, the content digest, and a digest of the provenance — control, system, scope, collecting identity, both timestamps, population counts, status — to the previous entry's hash. Content hashing alone would catch an edited artifact but not a deleted one, and an evidence package with a quietly removed inconvenient artifact is the failure that actually matters in an audit.

Eight tamper classes, each applied to a throwaway copy of the real database, each asserting which detector fired:

Tamper	Caught by
Delete the three self-approved changes from the change-approval artifact	content_mismatch
Relabel the misaimed artifact's scope from dev-1 to prod-1	provenance_mismatch
Move the 14-month-old attestation's effective date into this quarter	provenance_mismatch
Alter a ledger entry to match changed content	entry_hash_mismatch
Remove a ledger entry from the middle of the chain	chain_broken
Swap two entries so the collection order is misrepresented	entry_hash_mismatch
Drop the incomplete-coverage artifact out of the package	orphan_ledger_entry
Add a fabricated restore-test artifact that was never collected	unledgered_artifact

The two middle rows are the ones with a motive: each is a single-field edit that would turn a real gap green, and each is caught because provenance is hashed into the ledger rather than merely stored beside the content. The untampered store verifying clean is checked first — a detector that fires on an intact package would "catch" all eight and mean nothing.

The auditor disagrees in the right places

An independent reviewer grades artifact quality on five criteria. It does not import classify, effectiveness, or provenance, and it recomputes every hash with its own hashlib and json calls, so agreement between the two is a signal rather than a tautology. Both facts are enforced by tests that walk the import graph.

                        provenance  freshness  population  scope  attested   verdict
9 audit-ready controls      ok          ok         ok        ok      ok      accept
CC6.3                       ok          NO         ok        ok      ok      reject
CC6.7                       ok          ok         NO        ok      ok      reject
A1.2-a                      ok          ok         ok        NO      ok      reject
CC7.4                       ok          ok         ok        ok      NO      reject
CC7.1, CC8.1-a              ok          ok         ok        ok      ok      accept  <-

Two results, and the second is the interesting one:

• Every control this pipeline calls audit-ready passes independent review, 9 of 9. Without that, "audit-ready" is a label the tool grants itself.
• CC7.1 and CC8.1-a are accepted on quality and still refused by the pipeline. Their evidence genuinely is fine; their controls are failing. An auditor that rejected all six planted deficiencies would agree with the pipeline for the wrong reason, and CI fails if that separation ever collapses.

Each rejection also names a single specific criterion, tested — a generic rejection is not a finding anyone can act on.

Collectors are MCP tools, four servers, one credential each

evidence mcp-tools

Four servers, not one. Each holds exactly one system's credential and publishes only that system's collectors, so asking the CI server for identity evidence is refused by name:

CC6.1-a draws its evidence from 'idp'; this server only holds the 'ci' credential.
Ask the 'idp' server.

Every tool returns the evidence plus the provenance the store needs — the scope actually read, effective_at, observed and expected population and where the expectation came from, and a status of collected or failed. A tool that reports failed has not given you evidence, and the published description says so.

This is not a parallel implementation. evidence collect --via-mcp drives the entire collection run through the tools, and a test asserts the resulting artifacts are byte-identical to the in-process path — same content hashes, same provenance, same population counts. An MCP server that is built but never driven end to end can drift from the code it claims to expose, and nobody notices until an agent gets a different answer than the dashboard.

Scheduling: stale evidence stays due

Due-ness is computed from effective_at, which has a consequence worth stating: a control whose only evidence is a 415-day-old attestation stays due forever, no matter how many times the agent successfully re-fetches it. Re-collecting stale evidence is not progress. A scheduler keyed on collection timestamps would mark CC6.3 done every quarter and never surface it again.

uv run evidence collect --only-due   # collects CC6.3 and skips the other 14

Reproduce

uv sync --extra dev
uv run evidence init                 # collect the period + replay the review queue

uv run pytest                        # 172 tests
uv run evidence dashboard --compare  # both checkers, side by side
uv run evidence gaps                 # scored against the planted deficiencies
uv run evidence verify               # provenance over the whole ledger
uv run evidence tamper-demo          # 8 tamper classes, 8 expected detectors
uv run evidence audit                # the independent artifact-quality review
uv run evidence inject all           # revoke each credential, count silent gaps
uv run evidence review               # the human queue for judgment controls
uv run evidence mcp-tools            # the published collection surface
uv run evidence collect --via-mcp    # the same run, over the protocol
uv run evidence show CC6.7 --content # one control, end to end
uv run evidence report               # writes results/report.json + dashboard.md

python scripts/mutation_check.py     # break each guard; all 20 must be caught
python scripts/ci_local.py           # every CI step, parsed from the workflow YAML

Everything runs offline: no model, no network, no secrets. EVIDENCE_AS_OF moves the collection date if you want to watch evidence age out of its windows.

Layout

4.07-compliance-evidence-agent/
├── controls/
│   ├── soc2.yaml            # 15 controls -> evidence requirements, as configuration
│   └── signoffs.yaml        # this cycle's recorded human review decisions
├── src/evidence/
│   ├── systems/             # the four company stand-ins
│   │   ├── base.py          # credentials, scopes, and DENY vs FILTER (enforced)
│   │   ├── idp.py           # workforce, admins, provisioning, access review, devices
│   │   ├── cloud.py         # two accounts: inventory, network, logs, backup, alarms
│   │   ├── ci.py            # merged changes, branch protection, vulnerability scans
│   │   └── ticketing.py     # incidents and the IR tabletop record
│   ├── controls.py          # loads and hard-validates the framework
│   ├── collectors.py        # one per evidence kind, plus the population guard
│   ├── mcp_server.py        # the collectors as scoped MCP tools, per system
│   ├── gateway.py           # in-process and over-MCP paths, proven equivalent
│   ├── artifacts.py         # content hashing, provenance, the append-only ledger
│   ├── provenance.py        # verification: recompute everything, trust nothing
│   ├── collection.py        # the schedule, and the collection run
│   ├── effectiveness.py     # is the control OPERATING, as opposed to documented
│   ├── classify.py          # audit-ready or not, and exactly why; both arms
│   ├── review.py            # the human queue, and independence
│   ├── auditor.py           # independent artifact-quality review
│   ├── tamper.py            # the eight attacks on the evidence package
│   ├── inject.py            # credential revocation, and silent-gap scoring
│   ├── groundtruth.py       # the planted deficiencies (unreachable from detection)
│   ├── report.py            # assembles results/
│   └── cli.py               # evidence init|collect|dashboard|verify|audit|inject|...
├── scripts/
│   ├── mutation_check.py    # breaks each guard; every mutant must be caught
│   └── ci_local.py          # runs CI's steps verbatim from the YAML
├── results/                 # committed evidence (report, dashboard, mutation check)
└── data/                    # the evidence store (gitignored; rebuilt by `evidence init`)

Honesty notes

• The four systems are stand-ins, and say so. They are shaped like an IdP, a cloud account, a CI system, and a ticketing system — same pagination, same scope model, same two failure behaviours — but they are in-process Python, not real APIs. That is a deliberate trade: labelled compliance deficiencies at scale do not exist to test against, and a planted corpus is the only way to measure gap detection honestly.
• 1.00 precision and recall are on a corpus I designed. The meaningful claim is not the number, it is that all six deficiencies are invisible to an existence-only check and each guard is independently load-bearing — the mutation check removes any one of the twenty and a specific test fails. On real data these numbers would be lower; the method is the transferable part.
• min_coverage: 1.0 is a strict choice. Requiring the whole population makes the truncated scan a gap. A real engagement might accept a sampling threshold; that is a config change per control, and the point is that the comparison happens at all.
• Only 5 of 15 controls have an independent population cross-check. The other ten are single records or complete lists with no separate counter, so their expectation is the minimum the control requires. population_source records which is which, and the auditor reports it, rather than the tool implying all fifteen are cross-checked.
• The auditor shares controls with the pipeline, on purpose. An auditor is handed the framework; reading a different definition of the requirements would make any disagreement meaningless. It shares nothing else, and a test enforces that.
• Effectiveness rules are simple by design. They are readable predicates over the collected evidence, not a model. That is the right shape for a control that has to be explained to an auditor, and it means the rules are only as good as the evidence fed to them — which is why most of this project is about the evidence.
• Nothing here writes to a real system. Every collector is read-only, the systems are in-process, and no code path reaches an external API.
• Every number in this README is machine-generated. All of them come from results/report.json, written by evidence report, and CI regenerates and gates on it.

Bugs this build found in itself

Kept because each one would have produced a confidently wrong deliverable.

Defect	Why it mattered
A tamper case was a no-op and read as a missed detection	The content tamper zeroed a field that was already zero, so the store never changed, verification correctly found nothing, and the row reported "not detected". Anyone debugging that would go hunting in the verifier, which was working perfectly. Now every tamper hashes the store before and after and a no-op is a loud, specific error. The same guard exists in the mutation harness for the same reason.
Artifact ids collided across collection runs	Ids were derived from the content and a fixed clock, so the first failure injection after the baseline collected identical evidence, derived an identical id, and failed to insert. Two runs are two separate acts of collection and each needs its own ledger entry even when nothing changed. Fixed by putting the run in the derivation and giving each run its own deterministic instant.
The per-control provenance check ignored which run it was checking	assess_control was scoped to a run but its provenance verification looked up the newest artifact, so any historical assessment silently verified the wrong evidence. Invisible until injections started appending runs.
The report described degraded evidence as the baseline	Each injection appends a run, and the schedule and review queue were computed after the sweep — so the committed report's "what is due" and "what awaits review" were statements about a revoked-credential run. Now captured before the sweep, with a test that fails if the ordering regresses.
Injection damage was boolean, so "got worse" was invisible	CC6.7 sits at partial coverage in the baseline. When cloud-config-read-revoked destroyed its collection entirely, a boolean degraded flag reported no change, and every scenario listed the same standing deficiencies alongside its own damage. Made ordinal (full / partial / unusable), and each scenario now asserts it damaged exactly the controls it claims.
One guard was not load-bearing until it was tested	The mutation check found that removing the "population expectation is zero" guard broke no test — the coverage comparison caught the case anyway. But it produces a different diagnosis: "the endpoint that tells you how many stores exist returned nothing" is a broken measurement apparatus, while "40 of 52" is a capped scan, and those get fixed in different places. Added the zero-population test; all 20 mutants now caught.
Two provenance-field tests wrote back the values already there	Same class as the tamper no-op: the status and system cases used the row's existing values, so nothing changed and the tests read as holes in the hashing. Each case now asserts the field actually differs before editing it.
A ledger reorder collided on a unique index mid-swap	Swapping two entries directly violates ledger.artifact_id's uniqueness on the way through. Worth keeping the constraint: it also means an attacker cannot simply duplicate an entry.