Veritas

Veritas (Latin, "truth"). A custom Model Context Protocol server that lets an AI agent triage a compromised host the way a senior DFIR analyst does — and that makes the two failure modes the judges care about most, evidence spoliation and hallucinated findings, architecturally impossible rather than merely discouraged by a prompt.

SANS "Find Evil!" Hackathon submission — Approach #2 (custom MCP server).

---

The problem (and why prompts aren't enough)

The baseline approach to "agentic DFIR" hands a model a shell on a SIFT workstation and a prompt that says please be careful and don't make things up. Two things go wrong:

1. Spoliation. A shell can dd, del, mount read-write, or exfiltrate. One bad tool call destroys the evidence and the case. A prompt asking the model to behave is not a control an examiner can testify to.
2. Hallucination. LLMs assert tidy conclusions that the data never supported. In forensics an unsupported "finding" is not a minor error — it is the thing that gets a case thrown out.

Veritas removes both at the architecture level.

How Veritas is different

Concern	Prompt-only baseline	Veritas
Destroying evidence	"Please don't."	No tool exists that writes to, deletes, or mounts evidence. Evidence handles are opened O_RDONLY; writes resolving inside an evidence root are refused by EvidenceGuard.
Running arbitrary commands	execute_shell_cmd	No shell tool. Only 8 typed, allow-listed, read-only forensic tools. Shell metacharacters and forbidden binaries are rejected in depth.
Hallucinated findings	hope	Grounding enforcement: a finding that does not cite the exec_id of a real, recorded tool run is rejected by the server.
"Confirmed" vs guess	undifferentiated prose	Explicit confidence tiers (CONFIRMED/INFERRED/HYPOTHESIS/RETRACTED); CONFIRMED requires ≥2 independent source families (disk/log/memory).
First-pass mistakes	shipped as final	Self-correcting loop: retracts false positives against a known-good baseline, fills gaps the correlation engine finds (timestomping), and promotes corroborated findings — iterating to a fixed point.
Context-window overload	raw tool text dumped into the model	Raw output is parsed server-side into compact typed records; the model reasons over fields, not megabytes.
Explainability	none	Append-only JSONL audit log. Every tool run gets a stable exec_id; every finding traces back to it. Full chain of custody.
"Trust me, it works"	—	Reproducible accuracy benchmark vs documented ground truth, runnable offline with zero third-party deps.

Headline result (reproducible: python -m veritas.benchmark)

On the bundled synthetic case WEBSRV01, the same engine with self-correction turned off (the prompt-only baseline) vs on (Veritas):

	TP	FP	FN	Precision	Recall	F1	Hallucination
Prompt-only baseline	6	1	2	0.857	0.75	0.80	0.143
Veritas	8	0	0	1.00	1.00	1.00	0.00

The baseline misses timestomping and the C2 channel, and over-flags a benign OneDriveStandaloneUpdater.exe. Veritas catches the timestomp via $SI/$FN correlation, confirms the C2 across memory + network, and retracts the OneDrive false positive against a known-good baseline.

Quick start (60 seconds, no dependencies)

# from the repo root — pure standard library, simulation mode
$env:PYTHONPATH = "src"
python -m veritas.cli triage        # human-readable findings report
python -m veritas.benchmark         # the accuracy table above
python -m pytest -q                 # 46 tests, incl. spoliation-impossibility

Or install it and use the console script:

pip install -e .
veritas triage
veritas benchmark

To drive it from Claude Code as a real MCP server:

pip install -e ".[mcp]"
# .mcp.json + claude/CLAUDE.md + claude/settings.json are included; see docs/TRY_IT_OUT.md

Architecture at a glance

Host agent (Claude Code)
        │  MCP (stdio) — only typed, read-only tools are exposed
        ▼
┌──────────────────────── Veritas MCP server ────────────────────────┐
│  8 forensic tools ──► Executor (allow-list, shell=False) ──► EvidenceGuard │
│        │                         │                          (O_RDONLY)     │
│        ▼                         ▼                                          │
│   Parsers (raw→typed)      AuditLog (append-only JSONL, exec_id)            │
│        │                         ▲                                          │
│        ▼                         │ grounding check                         │
│   TriageLoop ──► FindingStore (rejects ungrounded; confidence tiers)        │
│        │            ▲                                                        │
│        ▼            │                                                        │
│   CorrelationEngine + KnownGoodBaseline (self-correction)                   │
└─────────────────────────────────────────────────────────────────────────┘
        │ writes ONLY to:  exports/  analysis/  reports/   (never evidence)

Full diagram, trust boundaries, and the prompt-guardrails vs. architectural-guardrails distinction are in docs/ARCHITECTURE.md.

Repository map

Path	What it is
src/veritas/	The engine (stdlib-only core; mcp is an optional extra)
fixtures/case_websrv01/	Synthetic but internally-consistent intrusion + simulated tool output
fixtures/ground_truth/	Documented ground truth used by the benchmark
tests/	46 tests, including proofs that spoliation/hallucination are blocked
logs/sample_agent_execution.jsonl	A real captured audit trail from one triage run
claude/	CLAUDE.md operating instructions + settings.json permissions
.mcp.json	MCP server registration for Claude Code
docs/	The eight submission documents (see below)

Submission documents

1. This README — overview & quick start.
2. docs/ARCHITECTURE.md — design, trust boundaries, guardrail taxonomy.
3. docs/ACCURACY_REPORT.md — methodology + reproducible numbers.
4. docs/DATASET.md — the synthetic case, how it was built, and why it's safe.
5. docs/TRY_IT_OUT.md — step-by-step, incl. Claude Code wiring.
6. docs/DEMO_SCRIPT.md — the 5-minute demo-video script.
7. docs/SUBMISSION.md — Devpost write-up / story & judging-criteria map.
8. logs/sample_agent_execution.jsonl — sample agent execution log (chain of custody).

License

MIT — see LICENSE. All bundled case data is synthetic; no real systems, persons, or indicators are represented.