Thread Keeper

tests Python License: MIT PyPI CLIs

Multi-agent shared brain across Claude Code/Desktop, Codex, Antigravity CLI (agy), Gemini legacy, Copilot, and VS Code. Cross-session memory, self-improving skill loops, and inter-agent signaling — one local MCP server turns parallel agent instances into a coordinated multi-agent system instead of N isolated chats.

Every connected client (Claude Code, Claude Desktop, Codex CLI + desktop, Antigravity CLI, Gemini legacy, Copilot, every MCP-aware VS Code extension) shares one SQLite store, one set of threads, one user model, and one learning loop that improves the skill library autonomously over time.

The brief format is dense — structural tags, opaque IDs, ~6 KB per session-start injection. Optimized for agent consumption, not human reading.

---

Why

Every agent CLI starts cold. Context dies at session boundaries. Skills you taught Claude don't transfer to Codex. Threads you closed in yesterday's Antigravity chat are invisible to today's Copilot. Parallel agent instances running the same task don't know about each other and duplicate work or step on each other's writes.

thread-keeper is the substrate underneath. Three things that together make it more than a memory store:

• Collective memory — threads, notes, verbatim quotes, dialectic claims about you. Survives session, restart, CLI swap. One agent records, every other agent (any CLI) reads. The brief injected at session start gives a new agent everything the previous one knew.
• Multi-agent coordination — spawn primitive launches child agents in parallel, each gets a self_cid + sees the same memory. broadcast / whisper / inbox / wait / ask / respond let concurrent sessions signal each other across CLIs. Parent / children / sibling agents become a coordinated swarm, not isolated chats.
• Self-improving skill library — autonomous background loops (auto-review on thread close, shadow-review daemon, extract harvester, candidate-reviewer, weekly Curator, and a thread-janitor that auto-closes idle threads so abandoned work reaches the harvest path — closing is reversible, a note reopens a closed thread) materialize class-level skills as the agents work. Adapted to multi-CLI: SKILL.md is the primary write target and gets mirrored to every known/configured skills root simultaneously (~/.claude/skills/, ~/.codex/skills/, ~/.gemini/config/skills/ for Antigravity, existing ~/.agents/skills/, extra roots from THREADKEEPER_EXTRA_SKILLS_DIRS, and ~/.threadkeeper/skills/), with lessons.md as a fallback for CLIs without a native skills loader.

Foreground MCP servers also run a daily self-update check by default. Source checkouts fast-forward their tracked git branch and reinstall the editable package; PyPI/pipx/venv installs run pip install --upgrade in the current interpreter environment only after the latest PyPI release files have matching Integrity API provenance from the expected GitHub Trusted Publisher. Dirty or diverged git checkouts are skipped rather than overwritten. Restarts are gated on install/setup success plus a subprocess import smoke check, so a broken or unverified update is recorded but the current server keeps running. Upstream PyPI publishing is intentionally gated: merge-to-main checks no longer dispatch uploads, and a release requires a maintainer-signed annotated v* tag plus the protected pypi GitHub Environment described in docs/RELEASING.md.

They also run a twice-weekly installed-skill updater by default. It keeps all configured CLI skill roots in sync, adopts newer local copies installed into a non-primary root, and updates GitHub-backed skills when a tracked upstream source changes.

---

Quickstart

The shortest path — PyPI + pipx (recommended):

pipx install 'threadkeeper[semantic]' && thread-keeper-setup

thread-keeper-setup detects every CLI you have installed (Claude Code / Claude Desktop / Codex CLI + desktop / Antigravity CLI agy / Gemini legacy / Copilot / VS Code), registers the MCP server in each one's config, copies hooks to ~/.threadkeeper/hooks/, and writes a managed instructions block into each CLI's per-user instructions file (CLAUDE.md / AGENTS.md / GEMINI.md / copilot-instructions.md — Claude Desktop and VS Code have no global instructions file, so that step is skipped for them).

Restart your CLI of choice. Hook-capable clients inject a brief on the first message; hookless clients such as Codex and Antigravity CLI either follow the managed instructions block and call brief() / context() before answering, or — on hosts that support MCP resources — pull the brief as the read-only memory://brief resource the host attaches automatically (see MCP primitives).

Alternative installs

If you don't have pipx and don't want to install it:

# uv (Rust-fast Python tool runner) — no clone, single binary on PATH
uv tool install 'threadkeeper[semantic]' && thread-keeper-setup

# Plain pip into a venv
python3 -m venv ~/.threadkeeper-venv
~/.threadkeeper-venv/bin/pip install 'threadkeeper[semantic]'
~/.threadkeeper-venv/bin/thread-keeper-setup

For development (editable install from a git checkout) or to track the bleeding edge:

# One-liner installer — clones to ~/thread-keeper, makes a venv,
# editable-installs, wires every detected CLI. Idempotent — re-run to
# update (it git-pulls + reinstalls).
curl -fsSL https://raw.githubusercontent.com/po4erk91/thread-keeper/main/install.sh | bash -s -- --semantic

# Or fully manual
git clone https://github.com/po4erk91/thread-keeper ~/thread-keeper
cd ~/thread-keeper && python3 -m venv .venv
.venv/bin/pip install -e '.[semantic]'
.venv/bin/thread-keeper-setup

To preview without writing anything:

thread-keeper-setup --dry-run

---

Multi-CLI integration

CLI	MCP config	Instructions file	Hooks	Transcripts ingested
Claude Code	~/.claude.json mcpServers	~/.claude/CLAUDE.md	~/.claude/settings.json hooks	~/.claude/projects/**/*.jsonl
Claude Desktop	~/Library/Application Support/Claude/claude_desktop_config.json mcpServers (macOS); %APPDATA%\Claude\… (Win); ~/.config/Claude/… (Linux)	none (GUI-only)	not supported by the app	none — chats live in Electron IndexedDB
Codex (CLI + desktop)	~/.codex/config.toml [mcp_servers] (shared between CLI and Codex.app)	~/.codex/AGENTS.md	not supported	~/.codex/sessions/**/rollout-*.jsonl
Antigravity CLI (agy)	~/.gemini/config/mcp_config.json mcpServers	~/.gemini/config/AGENTS.md	not wired yet	not yet parsed — sqlite/protobuf under ~/.gemini/antigravity-cli/conversations/*.db
Gemini legacy	~/.gemini/settings.json mcpServers	~/.gemini/GEMINI.md	~/.gemini/settings.json hooks	~/.gemini/tmp/<user>/chats/session-*.jsonl
Copilot	~/.copilot/mcp-config.json mcpServers	~/.copilot/copilot-instructions.md	~/.copilot/hooks.json	~/.copilot/session-store.db (sqlite)
VS Code	~/Library/Application Support/Code/User/mcp.json servers (macOS); %APPDATA%\Code\User\mcp.json (Win); ~/.config/Code/User/mcp.json (Linux)	none (per-workspace only)	not supported	none — extensions own their history

Every CLI that produces parseable transcripts feeds the same dialog_messages table with a source tag, so dialog_search() finds matches regardless of where the conversation happened. Claude Desktop, Antigravity CLI, and the VS Code adapter are the exceptions — MCP registration only; their chats don't reach the table for now (Electron IndexedDB on the Claude Desktop side; sqlite/protobuf on the Antigravity side; per-extension stores on the VS Code side).

VS Code's user-level mcp.json is the central host that every MCP-aware VS Code extension consumes — GitHub Copilot Chat, the Anthropic Claude IDE plugin, the OpenAI Codex IDE plugin, Continue, Cline, … — so a single registration there reaches all of them at once.

Adding a new CLI = one file under threadkeeper/adapters/ implementing the CLIAdapter contract. See CONTRIBUTING.md.

MCP primitives (tools, resources, prompts, elicitation)

MCP has three server primitives. thread-keeper uses all three, mapped to the read/act split, plus MCP elicitation for host-native confirmations:

Primitive	Control	What thread-keeper exposes	When to use
Tools	model-controlled (may act)	the full surface — brief, note, spawn, search, curator_review, …	the agent decides to call them
Resources	application-controlled, read-only	memory://brief, memory://context, memory://dashboard, memory://agent-status	the host attaches/pulls them automatically
Prompts	user-controlled templates	review_recent_threads, run_library_curation, audit_threadkeeper	the user runs them (Claude Code: /mcp__thread-keeper__<name>)

Resources back the genuinely read-only memory views with the same render functions as the matching tools, so the content is identical — memory://brief is brief(), memory://context is context(), and so on. The win is for hookless CLIs: instead of depending on the agent remembering to call brief() (agents focused on their task often skip it), a resource lets the host surface memory as attachable / @-mentionable context through a mechanical channel. The brief resource renders lean and agent-status uses a cached snapshot, so an automatic host pull is side-effect-free.

Prompts turn the curation / audit / review flows into discoverable, parameterized commands; each just drives the existing tools.

Elicitation is a client feature, not a server primitive. When a host advertises form-mode elicitation, high-stakes mutations can pause for a structured user choice instead of relying on an ignorable text nudge. The first flow using it is dialectic_supersede: supported hosts get a flat confirm/reject form before a user-model claim is replaced; unsupported hosts keep the previous immediate tool behavior.

Everything here is additive and capability-gated: a host that advertises the resources / prompts capabilities sees those primitives; one that advertises elicitation.form gets structured confirmations for covered high-stakes writes. Hosts without a capability fall back to the SessionStart hook plus the brief() / context() tools and the existing write behavior — same content, no regression. Static URIs only for now (resource templates with {param} are still unevenly supported across hosts).

Memory egress (cross-provider privacy)

thread-keeper is "one user model … shared across CLIs," and that sharing is by design. The flip side: the most sensitive memory it holds — verbatim_user quotes and the dialectic user-model (claims about you: style, values, workflow) — is rendered into every brief(), and brief() is consumed by whichever LLM vendor backs the active or spawned CLI. So by default, a quote you said to Claude, or a trait inferred about you, can be transmitted to OpenAI (Codex), Google (Gemini / Antigravity), or Microsoft-GitHub (Copilot) on the next session-start or spawn under that CLI. This is a deliberate default, not a leak — but it's worth stating plainly, and it's controllable.

THREADKEEPER_MEMORY_EGRESS scopes the egress of personal-class memory (verbatim + dialectic user-model). work-class (threads/notes/tasks) and shared-class (skills/lessons/concepts) memory always egress.

Value	Personal-class memory egresses to…
all (default)	every vendor — current behavior, brief is byte-identical to pre-policy
same-vendor	Claude / Anthropic only; omitted for OpenAI / Google / Microsoft CLIs
work-only	no vendor — personal memory never leaves the machine

Under a restricted policy, the gated brief() drops the verbatim and user_model (dialectic) sections and leaves a one-line egress policy=…: personal memory … withheld from <vendor> disclosure so the consuming agent knows personal context exists but was intentionally not sent. The native vendor is Anthropic because the brief format and personal memory are authored in Claude sessions. The gate applies on every consumption path: the foreground brief and any spawned child — spawn() tells the child which vendor will consume its brief, so a child spawned to a third-party CLI cannot retrieve more than the policy allows for that vendor. Set it in ~/.threadkeeper/.env (a real env override wins over .env):

THREADKEEPER_MEMORY_EGRESS=same-vendor

---

Core systems

Spawn — primary parallelism primitive

spawn(prompt, slim=True, role=..., visible=False, ...) launches a child Claude session via a claude -p subprocess. By default slim=True: the child loads only the thread-keeper MCP, no embeddings, no third-party servers. ~500 MB RSS versus ~1.3 GB for a full child. Heuristic for the parent: N≥2 modular independent units of ≥5 min each = spawn signal. Spawn also marks children with THREADKEEPER_SPAWNED_CHILD=1, so autonomous learning daemons cannot recursively start inside review forks.

A daemon in the foreground parent measures combined child RSS every 10 s; spawned children do not start their own ps polling loop, failed ps RSS samples keep the last-known value, and the liveness sweep covers every open task row so dead children stop counting against the cap. Admission control refuses a new spawn that would exceed THREADKEEPER_SPAWN_BUDGET_MB (3 GB default). Slim children that need semantic search delegate to the parent via search_via_parent — no per-child copy of the embedding model. Admission uses a SQLite BEGIN IMMEDIATE reservation: spawn() re-checks the budget and inserts the child task row with its RSS estimate before Popen, so two concurrent spawns cannot both squeeze through the cap.

The spawn wrapper also records each completed child's duration_s, tokens_in, tokens_out, tokens_total, and cost_usd when the underlying CLI emits a recognizable usage trailer. Optional daily ceilings THREADKEEPER_SPAWN_TOKEN_BUDGET and THREADKEEPER_SPAWN_COST_BUDGET_USD admission-deny new children once the recorded 24h spend reaches the configured limit; both default to 0 (disabled), so existing installs behave the same until a budget is set.

Visible (visible=True, Terminal.app) children persist pid=0, so the daemon resolves their live pid from the --session-id it carries in ps argv and measures the real RSS tree — they count their true memory, not the static estimate. A visible row whose session-id never resolves to a live process is reaped once it outlives THREADKEEPER_SPAWN_VISIBLE_TTL_S (1 h default; 0 disables), so an unresolvable row can't pin budget capacity forever.

The same daemon is also a wall-clock watchdog: a child that hangs while still alive — a wedged WebFetch/gh/git, an agent loop that never converges, a prompt that never arrives — would otherwise stall its loop's single-flight slot and burn tokens forever. Any child whose row outlives THREADKEEPER_SPAWN_MAX_RUNTIME_S (1 h default; 0 disables) is SIGTERM'd, then SIGKILL'd after THREADKEEPER_SPAWN_KILL_GRACE_S (10 s), and its row is closed with the timeout return_code 124 so the loop's single-flight releases. The watchdog then immediately starts a capped continuation retry: the new child receives the original assignment plus the previous task/cid/log and is instructed to inspect current workspace state, preserve completed work, repair partial work, and continue rather than restart blindly. THREADKEEPER_SPAWN_TIMEOUT_RETRY_LIMIT (default 3; 0 disables) bounds the retry chain, with THREADKEEPER_SPAWN_TIMEOUT_RETRY_DELAY_S available for a non-zero delay. Timed-out children are surfaced as tasks_timed_out in mp_dashboard and timed_out in agent_status.

tk-agent-status exposes autonomous learning loop status as structured JSON or compact text for external monitors:

tk-agent-status
tk-agent-status --json
tk-agent-status --cleanup-memory

apps/macos-agent-status/ contains a small macOS menu-bar app that polls this command every 15 seconds and shows every autonomous learning loop: enabled/off, running/idle/ready, last pass, backlog, and active child RSS when that loop has spawned a worker. PyPI wheels and sdists also bundle the same Swift source under threadkeeper/assets/macos-agent-status/, so a normal pipx/uv tool install does not need a git checkout for the widget to build. Active loops are sorted first (running, then ready), so background work stays at the top of the panel. tk-agent-status --cleanup-memory runs the safe cleanup path used by the widget: request server cache trims, apply the RSS guard, and remove orphan MCP server processes without killing active spawned child agents. The popover also has a power button that flips THREADKEEPER_DISABLE_BG_DAEMONS in ~/.threadkeeper/.env and requests a ThreadKeeper restart, so autonomous loops can be paused or re-enabled without opening Settings. The menu-bar status item is backed by AppKit NSStatusItem: it shows the black memorychip icon while idle, then swaps fixed-center, synchronized gear frames whenever running_loop_count reports at least one active autonomous loop. The status item is icon-only; loop counts live in the popover and tooltip. The app also has a Clean memory button, self-restarts when its own RSS crosses THREADKEEPER_MENUBAR_RESTART_RSS_MB (1024 MB default), requests macOS notification permission, and sends a notification when a newly completed autonomous child task produces a useful result in recent_results; the first poll only marks existing results as seen, so old completions do not spam notifications. Status polling and cleanup commands run off the main actor, so opening the popover does not wait for tk-agent-status --json. The header gear opens a separate Settings window for ~/.threadkeeper/.env: common knobs are grouped into guided controls, the raw .env remains editable for advanced values, three local presets can be saved and loaded, and Save & Restart writes the file then asks existing threadkeeper.server processes to exit so MCP hosts reconnect with the new configuration. Spawn CLI selectors collapse agy into canonical antigravity while keeping gemini as legacy, and model selectors use dropdowns with exact CLI model ids/labels instead of free-text fields. Probe backlog is due objective probes only, not every registered probe, so a healthy cooldown shows 0 due probes instead of looking stuck. On macOS, python -m threadkeeper.server automatically installs and launches it on MCP startup. The installed app records a source fingerprint, so package upgrades rebuild the helper even when an older bundle has a newer file timestamp, then restart any stale running menu-bar process. Set THREADKEEPER_MENUBAR_AUTO_LAUNCH=0 to disable that behavior.

Auto Update

The MCP server starts an auto-update daemon in foreground parent processes. By default it checks once per day (THREADKEEPER_AUTO_UPDATE_INTERVAL_S=86400):

• editable git checkout: skip if tracked files are dirty, otherwise fetch the tracked remote branch, fast-forward with git pull --ff-only, reinstall the editable package, and run the configured post-update setup check;
• installed package: run pip install --upgrade threadkeeper or threadkeeper[semantic] in the current interpreter environment, preserving semantic extras when they are already installed, but only after the candidate PyPI release's non-yanked files have PyPI Integrity API provenance from the expected GitHub Trusted Publisher (po4erk91/thread-keeper, publish.yml, environment pypi), then run the configured post-update setup check when the installed version changes.

Auto-update is standing consent for thread-keeper to fetch and run future maintainer code. A packaged update whose provenance is missing, whose publisher identity does not match policy, or whose attested subject digest does not match PyPI metadata is refused before pip runs and is recorded as auto_update_pass with mode=pip and refused. After a successful update, the daemon exits the current MCP process by default so the host can restart it on the new code. Before scheduling that exit, it imports threadkeeper.server in a subprocess; install/setup/import failures are recorded as auto_update_pass with restart=suppressed, and the current known-working process stays alive. Post-update setup defaults to THREADKEEPER_AUTO_UPDATE_SETUP=check, which runs thread-keeper-setup --dry-run only. It records setup=checked status=unchanged when configs already match and logs/records status=changes_pending if MCP registrations, hooks, or managed instruction blocks would be rewritten; it does not re-add config the user removed. Set THREADKEEPER_AUTO_UPDATE_SETUP=apply to give standing consent for auto-update to run the full setup writer after future successful updates, or skip to avoid even the dry-run check. Disable restart with THREADKEEPER_AUTO_UPDATE_RESTART=0, or disable the updater entirely with THREADKEEPER_AUTO_UPDATE_INTERVAL_S=0. The provenance gate is on by default; THREADKEEPER_AUTO_UPDATE_VERIFY_PROVENANCE=0 is a break-glass opt-out for private mirrors or disconnected installs. If a packaged release needs manual rollback, pin the previous version explicitly, for example pip install threadkeeper==<previous>. Each real check records an auto_update_pass event that appears in dashboard/status telemetry.

Skill Update

The MCP server also starts a skill updater in foreground parent processes. By default it checks twice per week (THREADKEEPER_SKILL_UPDATE_INTERVAL_S=302400):

• local root sync: scan every configured skill root, import the newest local copy of a skill into the primary ~/.claude/skills root, then mirror it back to ~/.codex/skills, Antigravity, ~/.agents/skills, extra roots, and the canonical ~/.threadkeeper/skills fallback;
• source-tracked updates: skills with .threadkeeper-skill-source.json, or skills whose name can be inferred from THREADKEEPER_SKILL_UPDATE_SOURCES, are compared with upstream GitHub directories and updated when the remote tree changes.

The pass is single-flight across live MCP servers and backs up replaced local skills under the thread-keeper state dir. If a source-tracked skill has local edits after the last applied upstream hash, the updater skips it instead of overwriting. Disable it with THREADKEEPER_SKILL_UPDATE_INTERVAL_S=0.

Manual fallback from a source checkout:

cd apps/macos-agent-status
./build.sh
open build/ThreadKeeperAgentStatus.app

Learning loops

Five loops turn raw agent dialog into a curated, multi-CLI-mirrored skill library — autonomously, without requiring agents to call note() / verbatim_user() / close_thread() on their own (audit shows agents focused on their primary task rarely do).

Pipeline at a glance:

   every CLI's transcripts
            │
            ▼  (ingest, every 30s — always-on)
   dialog_messages  ◄──────────────────────────────────────┐
            │                                              │
            ├────────► [1] auto_review on close_thread     │
            │              (agent triggers — rare)         │
            │                  │                           │
            ├────────► [2] shadow_review daemon            │
            │              (cron, every 15 min)            │
            │                  │                           │
            ├────────► [3] extract daemon                  │
            │              (cron, every 10 min)            │
            │                  │                           │
            │              extract_candidates              │
            │                  │                           │
            │                  ▼                           │
            │          [4] candidate_reviewer daemon       │
            │              (cron, every 1 h) ──────────────┤
            │                  │                           │
            ▼                  ▼                           │
         brief()    SKILL.md + lessons.md ─► skill_usage   │
            │              │          └─────► lesson_usage │
            │              ▼                  ▼            │
            │         (every configured       │            │
            │          skills/ root)          │            │
            │              │                  │            │
            │              └──────► [5] Curator daemon ───┘
            │                          (cron, every 7d)
            │                              │
            │                              ▼
            │                       REPORT-<date>.md
            ▼
   injected into every new session at SessionStart

Each loop in one row:

#	Loop	Default tick	Reads	Writes
1	auto_review on close_thread	on close_thread() for rich threads	the thread's notes	SKILL.md, lessons.md
2	shadow_review daemon	every 15 min (env knob)	recent dialog_messages window	SKILL.md, lessons.md
3	extract daemon	every 10 min (env knob)	recent dialog_messages window	extract_candidates pending queue
4	candidate-reviewer daemon	every 1 h (env knob)	pending candidates queue	SKILL.md (create/patch) / notes / verbatim / reject
5	Curator daemon	every 7 days (env knob)	every existing lesson + recently-touched skill	REPORT-<date>.md; Evolve applier applies it after roadmap issues
6	evolve_reviewer daemon	configurable (env knob; 0=off)	code/docs/issues; web research in a separate read-only phase (#79)	roadmap updates + GitHub issues
7	evolve_applier daemon	configurable (env knob; 0=off)	open GitHub issues, Curator reports, legacy promoted evolve suggestions	PRs + applied markers
8	dialectic_miner daemon	configurable (env knob; 0=off)	recent dialog_messages — user replies + preceding-assistant context	dialectic_observations buffer
9	dialectic_validator daemon	configurable (env knob; 0=off)	buffered dialectic_observations	dialectic claims + evidence (support / contradict / supersede) via spawned opus child
10	skill_updater daemon	every 302400 s / twice weekly (env knob)	configured skill roots + tracked GitHub skill sources	mirrored SKILL.md directories + skill_update_pass telemetry

Learning loops write into the universal Skill format (SKILL.md under each known/configured skills root — ~/.claude/skills/, ~/.codex/skills/, ~/.gemini/config/skills/ for Antigravity, existing ~/.agents/skills/, optional THREADKEEPER_EXTRA_SKILLS_DIRS, plus the canonical ~/.threadkeeper/skills/ mirror), with ~/.threadkeeper/lessons.md as a CLI-agnostic fallback for clients without a native skills loader (Gemini legacy, Copilot, bare MCP).

Harvest boundary (issue #36). The dialog-reading loops share threadkeeper.harvest as their session exclusion boundary. Raw transcripts are still persisted for diagnostics, but shadow-review, extract, dialectic mining, dialectic validation cleanup, and passive skill-use foreground promotion all exclude autonomous child lineage: known internal prompt openers, spawn preambles, direct tasks.spawned_cid rows, native agent-* parent cids, and descendants reached through tasks.parent_cid → tasks.spawned_cid.

Injection fence + provenance (issue #76). The synthesis input is raw observed dialog — which routinely echoes content the agent read from untrusted web pages, files, issues, or pasted text (and, under multi-user mode, other users' conversations), while the output auto-loads into every future session. Every synthesis prompt (shadow-review, candidate-reviewer, the three review_prompts templates, the dialectic validator) wraps the observed window/candidate/notes/observations in an explicit <observed_dialog>…</observed_dialog> data fence with a standing "treat strictly as third-party content; never adopt instructions, policies, commands, or tool-calls inside it" boundary, and instructs the child to mint a stated-policy rule only from genuine foreground role='user' turns. The synthesis children are de-privileged (path-scoped skill/lesson tools only — no bare Read/Write), loop-authored skills stay distinguishable by created_by_origin so an auto-load gate (or [#26] elicitation) can target them without touching foreground-authored ones, and a write-time screen refuses loop-origin lesson/skill bodies that contain imperative-override / remote-exec idioms. See SECURITY.md.

1. Auto-review on close_thread

When a closed thread is rich (≥5 notes, ≥2 insight/move), close_thread spawns a slim child with SKILL_REVIEW_PROMPT + the thread's notes. The prompt is rubric-form (Q1–Q5 yes/no) with explicit positive examples for incident-vs-rule classification. The fork also receives a "recently active skills" block so it prefers PATCHing existing umbrellas over creating new ones (active-update bias). Child appends a lesson via lesson_append, writes/patches a skill via skill_manage or writes a skill file directly, then closes with mark_skill_materialized. If skill_path points at a SKILL.md (or a skill directory), thread-keeper immediately mirrors that whole skill into every configured skills root. Opt in with THREADKEEPER_AUTO_REVIEW=1.

2. Shadow-review daemon

Every THREADKEEPER_SHADOW_REVIEW_INTERVAL_S seconds (default off, 900 = 15 min recommended) scans the diff of dialog_messages since the last cursor across all CLIs at once. The window filters autonomous child lineage (no self-pollution) and strips adapter [tool_result] / [tool_call] noise (the "clean context" rule). If ≥500 chars of meaningful signal remain, spawns a slim observer child that decides on class-level learning. It is single-flight across the shared DB: a non-blocking helpers.single_flight_lock("shadow-review") dispatch lock guards the running-child check and spawn, so if another MCP server is already in that critical section the daemon reports shadow_child_running ... (single-flight lock) and does not advance the cursor. If any shadow observer task is already running, the daemon also skips spawning another child and keeps the cursor unchanged. Shadow observer children are marked as spawned/background processes, so they cannot start their own shadow daemon even if a CLI drops the no-embeddings env. Idempotent through events.kind='shadow_review_pass'.

Before writing memory, the observer now checks existing lessons/skills and prefers patching broad skills. Shadow-origin lesson_append is a compact fallback only: oversized bodies are rejected, near-duplicate slugs are blocked, and semantic body matches are routed to the incumbent lesson or surfaced for curation instead of minting a sibling lesson.

3. Extract daemon

Every THREADKEEPER_EXTRACT_INTERVAL_S seconds (default off, 600 = 10 min recommended) scans recent dialog_messages with heuristic matchers: locale-aware "I want / next time / always" patterns, headers + insight markers, bullet regularities, and paraphrase clusters via cosine ≥ 0.80. Each match enqueues a row in extract_candidates.status='pending'. Same self-pollution filter as shadow_review (autonomous child lineage excluded) plus message-level noise filter (compaction summaries, SKILL.md injections, subagent role prompts, test-runner log dumps). The manual extract_recent() tool uses the configured sliding window directly; the daemon also keeps an extract_pass cursor and extends a pass back to the previous successful tick when THREADKEEPER_EXTRACT_INTERVAL_S is longer than THREADKEEPER_EXTRACT_WINDOW_MIN, so no dialog falls between ticks.

Where shadow extracts CLASS-LEVEL durable rules, extract harvests PER-INCIDENT decision-shaped utterances. Heuristic, not LLM — findings get refined by loop 4.

4. Candidate-reviewer daemon

Every THREADKEEPER_CANDIDATE_REVIEW_INTERVAL_S seconds (default off, 3600 = 1 h recommended) consumes the pending queue extract built up. Spawns a slim LLM child that decides per candidate or per coherent cluster:

• SKILL.create — class-level rule; merge 2-5 related candidates into one skill (active-update bias prefers PATCH over CREATE)
• SKILL.patch — refines a recently-active skill
• SKILL.write_file — adds references/<topic>.md under an existing umbrella
• NOTE — per-incident decision (requires thread_id)
• VERBATIM — user quote worth preserving in brief()
• REJECT — false positive that slipped past extract's filters

Hard limits: max 2 new skills per pass enforced inside skill_manage(action="create") for candidate-reviewer, shadow-review, and auto-review children; [PROTECTED] (pinned + foreground-authored) skills are off-limits. Closes the gap between heuristic harvest and SKILL.md materialization — previously pending candidates accumulated indefinitely waiting for an agent to call accept_candidate() manually. The loop is machine-wide single-flight: while one reviewer child is running, or while another process holds the shared dispatch lock, other foreground servers/ticks report candidate_review_running instead of spawning another child for the same queue. Before that lock, the pass also checks the last recorded candidate_review_pass high-water. A fresh MCP server restart, or a non-forced direct candidate_review_run(), returns not_due inside the configured interval and records that status without spawning; use candidate_review_run(force=True) for an immediate one-shot.

All spawning learning-loop daemons that enforce single-flight use the same non-blocking helpers.single_flight_lock() helper around the check-running-then-spawn section. The local fcntl.flock closes the same-host TOCTOU window; the tasks-table running-child check remains as the second layer for stale-pid cleanup and status visibility. That running-child check is keyed by each child's prompt prefix, so daemon prompts are composed from the same prefix constants their detectors query, with a consistency test guarding future prompt-opening edits. The helper is also used by the side-effecting auto-update, skill-update, and menu-bar autolaunch dispatch locks.

5. Autonomous Curator

Every THREADKEEPER_CURATOR_INTERVAL_S seconds (default off, 604800 = 7 days recommended) spawns a slim child that reviews the EXISTING lessons.md + lesson_usage + skill_usage inventory and writes ~/.threadkeeper/curator/REPORT-<isodate>.md with KEEP / PATCH / CONSOLIDATE / PRUNE recommendations. Pinned and foreground-authored entries are marked [PROTECTED] in the inventory so the curator never proposes destructive changes against them, and delete-class tools enforce the same boundary server-side. The pass is single-flight across processes — a non-blocking fcntl.flock pidfile (<db dir>/curator.lock) plus a running-children check serialize it, so multiple MCP server instances can't run overlapping (now destructive) passes against the same store. Before that lock, the pass also checks the last recorded curator_pass high-water, so fresh MCP server restarts and non-forced direct curator_review() calls return not_due inside the configured interval and record that status without spawning. A manual curator_review(force=True) bypasses the interval but still respects the lock.

Before spawning, the scheduler hashes the stable inventory state (lessons, lesson usage, active/stale skills, and concepts). If the hash matches the last recorded complete/endorsed curator pass, the wake-up records an unchanged_inventory no-op event and endorses the last report instead of asking another child to re-grade the same snapshot. curator_review_status() shows both the last endorsed inventory_sha256 and the current inventory hash so operators can tell whether the store is quiescent.

Curator applies its own PATCH / PRUNE / CONSOLIDATE directly by default (it writes the REPORT first, then mutates — lesson_remove is in its toolset so it can actually prune and consolidate duplicate lessons). Set THREADKEEPER_CURATOR_DESTRUCTIVE=0 for advisory REPORT-only. It never touches [PROTECTED] / foreground / user / pinned / validated entries. Lessons are stamped with an explicit origin=<THREADKEEPER_WRITE_ORIGIN> marker when appended; missing, legacy, or unknown lesson provenance is protected by default. lesson_remove and skill_manage(action='delete') refuse protected foreground/unknown-origin entries unless force=True is called from a foreground writer; curator/spawned children cannot elevate themselves with force. Before a destructive child is spawned, thread-keeper writes a recoverable snapshot under <reports_dir>/snapshots/<pass-id>/ (default ~/.threadkeeper/curator/snapshots/<pass-id>/). The snapshot contains lessons.md, copied in-scope skill dirs, a manifest.json, and per-action tombstones for curator prunes/deletes. Retention is bounded by THREADKEEPER_CURATOR_SNAPSHOT_RETENTION (default 10, current pass always kept). Use curator_restore(pass_id, lesson_slug="...") or curator_restore(pass_id, skill_name="...") to restore an item from a snapshot. Before lesson_remove or skill_manage(action='delete') removes anything, it also writes a recovery artifact under <db dir>/curator/trash/: lessons store the exact sentinel section plus usage row, and skills store the full skill directory plus usage row. Restore trash artifacts with lesson_restore(slug=...) or skill_manage(action='restore', name=...). Trash retention is bounded by THREADKEEPER_CURATOR_TRASH_TTL_DAYS (30 days by default) and swept on new trash writes. Advisory mode does not write snapshots. The existing Evolve applier is also the Curator apply worker: after the roadmap issue queue is empty, it looks for the latest complete Curator report (CURATOR_PASS_COMPLETE) that has not been marked applied, then spawns an evolve_applier child to apply only safe, still-current memory maintenance through lesson_append / lesson_remove / skill_manage / concept_manage. It never touches [PROTECTED], foreground/user, pinned, or validated entries. Only after the child finishes does it call evolve_mark_curator_report_applied(...), which prevents replaying the same report.

The shared lesson file has its own write serialization: lesson_append, lesson_remove, and lesson_restore hold a blocking fcntl.flock on lessons.md.lock around file creation/read/mutate/write, so foreground calls and learning-loop children cannot last-writer-win over each other's sections.

Lesson access is tracked the same way skill access is: lesson_list increments lesson_usage.view_count for displayed rows and lesson_get increments lesson_usage.use_count for the returned lesson. Curator dry runs include a ranked STALE LESSONS (dry-run decay ranking) section computed as access_frequency × exp(-days_since_access / tau), filtered to unprotected lessons with no recent access and low pull-count. That decay list is advisory only; it never becomes an automatic lesson_remove path by itself, and pinned or validated lessons are excluded. A lesson is unprotected only when its explicit origin marker is a known loop origin; foreground, legacy, empty, and unknown-origin lessons fail closed.

The curator also audits the concepts store (abstract regularities triangulated across paraphrase runs). Concepts are no longer write-only: register_concept and accepted concept candidates dedup on write — a re-surfaced equivalent invariant (description cosine ≥ 0.85) corroborates the existing concept, bumping its last_evidence_at and raising confidence, instead of inserting a near-duplicate — so last_evidence_at is a real corroboration-recency signal the brief orders on. The curator's CONSOLIDATE_CONCEPT / PRUNE_CONCEPT / confidence-review recommendations are applied via concept_manage (remove / consolidate / set_confidence). Concepts are all system-generated, so concept_manage needs no force guard.

Curator can also feed the roadmap loop upstream: when a skill or lesson exposes an important way to improve thread-keeper itself, the curator child may call evolve_format(...) and add an EVOLVE_CANDIDATE: line to its report. Evolve reviewer then audits that candidate and turns it into a GitHub issue when it is worth doing.

6. Evolve reviewer/applier — roadmap evolution loop

The Evolve reviewer is thread-keeper's upstream product/engineering auditor. On its interval it audits thread-keeper itself for security/privacy risks, memory leaks, runaway daemons, cost waste, reliability gaps, optimizations, and new ideas from current agent/MCP/memory tooling research. It does not implement code. Its durable outputs are updates to docs/ROADMAP.md and GitHub issues with problem statement, proposed direction, acceptance criteria, test/docs impact, and research sources when applicable. Legacy evolve_format(...) suggestions are still included as audit input, but durable implementation work should become GitHub issues. Before filing new issues, the privileged audit phase routes candidates through evolve_issue_create(...), which checks a paginated oldest-first GitHub REST view of open and closed issues, treats closed not_planned issues as duplicate/rejected work, and records reviewer-filed issue fingerprints in the local evolve_issues ledger. Duplicate candidates are skipped with telemetry, so deduplication is not limited to the newest 50 open issues or to the current reviewer pass.

To avoid completing the lethal trifecta — private-data access + untrusted web content + exfiltration — inside one privileged child (#79), the reviewer runs as two alternating phases, never co-granting web research and shell/bypassPermissions to the same child:

• research phase — a read-only child with WebSearch/WebFetch and read-only repo reads but no shell, no bypassPermissions, and no GitHub access. It distills external findings into a digest file under ~/.threadkeeper/evolve-research/. With no Bash/gh/network-write tool it has no exfiltration channel, so the untrusted pages it reads cannot act.
• audit phase — the privileged child (bypassPermissions + Bash/Edit/ Write) that audits the repo, opens the docs/ROADMAP.md PR, and creates or updates GitHub issues. It holds no web tools; it consumes the research digest as an explicit, fenced data block it must never read as instructions (mirroring #76's fencing, applied to the web source).

A full research → audit cycle therefore spans two due passes.

Before an audit child can open a roadmap-doc PR, the parent preflights open PRs with gh pr list --json ... files and reports any automation-owned PR already touching docs/ROADMAP.md. The child must append to that PR or skip when no change is needed; otherwise it uses the deterministic daily docs/roadmap-audit-YYYY-MM-DD branch and reuses an existing local/remote branch with that name instead of minting overlapping roadmap PRs.

The Evolve applier is the downstream implementer. evolve_apply_roadmap_issue() picks one open GitHub issue at a time (roadmap label first, then FIFO), but the automatic pass first scans already-open same-repo applier PRs for GitHub merge conflicts. A conflicted roadmap/… or evolve/… PR is repaired before any new issue/report/evolve work is started; if the PR sweep itself cannot read GitHub state, the pass fails closed instead of taking fresh work blind. The conflict-repair child checks out the existing PR branch, merges the current base branch, resolves conflicts, runs the full suite, and pushes back to the same branch. It then waits for GitHub checks on the pushed PR head and runs gh pr merge --squash --delete-branch, so GitHub lands the repaired PR into main through branch protection rather than a raw local git push origin main. The roadmap issue child skips issues carrying denylisted human-gate labels, skips issues with an active Evolve claim comment, posts its own claim comment before spawning, and advances to the next issue when an issue-local dispatch failure prevents startup. It implements exactly that issue, runs the full suite, opens a PR whose body includes Closes #N, and only then calls evolve_mark_roadmap_issue_applied(issue_number, pr_url). It never commits or pushes to main, and it never marks an issue applied without a real PR URL. If that PR is later closed without merging, the parent reconciles the marker against GitHub PR state, records roadmap_issue_requeued, and lets the issue flow through the normal retry backoff/dead-letter gates again. A manual evolve_apply_roadmap_issue(issue_number=N) remains exact: it reports why that issue cannot start instead of silently switching to another issue. The queue fetch uses paginated GitHub REST reads in oldest-created order, then applies the documented roadmap/FIFO sort locally. A generous local candidate window is retained as a runaway guard; if it ever truncates, the applier logs how many open issues were outside the window. All roadmap-automation GitHub calls share a local github_rate_budget ledger: the applier's parent-side gh calls and the PATH-prepended child gh wrapper honor the same per-account cooldown. Included REST response headers update remaining/reset values; primary 403s cool down until reset (bounded), and secondary-rate-limit / Retry-After responses use bounded exponential backoff. agent_status / tk-agent-status and evolve_apply_status() show the current remaining count or cooldown window so operators can see when GitHub is throttling the roadmap loop.

Before any PR-producing reviewer/audit or applier child is spawned, the parent checks the target checkout with git status --porcelain --untracked-files=no. Tracked-file WIP records skipped_dirty_worktree and no child is dispatched; untracked scratch files do not block. The child prompts also fetch the configured base branch and create feature branches from origin/main (or the configured THREADKEEPER_EVOLVE_REPO_BRANCH), not from whatever HEAD the daemon happens to have checked out. A shared git-writer running-task check prevents the privileged reviewer audit and code/PR applier from overlapping in the same checkout. If a killed conflict-repair child leaves an unresolved merge in the default auto-managed checkout, the next code-applying pass can recover it — but only when the current branch is roadmap/…/evolve/… and GitHub confirms that exact PR is already merged. Before returning the managed tree to fresh origin/<THREADKEEPER_EVOLVE_REPO_BRANCH>, thread-keeper archives the tracked diff under ~/.threadkeeper/evolve-recovery/stale-merge-pr-*.patch and records recovered_stale_merge telemetry. Open, closed-unmerged, missing, or unreadable PR state remains fail-closed. An explicit THREADKEEPER_EVOLVE_REPO_ROOT is never auto-reset.

Skip-label gate. Autonomous issue pickup refuses issues with labels listed in THREADKEEPER_EVOLVE_APPLY_SKIP_LABELS (default blocked,needs-design,wontfix,question,discussion,help wanted). These labels mean the issue needs human design, discussion, or intervention before a permission-bypassing implementer should try it. Queue mode excludes those issues and records roadmap_issue_skipped telemetry; exact mode returns skipped: label X for the named issue rather than selecting a different one. Set the knob to another comma-separated list, or to off, to override the default.

Author-trust gate (this repo is public). Any GitHub account can open an issue, and an open issue's body is injected into the permission-bypassing implementer child — so autonomous pickup is gated on the issue author's GitHub association. Only issues whose authorAssociation is in THREADKEEPER_EVOLVE_TRUSTED_AUTHOR_ASSOCIATIONS (default OWNER,MEMBER,COLLABORATOR) are auto-drained; everything else is skipped until a human promotes it — by applying a label listed in THREADKEEPER_EVOLVE_TRUST_LABELS (empty by default; on a public repo only collaborators can label, so a trust label is itself a maintainer endorsement), or by naming the exact issue number via evolve_apply_roadmap_issue(issue_number=N), which bypasses the gate as explicit promotion. This removes the untrusted input at the boundary and complements the in-prompt data-fencing of #22/#76. The public claim comment also carries only an opaque per-host token (a 6-char hash of the hostname), never the raw hostname/PID/git-rev; the full host identity is recorded in the local event log for multi-host triage.

Privilege + public-body guard (#22). Stored evolve suggestions and external GitHub issue bodies are wrapped in explicit data fences before a privileged child sees them. The exposed spawn() tool refuses permission_mode="bypassPermissions" unless the request comes from the evolve daemon role/write-origin pairs (evolve_reviewer/evolve, evolve_applier/evolve_apply) or the operator explicitly opts in with THREADKEEPER_ALLOW_BYPASS_PERMISSIONS_SPAWN=1. Privileged evolve children also get a PATH-prepended gh wrapper that scrubs gh issue create, gh issue comment, and gh pr create bodies before the real GitHub CLI sees them: home-directory paths and common token shapes are redacted, and a body is refused if a known unsafe pattern remains.

Fallback/manual paths remain:

• evolve_apply_conflicted_pr(pr_number=0) repairs the oldest conflicted same-repo applier PR, or a specific conflicted PR when numbered.
• evolve_apply_curator_report(report_path="") applies safe Curator memory maintenance when no roadmap issue is being drained.
• evolve_apply(evolve_id) still implements legacy promoted evolve_format(...) suggestions behind a PR and calls evolve_mark_applied(evolve_id, pr_url).

Set THREADKEEPER_EVOLVE_REVIEW_INTERVAL_S>0 to run periodic audit/research passes and THREADKEEPER_EVOLVE_APPLY_INTERVAL_S>0 to drain one issue per pass. Pin the agent/model with THREADKEEPER_SPAWN__LOOP__EVOLVE_APPLIER / THREADKEEPER_SPAWN__MODEL__EVOLVE_APPLIER. Single-flight (one applier child at a time, enforced by a short dispatch file lock plus running-task detection) and the shared git-writer guard keep code edits and roadmap PR writes from colliding. Reviewer roadmap-doc PRs also use a parent open-PR preflight and a daily deterministic docs/roadmap-audit-YYYY-MM-DD branch so repeated audit passes update or skip the existing roadmap PR rather than opening a second one. Automatic apply passes respect the configured interval so multiple foreground MCP server startups do not repeatedly spawn workers for the same open issue. Manual tools such as evolve_apply_conflicted_pr() and evolve_apply_roadmap_issue() dispatch immediately. If no conflicted applier PR or roadmap issue is startable, the pass falls back to Curator reports and then legacy promoted evolve_format(...) suggestions.

Honest take

What works without agent cooperation (passive, opt-in via env):

• Loop 2 (shadow), 3 (extract), 4 (candidate-reviewer), 5 (curator) — all run from the parent process, never require note() or close_thread() from the agent

What depends on the agent calling tools explicitly:

• Loop 1 (auto-review on close_thread) — only fires if the agent closes threads, which the audit shows agents focused on coding tasks rarely do
• Manual skill_record(outcome='wrong') — strongest feedback signal to the Curator, but agents need to remember to flag bad skills

The whole point of having five loops (not one) is graceful degradation: even when agents don't actively contribute, loops 2-5 keep the library growing from passive observation of the dialog stream.

Dialectic user model

A model of you, accumulated as you use the agent. dialectic_claim, dialectic_evidence (support / contradict), dialectic_synthesis, dialectic_supersede. Honcho-inspired weighted, smoothed ratio (Σw_support − Σw_contradict) / (Σw_support + Σw_contradict + 3) → low / medium / high / disputed confidence. Grouped by domain (style, values, workflow, ...) in brief().

Claims are bi-temporal: created_at records ingestion time, while valid_from / valid_to record when a preference or belief applies. New claims start at valid_from=created_at; dialectic_supersede preserves the old claim and its evidence but closes the old valid-time interval at the new claim's valid_from. Normal brief() / synthesis output remains the current active slice; dialectic_review(as_of=...) and dialectic_synthesis(include_history=True) expose past validity intervals.

Source-based evidence discount. Each evidence row's effective weight is base_weight × discount(WRITE_ORIGIN). Foreground (direct user / human signal) = 1.0. shadow_review / background_review / candidate_review / curator review-forks = 0.5. Structural defence against self-confirmation loops: a claim that surfaces in brief() and then gets "confirmed" by a review-fork reading the same dialog can't ride that internal evidence all the way to high confidence — internal evidence buys half as much.

Discrete tier on each claim — hypothesis → observed → validated (plus disputed). Independent of the continuous confidence band; tier is the action-gating signal:

• validated → agent applies by default (★ in brief)
• observed → agent references and may mention the assumption (· in brief)
• hypothesis → active probe; surfaces in a separate currently_testing block so the agent watches the next user moves through that lens

Transitions are discrete events (tier_promoted / tier_demoted in the events table) with timestamps for an auditable trail of when each claim earned trust. Thresholds:

• hypothesis → observed: w_support ≥ 2.0 (claim has real backing)
• observed → validated: w_support ≥ 4.0 and no contradict in 14 days
• validated → observed: any recent contradict (demote on user pushback)
• any → disputed: w_contradict > w_support
• disputed → hypothesis: support overtakes contradict (recovery path)

i18n bundle

All multilingual regex and prompt fragments live in threadkeeper/i18n.py — the rest of the codebase stays English-only. Currently ships ten locales: English, Mandarin Chinese, Hindi, Spanish, Portuguese, French, German, Arabic, Russian, Japanese (~82 % of the world's speakers).

Adding a new language is a two-file PR — see CONTRIBUTING.md.

---

Configuration

The most-used env knobs (full list in threadkeeper/config.py):

Knob	Default	Purpose
THREADKEEPER_DB	~/.threadkeeper/db.sqlite	SQLite file
THREADKEEPER_TASK_LOG_DIR	~/.threadkeeper/tasks	owner-only task spool for spawn logs, stdin prompts, command scripts, and small runtime logs
THREADKEEPER_RETENTION_INTERVAL_S	0 (off)	SQLite retention/compaction daemon tick; 0 disables the daemon
THREADKEEPER_DIALOG_RETENTION_DAYS	0	prune aged dialog_messages (their FTS entries follow via trigger) plus dialog_vec sidecars; 0 keeps forever
THREADKEEPER_TASK_RETENTION_DAYS	30	prune completed tasks rows older than this many days; 0 keeps forever
THREADKEEPER_SIGNAL_RETENTION_DAYS	0	prune handled old signals plus aged search_request/search_response; 0 keeps forever
THREADKEEPER_EVENTS_RETENTION_DAYS	0	prune old events on the retention pass; 0 keeps forever
THREADKEEPER_PROBE_RESULT_RETENTION_DAYS	0	prune old probe_results and refresh reliability aggregates; 0 keeps forever
THREADKEEPER_RETENTION_WAL_CHECKPOINT	false	run PRAGMA wal_checkpoint(TRUNCATE) during retention passes
THREADKEEPER_RETENTION_VACUUM_AFTER_ROWS	0	run VACUUM after a pass deletes at least this many rows; 0 disables VACUUM
THREADKEEPER_MEMORY_EGRESS	all	cross-provider scope for personal-class memory (verbatim quotes + dialectic user-model) in brief(). all = current behavior, egress to whichever vendor backs the consuming CLI. same-vendor = personal renders only for Claude/Anthropic, omitted for OpenAI/Google/Microsoft CLIs. work-only = personal never rendered, any vendor. See Memory egress
THREADKEEPER_AUTO_REVIEW	"" (off)	auto-review on close_thread
THREADKEEPER_AUTO_UPDATE_INTERVAL_S	86400	MCP self-update check interval; 0 disables
THREADKEEPER_AUTO_UPDATE_RESTART	"1"	exit MCP process after an update passes setup/import smoke checks so the host restarts on new code
THREADKEEPER_AUTO_UPDATE_TIMEOUT_S	600	max seconds for git/pip update commands
THREADKEEPER_AUTO_UPDATE_SETUP	check	post-update setup mode: check runs thread-keeper-setup --dry-run and logs pending CLI config rewrites without applying them; apply gives standing consent to rewrite MCP/hooks/instruction config after updates; skip disables the setup step
THREADKEEPER_AUTO_UPDATE_VERIFY_PROVENANCE	true	require PyPI Integrity API provenance before packaged pip self-upgrades
THREADKEEPER_AUTO_UPDATE_PYPI_BASE_URL	https://pypi.org	PyPI base URL used for JSON metadata and Integrity API checks
THREADKEEPER_AUTO_UPDATE_EXPECTED_PUBLISHER_REPOSITORY	po4erk91/thread-keeper	expected GitHub Trusted Publisher repository for packaged self-upgrades
THREADKEEPER_AUTO_UPDATE_EXPECTED_PUBLISHER_WORKFLOW	publish.yml	expected GitHub Actions workflow filename in PyPI provenance
THREADKEEPER_AUTO_UPDATE_EXPECTED_PUBLISHER_ENVIRONMENT	pypi	expected GitHub Actions environment in PyPI provenance
THREADKEEPER_SKILL_UPDATE_INTERVAL_S	302400	installed-skill update/mirror interval; 0 disables
THREADKEEPER_SKILL_UPDATE_TIMEOUT_S	300	max seconds for upstream skill source downloads
THREADKEEPER_SKILL_UPDATE_SOURCES	openai/skills@main:skills/.curated	comma-separated GitHub source roots (owner/repo@ref:path) used to infer upstream skill updates
THREADKEEPER_SKILL_UPDATE_INFER_SOURCES	true	infer upstream source by skill name from configured source roots
THREADKEEPER_SKILL_UPDATE_ALLOW_UNTRACKED_OVERWRITE	false	allow overwriting inferred untracked local skill copies; default false only adopts exact matches
THREADKEEPER_CONFIG_WATCH_INTERVAL_S	2	hot-config reload: poll the universal ~/.threadkeeper/.env (every host) + the host CLI's env-block file and re-apply changed env knobs in-process (no CLI restart); 0 disables
THREADKEEPER_CONFIG_WATCH_PATH	""	escape hatch: pin ONE settings file to watch (single-file mode); when unset, hybrid mode watches .env + the CLI file resolved via host identity
THREADKEEPER_SHADOW_REVIEW_INTERVAL_S	0 (off)	shadow daemon tick (s)
THREADKEEPER_SHADOW_REVIEW_WINDOW_S	900	sliding window for shadow scan (s)
THREADKEEPER_EXTRACT_INTERVAL_S	0 (off)	extract daemon tick (s); 600 = 10 min recommended; if this exceeds the base window, the daemon extends from the previous successful extract_pass cursor so ticks do not leave gaps
THREADKEEPER_EXTRACT_WINDOW_MIN	30	base sliding dialog window per extract pass (min); daemon runs may scan farther back only to cover an interval/window gap
THREADKEEPER_CANDIDATE_REVIEW_INTERVAL_S	0 (off)	candidate-reviewer daemon tick (s), restart-throttled by the last candidate_review_pass; 3600 = 1h recommended
THREADKEEPER_CANDIDATE_REVIEW_MIN	3	min pending candidates before reviewer engages
THREADKEEPER_LEARNING_LOOP_SKILL_CREATE_LIMIT	2	max new skills one autonomous learning-loop child (candidate_review, shadow_review, or background_review) may create in its session; foreground creation is unaffected
THREADKEEPER_CURATOR_INTERVAL_S	0 (off)	curator daemon tick (s), restart-throttled by the last curator_pass; 604800 = 7d recommended
THREADKEEPER_CURATOR_MIN_LESSONS	3	min lessons before curator engages
THREADKEEPER_CURATOR_DESTRUCTIVE	1 (on)	curator child writes its REPORT then applies its own PATCH/PRUNE/CONSOLIDATE directly (incl. lesson_remove for prune/consolidate); set 0 for advisory REPORT-only. [PROTECTED] entries are refused server-side
THREADKEEPER_CURATOR_SNAPSHOT_RETENTION	10	number of destructive curator pre-mutation snapshots to retain under <reports_dir>/snapshots; current pass is always retained
THREADKEEPER_CURATOR_TRASH_TTL_DAYS	30	days to retain recovery artifacts under <db dir>/curator/trash for lesson_remove and skill_manage(action='delete'); expired artifacts are swept on new trash writes
THREADKEEPER_PROBE_INTERVAL_S	0 (off)	probe daemon tick (s); 1800 = 30 min recommended so finished probe answers are graded promptly
THREADKEEPER_PROBE_COOLDOWN_S	604800	per-category probe cooldown; 86400 = 1d recommended for active reliability tracking
THREADKEEPER_SPAWN_BUDGET_MB	3072	combined child RSS cap (MB); 0 disables
THREADKEEPER_ALLOW_BYPASS_PERMISSIONS_SPAWN	"" (off)	explicit override that lets ordinary spawn() calls request permission_mode="bypassPermissions"; default off means only evolve daemon role/write-origin pairs can use the dangerous mode
THREADKEEPER_SPAWN_TOKEN_BUDGET	0	recorded 24h spawned-child token ceiling; 0 disables
THREADKEEPER_SPAWN_COST_BUDGET_USD	0	recorded 24h spawned-child dollar ceiling; 0 disables
THREADKEEPER_SPAWN_MAX_RUNTIME_S	3600	wall-clock lifetime cap (s) for a spawned child; over-cap live children are SIGTERM→SIGKILL'd and closed with return_code 124; 0 disables
THREADKEEPER_SPAWN_KILL_GRACE_S	10	grace between SIGTERM and SIGKILL when the watchdog kills a timed-out child
THREADKEEPER_SPAWN_TIMEOUT_RETRY_LIMIT	3	immediate continuation retries after a watchdog kill; 0 disables
THREADKEEPER_SPAWN_TIMEOUT_RETRY_DELAY_S	0	delay before a watchdog continuation retry
THREADKEEPER_MENUBAR_AUTO_LAUNCH	true	macOS: auto install/launch status menu-bar app on MCP startup
THREADKEEPER_MENUBAR_RESTART_RSS_MB	1024	macOS widget self-restart RSS threshold; 0 disables
THREADKEEPER_MEMORY_GUARD_POLL_S	30	server RSS guard tick (s); 0 disables
THREADKEEPER_MEMORY_GUARD_WARN_MB	1536	notify/log when a server crosses this RSS
THREADKEEPER_MEMORY_GUARD_KILL_MB	3072	SIGTERM server above this RSS; 0 disables killing
THREADKEEPER_MEMORY_GUARD_AGG_WARN_MB	2048	notify/request trim when all server RSS crosses this
THREADKEEPER_MEMORY_GUARD_AGG_KILL_MB	3072	under aggregate pressure, retire stale idle servers
THREADKEEPER_MEMORY_GUARD_RECLAIM_MB	1024	local RSS floor before warn-triggered self trim
THREADKEEPER_MEMORY_GUARD_EMBED_HOT_S	300	don't unload an embedding model used within this window (an active ingester reloads it seconds later, making the trim net-negative); ineffective reclaims also back off exponentially (30m→4h); 0 disables the hot guard
THREADKEEPER_MEMORY_GUARD_TARGET_SERVERS	1	aggregate-pressure target after retiring stale idle servers
THREADKEEPER_MEMORY_GUARD_RETIRE_IDLE_S	900	heartbeat age before a non-self server is retireable
THREADKEEPER_MEMORY_GUARD_RETIRE_LIVE	"" (off)	allow retiring parent-alive MCP servers; off protects live clients
THREADKEEPER_MEMORY_GUARD_NOTIFY	"1"	send macOS desktop notification when possible
THREADKEEPER_INGEST_INTERVAL_S	3	transcript ingest tick (s)
THREADKEEPER_REDACT_DIALOG_SECRETS	true	scrub common credential-shaped values before transcript text is persisted to dialog_messages / dialog_fts; set 0 only for rare local debugging where raw transcript fidelity is more important than durable secret protection; the v2 schema migration also scrubs legacy pre-redaction rows in place
THREADKEEPER_NO_EMBEDDINGS	""	force-disable the embedding model (FTS5 + delegate only)
THREADKEEPER_EMBED_BACKEND	onnx	embedding runtime: onnx (fastembed, no PyTorch) or sentence-transformers (legacy fallback)
THREADKEEPER_EMBED_MODEL	paraphrase-multilingual-MiniLM-L12-v2	384-dim cross-lingual embedding model
THREADKEEPER_SPAWNED_CHILD	""	spawn-internal marker; disables autonomous daemons in children
THREADKEEPER_SKILL_NUDGE_INTERVAL	10	events between skill_hint nudges
THREADKEEPER_DIALECTIC_MINE_INTERVAL_S	0 (off)	dialectic_miner daemon tick (s); 0 disables mechanical observation capture
THREADKEEPER_DIALECTIC_VALIDATE_INTERVAL_S	0 (off)	dialectic_validator daemon tick (s); 0 disables LLM-driven claim synthesis
THREADKEEPER_DIALECTIC_VALIDATE_MIN	5	min buffered observations before validator engages
THREADKEEPER_DIALECTIC_VALIDATE_BATCH_SIZE	50	max observations sent to one validator child; prevents oversized prompts and drains large queues incrementally
THREADKEEPER_EVOLVE_REVIEW_INTERVAL_S	0 (off)	evolve-reviewer daemon tick (s); audits thread-keeper for safety/leaks/optimization/new ideas, updates roadmap/issues, and includes legacy evolve suggestions as input. Runs as two alternating phases — read-only web research, then a privileged web-free audit that consumes the fenced research digest (#79) — so a full cycle spans two ticks
THREADKEEPER_EVOLVE_APPLY_INTERVAL_S	0 (off)	evolve-applier daemon tick (s); implements one open GitHub issue at a time, then falls back to Curator reports and promoted legacy evolve suggestions. Empty checks are throttled between intervals; actionable work and manual apply tools still dispatch
THREADKEEPER_EVOLVE_REPO_ROOT	(auto)	absolute path to the thread-keeper git checkout the evolve reviewer/applier branch, test, and open PRs against. When empty, the repo is resolved automatically: the package's parent dir for an editable install.sh, else a managed checkout under the DB dir that is auto-cloned on first use. Set this to pin an explicit checkout
THREADKEEPER_EVOLVE_AUTO_CLONE	true	auto-provision (git clone + .venv with [semantic,dev]) a managed checkout when installed without a source tree (PyPI/site-packages), so the evolve loops work by default. Set 0/false to disable — then a non-checkout install requires an editable install or an explicit EVOLVE_REPO_ROOT, otherwise the loops return ERR evolve_repo_unavailable
THREADKEEPER_EVOLVE_REPO_URL	upstream repo	git URL the managed checkout is cloned from
THREADKEEPER_EVOLVE_REPO_BRANCH	main	branch the managed checkout tracks
THREADKEEPER_EVOLVE_APPLY_SKIP_LABELS	blocked,needs-design,wontfix,question,discussion,help wanted	comma-separated labels that exclude GitHub issues from autonomous Evolve applier pickup. Exact-number apply returns skipped: label X; set to off to clear
THREADKEEPER_EVOLVE_TRUSTED_AUTHOR_ASSOCIATIONS	OWNER,MEMBER,COLLABORATOR	comma-separated GitHub author associations eligible for autonomous issue pickup on this public repo; issues from other authors are skipped unless promoted (trust label or exact-number invocation)
THREADKEEPER_EVOLVE_TRUST_LABELS	(empty)	comma-separated labels that promote an untrusted-author issue into the autonomous queue; on a public repo only collaborators can apply labels, so a trust label is a maintainer endorsement
THREADKEEPER_ROADMAP_ISSUE_MAX_ATTEMPTS	3	poison-issue dead-letter cap: after this many implementer spawns for a roadmap issue with no resulting PR, the issue gets a blocked label + one summary comment and is excluded from the auto-drain until a human intervenes. A manual evolve_apply_roadmap_issue(issue_number=N) bypasses the cap, but the default skip-label gate still refuses the blocked label until it is removed or reconfigured
THREADKEEPER_ROADMAP_ISSUE_BACKOFF_BASE_S	172800 (2d)	base failure-backoff window for a roadmap issue; doubles per attempt (base * 2^(attempts-1), capped at 30d). Defers re-selection of a repeatedly-aborting issue beyond the fixed 24h claim TTL
THREADKEEPER_DIALECTIC_MAX_NEW_CLAIMS	3	max new dialectic claims the validator may create per pass
THREADKEEPER_DAEMON_HOST	0 (off)	Phase 1 rollout flag (dark by default; no CLI config change). 1 = one headless host (python -m threadkeeper.host) owns the background loops + the warm embedding model + the embed socket, and per-session servers go thin (no daemons, no ONNX). See Embeddings below
THREADKEEPER_ROLE	server	process role: server (default; per-session MCP server) or host. Set to host only by python -m threadkeeper.host — do not set this by hand
THREADKEEPER_HOST_SOCK	(auto)	embed-only unix socket the thin servers dial and the host binds; empty resolves to <db dir>/host.sock
THREADKEEPER_HOST_HEARTBEAT_TTL_S	120	host liveness window (s): how stale the host's presence heartbeat may get before memory_guard/a thin server treats it as dead and spawns a replacement
THREADKEEPER_THIN_EMBED_FALLBACK	fts	how a thin server embeds a query when the host is unreachable: fts (default) falls back to FTS-only search; local lazily loads the ONNX model in-process instead

Persist them in ~/.threadkeeper/.env (copy from .env.example) — one file, read via pydantic-settings; real environment variables still override it. On macOS, the menu-bar app's gear button can edit the same file visually, save up to three local presets, and request a ThreadKeeper restart after saving. At startup and hot-reload, unknown THREADKEEPER_* keys present in the process environment are logged as warnings so mistyped host env-block overrides do not fail silently. Hot-config reload is implemented (shipped in #2, generalized cross-CLI in #133): the config_watcher daemon re-applies changed THREADKEEPER_* knobs in-process within ~2 s, with no CLI restart. It watches two layers — the universal ~/.threadkeeper/.env (read by every host's Settings(), so an edit hot-reloads on all seven CLIs and stays precedence-correct: real env > .env > default) and the host CLI's own env-block file (Claude Code → ~/.claude/settings.json, resolved via host identity; a key a higher scope pinned at spawn is never overridden by the lower-priority user file). Toggle via THREADKEEPER_CONFIG_WATCH_INTERVAL_S (above; 0 disables) and inspect with config_watch_status(), which reports both watched files.

Per-loop agent dispatch

By default every learning-loop spawn runs through the same CLI that hosts thread-keeper — Opus-session ⇒ Opus spawn, Codex-session ⇒ Codex spawn, etc. Detection: process-tree walk at startup, cached for the server lifetime. The MCP tool spawn_status() shows the live resolution table.

Override per role in ~/.threadkeeper/.env (there is no longer a spawn.toml — all config lives in the one .env). Spawn routing uses nested __ keys; dict keys are lowercased:

# default agent for roles with no explicit pin ("" / unset = use the active CLI)
THREADKEEPER_SPAWN__DEFAULT=claude
# per-role CLI:  THREADKEEPER_SPAWN__LOOP__<ROLE>=<cli>
# supported CLI keys: claude, codex, antigravity (agy executable), gemini (legacy), copilot
THREADKEEPER_SPAWN__LOOP__SHADOW_OBSERVER=claude   # heaviest reasoning → keep on Claude
THREADKEEPER_SPAWN__LOOP__CURATOR=codex            # weekly audit → Codex is fine
THREADKEEPER_SPAWN__LOOP__CANDIDATE_REVIEWER=auto  # "auto" = follow active CLI
# model pin per CLI or per role:  THREADKEEPER_SPAWN__MODEL__<KEY>=<model>
THREADKEEPER_SPAWN__MODEL__CLAUDE=opus
THREADKEEPER_SPAWN__MODEL__CODEX=gpt-5.5
THREADKEEPER_SPAWN__MODEL__AGY="Gemini 3.1 Pro (High)"
THREADKEEPER_SPAWN__MODEL__GEMINI=gemini-3.1-pro-preview
THREADKEEPER_SPAWN__MODEL__DIALECTIC_VALIDATOR=opus

Resolution per role: SPAWN__LOOP__<role> → SPAWN__DEFAULT → active CLI → claude; "auto" (or unset) defers to the active CLI. Real environment variables override the .env. Force host detection with THREADKEEPER_ACTIVE_CLI=claude (or codex, antigravity/agy, gemini, copilot). agy is normalized to antigravity; gemini remains a legacy Gemini CLI adapter for old installs/enterprise paths. See .env.example for the full knob list. spawn_status() includes warnings when a configured spawn CLI is unsupported or a model key does not match a supported CLI/startup role, while keeping the same fallback resolution.

Adapters without headless support (Claude Desktop, VS Code) can't be spawn targets — spawn_status() reports them as "no adapter" and any override pointing at them falls back to the next priority level.

---

Hygiene tools

Three tools keep the memory tidy. consolidate() and forget() default to dry_run=True; run them with dry_run=False to apply:

• consolidate() — dedup near-identical notes (intra-thread cosine ≥ 0.95), deduplicate verbatim quotes, demote untouched-active threads to idle after 30 days, release orphaned thread claims, prune ended tasks rows outside the configured retention window, and remove orphaned task spool files (.log, .stdin.txt, .command) from TASK_LOG_DIR. Live tasks (ended_at IS NULL) are never pruned. THREADKEEPER_TASK_RETENTION_DAYS defaults to 30 and THREADKEEPER_TASK_RETENTION_COUNT defaults to 1000; a row is kept if it is protected by either bound. Set either knob to 0 to disable that bound.

• forget(selector, selector_type="auto", dry_run=True) — targeted privacy erasure for one session/cid/thread/dialog UUID. Dry-run reports the rows that would be removed from dialog_messages, FTS/vector sidecars, notes, verbatim, dialectic observations/evidence/claims, extract candidates, task rows, task spool files, signals, and session sidecars. Applying deletes those rows and leaves dialog_fts, dialog_vec, dialog_vec_map, notes_fts, and notes_vec without orphaned rows. Lessons and skills that cite the purged source are listed for manual re-review instead of being silently kept or automatically edited. The same operation is available as tk-forget; it is also dry-run by default and uses --apply to delete.

• validate_threads() — heuristic triage of active threads with four categories (first match wins per thread):

  • no_notes_old — active with zero notes ≥ 7 days → close as abandoned.
  • shipped — last note matches a shipped-marker regex (EN+RU: shipped/fixed/works/passed/done/merged/закрыто/готово/сделано/…) and has settled ≥ 3 days → close with the last move as outcome.
  • dropped_open_q — last note is an open_q left unfollowed ≥ 14 days → close as dropped.
  • stale_idle — any active not touched in ≥ 30 days → demote to idle (not closed — revives on next note()).

  Idle threads are never touched. Tunable via no_notes_days, shipped_settle_days, drop_open_q_days, stale_days, and shipped_markers (comma-separated extra tokens).

---

Telemetry

• mp_dashboard(window_days=7) — one-call rollup of the whole system, read-only. Three sections: stores (threads by state, notes/dialog/distill/concepts counts, skills + claims by tier, extract-candidate and evolve queues, probe/task counts), loops (how many times each autonomous daemon fired in the window vs 30 days, plus last-fire age and 24h spend/tokens/mutation counts — the loop list is derived from the same source as agent_status, so it covers every daemon including the paid-spawn dialectic_validate / evolve_apply and the thread_janitor), and outcomes (what those loops actually produced — skills materialized, tier promotions, candidate accept-vs-reject rate, plus knowledge-store mutation counts: lesson_append / lesson_remove, curator_report_applied, roadmap_issue_applied, roadmap_issue_skipped, evolve_applied, dialectic_claim / dialectic_supersede). A curator_net_change added/removed/patched/net line makes a loop silently shrinking the lessons store visible at a glance, and curator_destructive_actions breaks destructive curator passes down into snapshot, lesson prune, lesson patch/consolidate, and skill delete/patch counts for the window. Surfaces the gaps the point-tools can't: a loop firing constantly while its outcomes stay flat, or a queue backing up. Complements the per-loop *_status tools (mp_health, spawn_budget_status, shadow_review_status).
• db_compact() — one-shot maintenance: VACUUM the SQLite file and rebuild dialog_fts (mandatory after VACUUM — rowid renumbering). Single-flight; fails soft with a retry hint when the DB is busy.
• shadow_review_status(snapshot_path="") — config, recent passes, and a per-loop production-validation rollup for the 24h and 7d windows: how often the daemon fired, the outcome mix (no_window / too_short / spawned / deferred / error), the MATERIALIZED-vs-SKIP hit rate of the evaluator children it spawned, the durable skill writes attributable to write_origin='shadow_review', and the total Claude-spawn time spent — so you can tell whether the loop earns its Opus minutes or just emits SKIPs. Pass snapshot_path to also dump a markdown report for human review. The verdict is read from each child's captured log tail; logs aged out of the ephemeral task-log dir (or skipped past the read cap) are counted as unknown so the hit-rate denominator stays honest.
• agent_status(json_output=False, refresh=True) — autonomous learning loop status, shaped for UI clients. Shows every loop's enabled/running/ready state, last pass, backlog, and active spawned-child RSS; running child agents are included as detail rows in the JSON. The JSON also includes github_budget (GitHub remaining/reset or active cooldown for roadmap automation) and recent_results for useful completed loop tasks, which the macOS menu-bar app uses for notifications. The tk-agent-status console command and macOS menu-bar app use the same underlying snapshot.

---

Storage

~/.threadkeeper/db.sqlite (overridable via THREADKEEPER_DB). WAL lets many readers proceed alongside a writer; SQLite still serializes writers. Optional notes_vec / dialog_vec HNSW indexes through sqlite-vec provide sub-linear semantic search, with Python-side cosine as the extension-free fallback.

The DB runtime separates three lifecycles. bootstrap_db() performs path hardening, WAL/schema migration, and vec-table setup once per process. read_db() opens a short-lived autocommit connection with PRAGMA query_only=ON, so retrieval cannot accidentally migrate, heartbeat, or write. run_write() opens a fresh connection, acquires BEGIN IMMEDIATE, runs a DB-only callback, and closes it; only SQLITE_BUSY/SQLITE_LOCKED are retried with bounded jitter. get_db() remains a compatibility API for older low-level call sites.

Schema migration uses SQLite PRAGMA user_version: a current database skips legacy ALTER TABLE work, while an old or fresh v0 database migrates once under a writer transaction and records the current version. Duplicate-column migrations are the only expected no-op; other DDL errors are logged and raised.

On POSIX systems, startup and get_db() harden the default local store best-effort: ~/.threadkeeper is 0700, while db.sqlite, SQLite -wal/-shm sidecars, ~/.threadkeeper/.env, curator REPORT-*.md files, and headless spawn logs are owner-only (0600).

Use tk-backup for disaster recovery. It uses SQLite VACUUM INTO, so committed frames still living in the live -wal sidecar are included in a compacted snapshot without quiescing background writers:

tk-backup create ~/threadkeeper-backup.sqlite
THREADKEEPER_DB=/path/to/db.sqlite tk-backup create ./backup.sqlite

Restore is intentionally explicit because it replaces the store. Stop thread-keeper servers and CLI sessions first, then swap in the verified single-file backup; the command removes stale db.sqlite-wal and db.sqlite-shm sidecars around the swap.

tk-backup restore ~/threadkeeper-backup.sqlite --yes

A raw cp ~/.threadkeeper/db.sqlite backup.sqlite is not a safe live backup in WAL mode because recent committed transactions may exist only in db.sqlite-wal. If you insist on raw filesystem copies, stop every writer first and copy db.sqlite, db.sqlite-wal, and db.sqlite-shm together. To wipe memory, also stop thread-keeper first, then remove the main DB and both sidecars.

Selective Erasure

For one regretted or sensitive conversation, use targeted erasure instead of removing the whole database:

tk-forget <session-or-cid>          # dry-run
tk-backup create ~/threadkeeper-before-forget.sqlite
tk-forget <session-or-cid> --apply

The MCP equivalent is forget(selector, dry_run=True), with dry_run=False for the destructive call. selector_type="auto" treats thread IDs and dialog UUIDs specially, otherwise it treats the selector as the conversation session_id/cid used by dialog_messages. The deletion cascades through the stores that can hold direct content or durable derivatives. lessons.md and SKILL.md files are not rewritten automatically because they may contain generalized guidance mixed with the cited source; the report lists matching lessons/skills under review_required so a human or foreground agent can decide whether to edit, keep, or remove them.

Retention

Retention is opt-in. All destructive windows default to 0 (keep forever), so upgrading does not delete historical transcripts, tasks, signals, events, or probe results. Set THREADKEEPER_RETENTION_INTERVAL_S plus the per-table day windows above to prune aged rows on a deterministic daemon tick. Dialog pruning keeps dialog_fts, dialog_vec, and dialog_vec_map consistent with dialog_messages.

mp_dashboard() reports DB file size, WAL/SHM sidecar size, and row counts for the high-volume tables (dialog_messages, dialog_fts, dialog_vec, signals, events, tasks, probe_results) so growth is visible before it becomes a problem.

db_compact() is the opt-in disk-reclaim tool: VACUUM + a mandatory dialog_fts rebuild (schema v2 keys the FTS index on dialog_messages rowids, which VACUUM is permitted to renumber — the rebuild is what keeps search correct). Run it once in a quiet window after upgrading to the v2 schema to shrink the DB file by roughly the old FTS shadow copy (~465 MB on a 2.7 GB DB); day-to-day it is never required.

Hooks and small runtime artifacts: ~/.threadkeeper/hooks/.

Spawn task spool files live in THREADKEEPER_TASK_LOG_DIR (default ~/.threadkeeper/tasks). The directory is created owner-only (0700) inside the hardened ~/.threadkeeper perimeter by default; explicit overrides are refused when the configured directory is a symlink or is not owned by the current user. spawn() creates captured headless .log, stdin prompt spool, and visible .command files with no-follow owner-only opens. consolidate() garbage-collects task spool files once their task row is no longer retained.

---

Embeddings

Semantic search runs paraphrase-multilingual-MiniLM-L12-v2 (384-dim, RU+EN+50 langs). The default backend is fastembed / ONNX Runtime — no PyTorch. A model-loaded process sits at ~700 MB physical footprint (~850 MB RSS), down from ~1.8 GB on the PyTorch backend.

A sentence-transformers (PyTorch) backend is kept as an opt-in fallback. It is heavier (~1.8 GB RSS) and produces vectors that are not numerically identical to the ONNX backend's, so switching backends warrants a recompute:

# Install the fallback runtime and switch to it:
pip install -e '.[semantic-st]'
export THREADKEEPER_EMBED_BACKEND=sentence-transformers

# After any backend switch, homogenize the stored corpus so queries and
# stored vectors live in the same space:
tk-migrate-embeddings --all          # or --notes-only / --dialog-only
tk-migrate-embeddings --dry-run      # report stale counts only

The migration is batched, resumable, and idempotent (a second run finds nothing stale). Both backends emit 384-dim vectors, so the vec0 schema is unchanged.

Stored rows carry an embedding-generation fingerprint, not just the backend: backend, model ID, vector dimension, pooling contract, and compatible runtime version. Search never compares a current query vector with a stale generation; those rows remain retrievable through FTS until tk-migrate-embeddings refreshes them. mp_dashboard() shows total/current-generation/vec coverage for notes and dialog rows.

Retrieval is hybrid by default. FTS candidate generation always runs, even when embeddings are installed or only part of the corpus has vectors. Dense and lexical candidates are over-fetched and fused with reciprocal-rank fusion; role filters are applied before dialog top-k selection. An over-specific FTS AND query retries once as BM25-ranked OR, while raw dense evidence below the calibrated cosine floor is discarded before fusion. Consequently a missing host, empty vec index, partial re-embedding, or irrelevant nearest neighbour degrades to lexical recall/abstention instead of returning noise.

Swapping in a different-width model. The notes_vec / dialog_vec tables are created as FLOAT[EMBED_DIM], default 384. If you point THREADKEEPER_EMBED_MODEL at a model of a different dimension, also set THREADKEEPER_EMBED_DIM to its width and recreate the *_vec tables — otherwise every vec0 insert mismatches the schema and the fast KNN path goes dead (semantic search still works via the legacy BLOB cosine path). thread-keeper logs a one-line warning naming both dimensions and this knob when it detects the mismatch, rather than failing silently.

Daemon-host + thin servers (Phase 1, dark by default). Behind THREADKEEPER_DAEMON_HOST (0 by default; no CLI config change), one headless host process per machine (python -m threadkeeper.host) owns the warm embedding model, the background loops, and a narrow embed-only unix socket (THREADKEEPER_HOST_SOCK, default <db dir>/host.sock). Per-session servers run thin instead — no ONNX, no daemon threads — and send any text needing a vector to the host over that socket instead of loading a model locally; the host's own background ingest daemon does the ongoing content-embedding work. If the host is unreachable a query embedding returns nothing and the caller falls back per THREADKEEPER_THIN_EMBED_FALLBACK: fts (default) runs FTS-only search, local lazily loads the model in-process instead. The host is elected via a flock and spawned detached by the first thin server that needs one; memory_guard supervises it — respawning it if its heartbeat goes stale past THREADKEEPER_HOST_HEARTBEAT_TTL_S — instead of idle-retiring it the way a thin server would be. See docs/ARCHITECTURE.md for the full design.

---

Verifying ingest across CLIs

python scripts/tk_verify_ingest.py            # both checks below
python scripts/tk_verify_ingest.py --contract # parse/ingest contract only
python scripts/tk_verify_ingest.py --live      # production verdict only
python scripts/tk_verify_ingest.py --live --json   # machine-readable

Two read-only checks:

• Contract test (--contract) — walks every installed CLI adapter, parses recent transcripts into an isolated tempdir DB, reports per-source message counts and flags any adapter that parsed messages but silently failed to persist them. Answers "does the pipeline work?"
• Production verification (--live) — reads the live dialog_messages table read-only and scores the three acceptance criteria from roadmap issue #1: (1) every targeted CLI slot has production rows, (2) shadow-review sees more than one adapter in the same recent window, (3) the learning loop has fired on non-Claude sessions. Emits a PASS / PARTIAL / FAIL verdict. The four slots are claude-code, codex, copilot, and google — where the Google slot is satisfied by either the legacy gemini adapter or its successor Antigravity (agy), since both live under ~/.gemini.

--strict makes the process exit non-zero unless the live verdict is PASS, so it can gate CI; PARTIAL (e.g. a box that doesn't run all four CLIs) is a valid real-world state and exits 0 by default. The reusable verdict logic lives in threadkeeper/verify_ingest.py.

---

Memory-quality evaluation

The ingest verifier above answers "did we capture the data?". The memory-quality harness answers the harder question — "when we retrieve it, do we recall the right fact, and do we refuse to answer about things that never happened?" It's modeled on LongMemEval (ICLR 2025) plus mem0's 2026 tokens-per-retrieval cost axis, and runs the real search() / dialog_search() / brief() tools as the systems-under-test.

python scripts/memory_eval/run.py                 # bundled demo corpus, lexical judge
python scripts/memory_eval/run.py --json          # machine-readable report
python scripts/memory_eval/run.py --db snap.sqlite --ground-truth my_labels.json
python scripts/memory_eval/run.py --semantic      # use embeddings if installed
python scripts/memory_eval/run.py --judge llm      # LLM-graded (needs ANTHROPIC_API_KEY)

It reports four headline groups over a fixed ground-truth set:

• accuracy — fraction of questions whose retrieval recalled the gold fact, broken out per the five LongMemEval axes (information extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention).
• abstention rate — of the never-happened questions, the fraction the system correctly refused. This is the highest-payoff axis: it directly measures whether the auto-injected brief() context fabricates or surfaces stale facts.
• tokens-per-retrieval — mean / median / max tokens of what each query returned, so recall is never read apart from cost (a wider window that recalls more also costs more).
• retrieval latency — mean / p50 / p95 / max wall-clock milliseconds. With --semantic, the backend is reported as hybrid, because dense candidates augment rather than replace FTS.

For DB concurrency, run the reproducible local gate:

python scripts/db_stress.py --processes 12 --ops 200

The JSON result includes expected/actual writes, throughput, p50/p95/p99/max write latency, worker errors, elapsed time, and PRAGMA quick_check; a non-zero exit means a lost write, worker failure, or integrity failure.

With no --db the harness builds the bundled fixture (scripts/memory_eval/ground_truth.json — a fictional "billing service" told across three sessions) into a throwaway DB; it's a golden baseline where a faithful retrieval scores 100%, so a regression in the retrieval tools drops the number. --db runs read-only: the snapshot is copied to a temp file and the original is never opened for writing. The default judge is lexical (deterministic, offline, no API key, no embeddings) so the command is reproducible and CI-safe; --judge llm grades answer reasoning (not just retrieval recall) with an Anthropic model when a key is set — the intended optimization target for lesson-decay tuning (#27) and bi-temporal claims (#28) work. See docs/ARCHITECTURE.md for how the axes map onto thread-keeper's retrieval surface.

Evaluating learning-loop decision quality

verify_ingest answers "did we capture the data?". The decision-quality harness answers the orthogonal question — "when the shadow-review and candidate-reviewer daemons make a materialize/skip or accept/reject call, are those calls right?" The codebase has decision telemetry but no labeled set and no precision/recall (roadmap issue #72); this harness adds both, modeled on the evidently.ai LLM-as-a-judge guide (build a labeled set, measure judge↔human agreement, calibrate before trusting a judge).

python -m threadkeeper.eval                 # bundled golden fixtures, offline rubric judge
python -m threadkeeper.eval --json          # machine-readable report
python -m threadkeeper.eval --judge llm     # replay the real prompt (needs ANTHROPIC_API_KEY)
python -m threadkeeper.eval --fixtures-dir my_labels/   # your own labeled set

It reports, over a small hand-labeled, anonymized fixture set checked into threadkeeper/eval/fixtures/:

• precision / recall / F1 for the shadow-review (materialize vs skip) and candidate-reviewer (accept vs reject) decisions, against the human labels.
• judge ↔ human agreement (raw accuracy + Cohen's kappa) for the open-ended "is this a high-quality skill?" judgment — the calibration number that makes a drifting judge visible.
• a PASS / PARTIAL / FAIL verdict on harness readiness (enough labels with both classes present), surfaced the same way as verify_ingest — not a fixed quality threshold.

The default rubric judge is deterministic, offline, and needs no API key: each fixture carries the human-tagged rubric signals it contains, and a signal only counts if its anchor phrase is still present in the live daemon prompt — so editing a rubric (dropping a signal class) deactivates those signals and moves the metric, which CI catches as a regression against the golden baseline. --judge llm replays the actual SHADOW_REVIEW_PROMPT / CANDIDATE_REVIEW_PROMPT over each item and parses the daemon's own verdict — the high-fidelity measurement, when a key is set. The fixtures are fully synthetic (a test asserts they carry no secrets or private paths); point --fixtures-dir at your own labeled set to score real decisions. See docs/ARCHITECTURE.md for how the harness couples to the daemon prompts.

Tests

pip install -e '.[semantic,dev]'
python -m pytest

869 tests passing on Python 3.11 / 3.12 / 3.13 (1 skipped). CI runs the suite on every push and PR.

---

Project layout

threadkeeper/
├── server.py             # MCP entry: python -m threadkeeper.server
├── _mcp.py               # FastMCP singleton + read_tool()/write_tool() annotation wrappers
├── tool_schemas.py       # typed outputSchema models for the structured status tools
├── _setup.py             # `thread-keeper-setup` installer
├── config.py             # env-driven defaults
├── db.py                 # SQLite schema + sqlite-vec loader
├── identity.py           # session, self-cid, daemon launchers
├── ingest.py             # adapter-driven transcript ingest
├── verify_ingest.py      # cross-CLI production verification verdict
├── eval/                 # offline learning-loop decision-quality harness (python -m threadkeeper.eval)
├── brief.py              # render_brief / render_context
├── shadow_review.py      # autonomous learning observer
├── i18n.py               # 10 locales of regex + prompt bundles
├── adapters/             # one file per supported CLI
│   ├── claude_code.py
│   ├── claude_desktop.py
│   ├── codex.py
│   ├── antigravity.py
│   ├── gemini.py
│   ├── copilot.py
│   └── vscode.py
└── tools/                # @read_tool()/@write_tool() entries — 113 of them
    ├── threads.py
    ├── peers.py
    ├── spawn.py
    ├── skills.py
    ├── dialectic.py
    ├── validate.py
    └── ...

Tool annotation contract (#67). Every tool registers through @read_tool() or @write_tool(destructive=…, idempotent=…) (in _mcp.py), so tools/list carries MCP 2025-06-18 ToolAnnotations for all 113 tools: readOnlyHint=True for pure reads (brief, context, search, dialog_search, the status tools, …) and readOnlyHint=False for mutations. lesson_list / lesson_get are classified as non-destructive writes because they bump lesson access counters. The ten delete/overwrite/kill tools carry destructiveHint=True (compost is read-only — it only surfaces idle threads). A confirmation/elicitation host reads this to decide which calls warrant a prompt. The five status tools (context, spawn_budget_status, spawn_status, mp_health, agent_status) additionally advertise an outputSchema and return structuredContent alongside the legacy text block. The contract is enforced by tests/test_tool_annotations.py.

Elicitation contract (#26). threadkeeper/elicitation.py contains the shared form-mode confirmation helper. It probes the host's elicitation capability before prompting, uses only a flat primitive schema, and leaves unsupported clients on the existing text/tool fallback path. The first protected write is dialectic_supersede.

Detailed map in docs/ARCHITECTURE.md. Open work in docs/ROADMAP.md and the Issues tab.

---

Contributing

PRs welcome — see CONTRIBUTING.md for the project map, test workflow, and recipes for adding a new CLI adapter or a new locale. Look for the good-first-issue label.

---

License

MIT — see LICENSE.