TokenMizer

Keep your AI context alive across sessions.

Graph-backed memory · session checkpointing · intelligent compression
Drop-in proxy for Claude, GPT, Gemini, Grok, DeepSeek, Ollama — any LLM.

Quick Start · How it works · Benchmarks · Claude Code · Contributing

_{Real run: 25-node graph, checkpoint ckpt_21a0959c3ddf, 233-token resume. Regenerate with python scripts/gen_demo_gif.py.}

The Problem

Every AI session has a context limit. When you hit it:

• The model forgets every decision, rationale, and context built over hours
• You waste 10–30 minutes re-explaining the project every new session
• Large files (CSV, PDF, Excel) eat your entire token budget instantly

How TokenMizer Solves It

TokenMizer is a local proxy between your app and any LLM. Every request goes through a pipeline that builds a live knowledge graph, compresses inputs, caches responses, and auto-checkpoints before context runs out.

Your App  →  TokenMizer (:8000)  →  Claude / GPT / Gemini / any LLM
                    │
          ┌─────────┴──────────────┐
          │   6-Layer Pipeline     │
          │   L0  File Intel       │  CSV/PDF/Excel → schema + sample
          │   L1  Compression      │  15–40% input reduction
          │   L2  Output Trim      │  5–15% output reduction
          │   L3  Semantic Cache   │  100% on repeated queries
          │   L4  Graph Memory     │  session continuity
          │   L5  Prompt Cache     │  90% on repeated system prompts
          └────────────────────────┘

Architecture

Decision Memory — 4-State Model

History is never deleted. "Why did we switch from React to Next.js?" — always answerable: ask GET /api/graph/{session}/why?q=react (or the why_decision MCP tool) and get the full old → new trail with trigger, reason, and evidence per hop.

From Storage to Reasoning

The graph doesn't just store facts — it answers questions over them:

All reasoning is deterministic and local — no LLM calls, no extra cost.

Quick Start

python --version

pip install "tokenmizer[anthropic,cache]"

setx TOKENMIZER_ANTHROPIC_API_KEY "sk-ant-YOUR-KEY"

export TOKENMIZER_ANTHROPIC_API_KEY=sk-ant-YOUR-KEY

tokenmizer serve

1. Install

Works on Windows, macOS, and Linux (Python 3.10+). Same command everywhere:

# Recommended
pip install "tokenmizer[anthropic,cache]"

# All providers
pip install "tokenmizer[anthropic,openai,gemini,cohere,cache]"

# macOS:   brew install ollama
# Windows: winget install Ollama.Ollama   (or download from ollama.com)
# Linux:   curl -fsSL https://ollama.com/install.sh | sh

ollama pull llama3
pip install tokenmizer
# then set provider: ollama in tokenmizer.yaml

2. Set your API key

macOS / Linux (bash, zsh):

export TOKENMIZER_ANTHROPIC_API_KEY=sk-ant-...

Windows (PowerShell):

$env:TOKENMIZER_ANTHROPIC_API_KEY = "sk-ant-..."      # current session
setx TOKENMIZER_ANTHROPIC_API_KEY "sk-ant-..."         # persistent (new terminals)

Other providers: TOKENMIZER_OPENAI_API_KEY, TOKENMIZER_GEMINI_API_KEY, etc. — full table in Supported Providers.

3. Start

tokenmizer serve
# → Proxy:     http://localhost:8000/v1/chat/completions
# → Dashboard: http://localhost:8000
# → API docs:  http://localhost:8000/docs

4. Use — change one line

from openai import OpenAI

client = OpenAI(
    api_key="your-key",
    base_url="http://localhost:8000/v1",  # ← only this changes
)

response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Let's build an auth service"}],
    extra_body={"session_id": "my-project"},  # enables graph memory
)

✅ Streaming works (v0.3+): stream: true gives real SSE passthrough for Anthropic, OpenAI, DeepSeek, Mistral, OpenRouter, Grok and Ollama. Cursor and Continue.dev work with default settings — no config changes needed.

Claude Code Integration

Option A — Plugin (recommended)

# Add TokenMizer as a plugin marketplace
/plugin marketplace add Shweta-Mishra-ai/tokenmizer

# Install
/plugin install tokenmizer@Shweta-Mishra-ai/tokenmizer

Then use skills directly:

/tokenmizer:checkpoint my-project      → save session to graph memory
/tokenmizer:resume my-project          → load previous session (300 tokens)
/tokenmizer:resume my-project full     → full 600-token context
/tokenmizer:analyze /data/sales.csv    → analyze file (99% token savings)
/tokenmizer:stats                      → token savings report

Option B — MCP server (Claude Desktop, Claude Code, Cursor, VS Code, Zed)

mcp-name: io.github.Shweta-Mishra-ai/tokenmizer

Add this mcpServers block to your client's MCP config file:

{
  "mcpServers": {
    "tokenmizer": {
      "command": "tokenmizer-mcp",
      "env": { "TOKENMIZER_URL": "http://localhost:8000" }
    }
  }
}

Where the config file lives:

[mcp_servers.tokenmizer]
command = "tokenmizer-mcp"
env = { TOKENMIZER_URL = "http://localhost:8000" }

Then restart the client. Keep tokenmizer serve running for the checkpoint/resume/stats/reasoning tools (file analysis works without it). If tokenmizer-mcp isn't on your PATH, use "command": "python", "args": ["-m", "tokenmizer.mcp.server"] instead.

Tools exposed (6): checkpoint_session, resume_session, get_graph_stats, analyze_file, get_savings_stats, and why_decision — ask your agent "why did we pick X?" and it traces the decision's supersession chain with reasons and evidence.

Other Tools

Cursor / Continue.dev / any OpenAI-compatible tool:

API Base URL:  http://localhost:8000/v1

Session Resume

tokenmizer checkpoint my-project
tokenmizer resume my-project

Goal: Build FastAPI auth service with JWT + PostgreSQL
Done: Project setup | User model | Login endpoint | Fix 422 | 18 tests passing
In progress: Refresh token rotation
Decided: PostgreSQL (concurrent writes) | bcrypt | Redis for refresh tokens
Changed: ~~React~~ → Next.js (better SEO)
Files: api/auth.py, api/models.py, config.py
Continue: Implement token refresh endpoint

247 tokens replaces 25,000+ tokens of conversation history.

File Intelligence

from tokenmizer.filters.file_intelligence import FileIntelligence

fi = FileIntelligence()
result = fi.process(open("sales.csv","rb").read(), "sales.csv",
                    token_budget=500, query="which regions underperforming")
# 412,000 tokens → 447 tokens  (99.9% saved)

Works Alongside Caveman & CodeBurn

TokenMizer complements — does not replace — these tools:

Tip: If using Caveman, set terse_output: enabled: false in tokenmizer.yaml to avoid conflicting system prompts.

Supported Providers

Model strings pass through unchanged — the newest models work out of the box: claude-fable-5, claude-opus-4-8, claude-sonnet-5, claude-haiku-4-5, GPT-4o/o-series, Gemini 1.5/2.0, and any Ollama/OpenRouter model.

Configuration

# tokenmizer.yaml
provider: anthropic
default_model: claude-sonnet-4-6

graph_checkpoint:
  enabled: true
  trigger_at_percent: 0.85
  use_llm_extraction: false     # true = hybrid LLM+heuristic extraction
                                # (needs a provider key, ~$0.001/turn;
                                # requires v0.3.2+ — see CHANGELOG)

compression:
  enabled: true

cache:
  enabled: true
  max_size: 10000

state_backend: memory           # memory | redis (production)

All settings via env vars: TOKENMIZER_PROVIDER, TOKENMIZER_API_KEY, etc.

Docker

# Quick start
docker-compose up tokenmizer

# With Redis (production)
ANTHROPIC_API_KEY=sk-ant-... docker-compose up

# With proxy auth
TOKENMIZER_API_KEY=strong-key docker-compose up

API Reference

Security

• API key auth — TOKENMIZER_API_KEY (constant-time comparison)
• Secret/PII redaction applied once at ingestion, before graph storage, checkpoint storage, and every LLM call (main chat and the background extraction model). Patterns cover Anthropic/OpenAI/Google/GitHub/AWS/ Slack/Stripe/JWT/OpenRouter/HF/xAI keys, URL-embedded credentials (postgres://user:pass@host), and generic key=/password= assignments. Best-effort by nature — an unrecognized format with no keyword context can still slip through. The checkpoint layer independently re-redacts what it persists (defense in depth).
• Session-isolated cache (sensitive data never shared across sessions)
• Basic prompt-injection keyword filter — catches copy-pasted jailbreak templates only; not a security boundary against a motivated adversary. See SECURITY.md for exactly what it does and doesn't catch.
• CORS restricted to configured origins by default

Benchmarks

python benchmarks/checkpoint_accuracy/runner_v2.py
pytest tests/ -v

Benchmark v2 — Graph vs plain Summary (3 sessions, heuristic-only, measured 2026-07-02 on v0.2.4):

Avg resume size: 254 tokens vs ~1,500+ tokens of raw history. (n=3 synthetic sessions — small sample; treat as directional, reproduce with the command above.)

Enable use_llm_extraction: true for hybrid extraction (LLM + heuristic merge).

On LLM/hybrid recall numbers — read this before trusting any percentage here: earlier versions of this README quoted "90-100% hybrid recall" sourced from runner_v3.py's MockLLMProvider. That mock sampled its fake output directly from the same ground-truth dict used to score recall — circular by construction, guaranteed to look good regardless of what the real extraction logic did. It measured nothing about actual LLM extraction quality. That number has been removed rather than replaced with a better-sounding one we can't back up.

What runner_v3.py now actually does:

• Default mode verifies HybridExtractor.merge()'s logic contract against fixtures with deliberately known overlap (corroborated / LLM-only / heuristic-only items) — confirms merge never drops an item either source found, and applies confidence tiers (0.95 corroborated, 0.80 LLM-only, 0.65 heuristic-only) correctly. This is a real, non-circular check, but it's a logic-contract test, not a recall measurement.
• --live mode calls a real configured provider (ANTHROPIC_API_KEY or OPENAI_API_KEY) and scores its actual output against ground truth. This is the only path that produces a number meaningful enough to put in a table. Run it yourself — we're not publishing a live-mode number here because n=3 sessions is too small a sample to generalize, and publishing one without a large, ongoing benchmark would just be swapping one unsubstantiated number for another.

Heuristic-only numbers above (76-100%) ARE real, deterministic, reproducible measurements — runner_v2.py runs actual heuristic extraction against actual ground truth with no LLM and no mocking involved, which is why those numbers are presented with confidence and the LLM ones currently are not.

Why TokenMizer and not X?

Engineers ask this every time. Honest answers:

Why not just use Git history? Git stores what changed, not why you decided to change it. You can't ask Git "what did we decide about auth?" or "why did we switch from MySQL to PostgreSQL?" TokenMizer stores decisions with trigger, reason, and evidence — not diffs.

Why not RAG (retrieval-augmented generation)? RAG retrieves relevant chunks — it doesn't model decision state. If you switched from bcrypt to Argon2 mid-session, RAG might retrieve both and confuse the model about which is current. TokenMizer tracks decision supersession explicitly: old decision is marked SUPERSEDED, new decision is ACTIVE. Resume context only includes current state.

Why not a plain summary at the start of each session? Summaries lose structure. You can't query "all superseded decisions" or "what triggered the auth change" from a blob of text. Our benchmark shows graph memory preserves +5% more information than a summary baseline — and unlike summaries, the graph is queryable, editable, and grows incrementally without re-summarizing everything each turn.

Why not Mem0 or Zep? Mem0 and Zep store facts ("user prefers Python"). TokenMizer stores decisions with rationale — the full causal chain: what was decided, what replaced it, why, what evidence triggered the change, and how confidence shifted. If you need "remember my name across sessions," use Mem0. If you need "remember that we switched from PostgreSQL to SQLite because of cost, and here's the evidence," use TokenMizer.

Why not just a longer context window? Longer context = higher cost + slower inference + model attention dilution on long histories. TokenMizer compresses a 50-turn session into ~246 tokens of structured context — not by summarizing, but by extracting what actually matters: goals, active decisions, current tasks, recent errors.

CLI

tokenmizer serve [--port 8000]
tokenmizer checkpoint <session-id>
tokenmizer resume <session-id> [--level standard|full|critical]
tokenmizer stats

Note on file analysis: /tokenmizer:analyze (used from inside Claude Code, see Claude Code Integration above) is real and works — it's a plugin skill (.claude-plugin/skills/analyze/) that calls FileIntelligence directly via an inline Python snippet, independent of the CLI/API layer. What does not exist is a bare tokenmizer analyze <file> terminal command or a /api/analyze HTTP endpoint — useful if you want file analysis from a plain shell or a non-Claude-Code tool (Cursor, a script, curl, etc.) rather than inside Claude Code specifically. Found during a documentation accuracy pass: an earlier version of this README listed tokenmizer analyze <file> in this CLI section as if it were a cli.py command — it never was. Removed from here rather than left in place pointing at something that would fail. Tracked as a real, wanted gap — contributions adding a /api/analyze endpoint + thin CLI wrapper (following the existing pattern in cli.py) are welcome.

Roadmap

Have a use case that doesn't fit? Open an issue — extraction misses have their own issue template.

Contributing

Contributions welcome — this project merges fast (median PR review < 1 day).

git clone https://github.com/Shweta-Mishra-ai/tokenmizer
cd tokenmizer
pip install -e ".[dev]"
pytest tests/ -v && ruff check tokenmizer/     # 302 tests, must stay green
python scripts/mcp_e2e_check.py                # full-pipeline e2e check

Highest-impact areas right now:

1. Graph extraction quality — real-world transcripts where extraction misses tasks/decisions (file an extraction-miss issue even if you don't fix it — the failing transcript itself is the contribution)
2. Decision tracker edge cases — negation, semantic opposites, and same-decision matching are an active area (see recent merges below)
3. Reasoning and ontology (graph_memory/reasoning.py, graph_memory/ontology.py) — new in v0.4, still growing
4. Benchmark sessions — add a real session + ground truth to benchmarks/

Every PR runs the full CI gauntlet (tests × 3 Python versions on Linux, one Python version on Windows, lint, Docker build). See CONTRIBUTING.md for guidelines and TESTING.md for the test architecture.

Contributors

Thanks to everyone who has sent a fix upstream:

• @0xfroOty — negated-decision handling in the decision tracker (#22), OutputTrimmer level alignment (#25), streaming cache-hit analytics (#31)
• @pollychen-lab — graph node IDs derived from stored (truncated) labels (#21), semantic-opposite decision detection (#26)

Open a PR — CONTRIBUTING.md covers setup and review expectations.

Support the project

TokenMizer is built and maintained by one person. If it saved you tokens, time, or a lost session:

• ⭐ Star the repo — the single best way to help others find it
• 🐛 Report a bug — especially extraction misses
• 📣 Share your before/after token numbers (tokenmizer stats) — real usage data shapes the roadmap

License

MIT © Shweta Mishra

_{Built for developers who spend too much time re-explaining their projects to AI.}