TokenPilot

Automatic token optimization for Claude Code. Extends session duration by reducing wasted tokens across every dimension — effort tuning, redundant file reads, tool cost routing, context health tracking, and smart task classification.

Built as a Claude Code hooks + MCP server system. Works alongside RTK for shell compression and MCP Compressor for schema reduction.

How It Works

TokenPilot runs as four layers:

1. Hooks — intercept Claude Code lifecycle events (session start, prompt submit, pre/post tool use)
2. MCP Server — exposes tools for real-time control and monitoring
3. SQLite Database — persists session state across hook subprocess calls with WAL mode + serializable isolation
4. Tool Registry — maps known tools to estimated costs and cheaper alternatives

┌── Claude Code Hooks ──────────────────────────────────────┐
│                                                            │
│  SessionStart        → init session, inject hints          │
│  UserPromptSubmit    → classify task → suggest effort      │
│  PreToolUse (Read)   → dedup file reads + suggest cheaper  │
│  PostToolUse (all)   → track real tool output token costs  │
│                                                            │
└────────────────┬───────────────────────────────────────────┘
                 │
    ┌────────────▼──────────────────┐
    │   TokenPilot MCP Server       │
    │                               │
    │   set_level(1-10)             │  Aggressiveness dial
    │   get_stats()                 │  Live session metrics
    │   get_savings()               │  Token savings report
    │   get_context_health()        │  Context window status
    │   get_tool_report()           │  Most expensive tools
    │   get_file_report(path)       │  File read history
    │   explain_classification(p)   │  Debug classifier
    │   reset_file_tracking()       │  Clear dedup cache
    │                               │
    │   SQLite + Tool Registry      │  Persistent state
    └───────────────────────────────┘

Aggressiveness Scale

Default: 4 (conservative-balanced). Adjustable 1-10 at any time via /tp level N.

Level	Effort Suggestion	File Read Dedup	Thinking Cap	Compact Reminder
1-2	Never	Notify only	No cap	90% context
3-4	Trivial tasks only	Warn on redundant	No cap	75% context
5-6	All tasks	Warn + suggest alternatives	Adaptive (10-30K)	65% context
7-8	Strong recommendation	Block re-reads	Adaptive (6-18K)	55% context
9-10	Enforce	Block + auto-range	Adaptive (4-12K)	45% context

Thinking caps are adaptive — they scale based on task complexity and classifier confidence. A "trivial" task gets a tighter cap than a "complex" task. If the classifier is uncertain (confidence < 0.5), no cap is applied.

Task Classifier (v2)

Lightweight regex + keyword classifier with negation detection, adjacency scoring, and quoted-code filtering. No LLM calls, <10ms execution.

Category	Effort	Model Hint	Example
trivial	low	haiku	"fix typo in README"
research	medium	sonnet	"explain how the API routes work"
standard	medium	sonnet	"add a loading spinner"
complex	high	opus	"refactor auth across all microservices"

v2 improvements:

• Negation detection: "don't refactor" no longer matches the refactor pattern
• Quoted-code filtering: backtick-wrapped code is stripped before classification
• Adjacency scoring: "add auth to 12 routes" correctly detects complexity from keyword pairs
• Confidence calibration: very short prompts get low confidence (0.3) instead of false high confidence

Debug any classification with /tp explain <prompt>.

Tool Cost Registry

TokenPilot knows the estimated token cost of common tools and suggests cheaper alternatives:

Tool	Avg Tokens	Alternative	Alt Tokens	Savings
Read	~2000	jCodeMunch symbol lookup	~200	90%
WebSearch	~2000	Context7 docs query	~800	60%
WebFetch	~3000	Context7 docs query	~800	73%

At level 5+, TokenPilot suggests alternatives when a cheaper tool could do the job.

Installation

Prerequisites

• Python 3.10+
• Claude Code CLI
• FastMCP (pip3 install fastmcp)

Setup

1. Clone to your MCPs directory:

git clone https://github.com/rish-e/tokenpilot.git ~/MCPs/tokenpilot

2. Install dependencies:

pip3 install -r ~/MCPs/tokenpilot/requirements.txt

3. Add hooks and MCP server to ~/.claude/settings.json:

{
  "hooks": {
    "SessionStart": [
      {
        "hooks": [
          { "type": "command", "command": "~/MCPs/tokenpilot/hooks/session_start.sh", "timeout": 5 }
        ]
      }
    ],
    "UserPromptSubmit": [
      {
        "hooks": [
          { "type": "command", "command": "~/MCPs/tokenpilot/hooks/classify.sh", "timeout": 5 }
        ]
      }
    ],
    "PreToolUse": [
      {
        "matcher": "Read",
        "hooks": [
          { "type": "command", "command": "~/MCPs/tokenpilot/hooks/check_read.sh", "timeout": 5 }
        ]
      }
    ],
    "PostToolUse": [
      {
        "matcher": ".*",
        "hooks": [
          { "type": "command", "command": "~/MCPs/tokenpilot/hooks/post_tool.sh", "timeout": 3 }
        ]
      }
    ]
  },
  "mcpServers": {
    "tokenpilot": {
      "command": "python3",
      "args": ["~/MCPs/tokenpilot/server.py"],
      "env": { "PYTHONPATH": "~/MCPs/tokenpilot" }
    }
  }
}

4. Install the /tp slash command:

cp ~/MCPs/tokenpilot/commands/tp.md ~/.claude/commands/tp.md

5. Restart Claude Code.

Optional: RTK for Shell Compression

brew install rtk-ai/tap/rtk
rtk init -g

Adds 60-90% token savings on shell output (build logs, test output, git).

Usage

TokenPilot runs automatically after installation. You'll see [TokenPilot] messages when it detects optimization opportunities.

Slash Commands

5 commands. That's it.

Command	What it does
/tp <1-10>	Set aggressiveness level
/tp on / off	Enable/disable TokenPilot
/tp stats	Full session dashboard
/tp note <text>	Add a note to the Project Brain
/tp explain <prompt>	Debug why a prompt was classified

MCP Tools

For power users, all tools are callable directly:

set_level toggle get_stats get_savings get_context_health get_tool_report get_file_report explain_classification add_note reset_file_tracking

CLI (for testing)

cd ~/MCPs/tokenpilot

python3 server.py init 4                    # Initialize session
python3 server.py classify "fix typo"       # Classify prompt
python3 server.py classify_debug "fix typo" # Debug classification
python3 server.py check_file "/src/app.py"  # Check file dedup
python3 server.py context_health            # Context window status

Project Brain

TokenPilot auto-maintains a tpcontext.md file in each project root. This is persistent memory across Claude Code sessions — when you start a new chat, Claude immediately knows where you left off.

Fully automatic:

• First install — bootstraps from git history (commits, active files, branch)
• Every session start — auto-saves previous session, loads brain into context
• No manual save needed — it just works

What it captures:

• Files modified (from git diff)
• Recent commits
• User notes (via /tp note "...")
• Session stats (duration, prompt count)
• Most active files

Add context for future sessions:

/tp note "switched to GraphQL — don't touch REST endpoints"

Stays under 2K tokens. Keeps last 5 sessions, older ones rotate out.

Smart Warnings

TokenPilot automatically detects and warns about token-wasting patterns:

• Rapid-fire prompts — 3+ short messages in a row triggers a "batch your questions" suggestion
• Session age — every 15 prompts, suggests /compact or starting fresh
• Peak hours — warns once per session during 5-11am PT weekdays (Anthropic burns limits faster during peak)

All warnings appear as [TokenPilot] messages and respect the on/off toggle.

File Structure

tokenpilot/
├── server.py            # FastMCP server + CLI entry point
├── classifier.py        # Task classifier (v2: negation, adjacency, debug)
├── config.py            # Aggressiveness scale + adaptive thinking caps
├── db.py                # SQLite persistence (WAL, indexed, serializable)
├── brain.py             # Project Brain — auto-generated tpcontext.md
├── tool_registry.py     # Tool cost estimates + cheaper alternatives
├── tracker.py           # In-memory tracker (used by MCP server process)
├── requirements.txt
├── commands/
│   └── tp.md            # /tp slash command (copy to ~/.claude/commands/)
├── hooks/
│   ├── session_start.sh # SessionStart — init + load brain
│   ├── classify.sh      # UserPromptSubmit — classify + rapid-fire + peak hours + session age
│   ├── check_read.sh    # PreToolUse (Read) — dedup + tool routing
│   └── post_tool.sh     # PostToolUse — real token tracking
└── templates/
    └── claudeignore-default

How Token Savings Stack

Layer	What	Savings
TokenPilot classifier	Right effort level per task	Thinking token reduction
TokenPilot file dedup	Skip redundant file reads	~2K tokens per blocked read
TokenPilot tool routing	Suggest cheaper tool alternatives	60-90% per substitution
TokenPilot PostToolUse	Track actual token costs (visibility)	Measurement enables optimization
TokenPilot smart warnings	Batch prompts, session age, peak hours	Prevents context blowup
TokenPilot Project Brain	Resume sessions without re-explaining context	3-5 messages saved per session start
RTK	Compress shell output	60-90% on Bash results
MCP Compressor	Compress MCP tool schemas	70-97% per wrapped server
.claudeignore	Exclude build artifacts from search	30-40% on exploration

License

MIT