TokenSkein

TokenSkein is a local context-economy gateway for GPT clients and Codex. It combines the useful ideas behind pxpipe, Headroom, RTK, and Caveman-style output discipline without pretending that every optimization helps every request.

The name combines token with skein: schemas, recovered text, image pages, shell output, style, and reasoning are separate strands managed as one measured bundle.

The working MVP can:

• proxy OpenAI Responses API requests;
• compact tool schemas and older tool outputs;
• archive every compacted output locally so exact text remains retrievable;
• expose compression, retrieval, and statistics over MCP;
• route reasoning effort from task complexity;
• inject a concise-output policy with safety exceptions;
• optionally encode very dense text as PNG pages when an explicit economic gate passes;
• optionally place a safe output filter after RTK for recognized read-only shell commands.

Status: working experimental MVP. Text compaction, storage, MCP, routing, proxying, and the shell adapter are implemented and tested. The vision lane is implemented but disabled by default until model-specific A/B evaluation establishes its real cost and quality.

Why this exists

Context cost has several different causes, so one trick is not enough:

• repeated schemas waste input tokens;
• large logs and JSON consume context even when only a few lines matter;
• verbose responses consume output tokens;
• easy tasks do not always need high reasoning effort;
• dense visual text can sometimes cost fewer model tokens than raw text, but can also damage exactness or cost more;
• lossy compression becomes dangerous when the omitted detail is later needed.

TokenSkein treats those as separate lanes with separate gates. Its core rule is: compact what is replaceable, retain what is exact, and keep the original recoverable.

Architecture

flowchart LR
    C[Codex or GPT client]
    U[OpenAI Responses API]
    D[(Protected local context store)]
    E[(Metrics JSONL)]
    N[(Usage JSONL)]

    subgraph L[TokenSkein]
      P[Responses proxy]
      Q[Request policy pipeline]
      M[MCP server]
      H[Codex hook and shell filter]
      A[Economics usage recorder]
      K[Dated price catalog]
      V[Dashboard server]
    end

    C -->|Responses request| P
    P --> Q
    Q -->|optimized request| U
    U -->|stream or response| P
    P --> C

    Q -->|archive exact originals| D
    C <-->|compress, retrieve, stats| M
    M <--> D
    C -->|safe read command| H
    H -->|optional RTK rewrite| H
    H -->|archive large output| D
    Q --> E
    M --> E
    H --> E
    P -->|provider usage| A
    K -->|cache-aware cost| A
    A --> N
    V -->|read| E
    V -->|read| N

Inside the request policy pipeline, operations run in this order:

1. remove non-operational tool-schema metadata;
2. inject the concise style policy when enabled;
3. select reasoning effort unless the caller already chose one;
4. leave the recent input tail intact;
5. compact eligible older function_call_output items;
6. optionally append PNG pages only when model allowlisting and the profitability estimate both pass;
7. forward the transformed request to the configured upstream.

A freeform editable copy lives at docs/architecture.excalidraw; the mermaid block above is the source of truth and may be ahead of it.

Module map

Module	Responsibility
src/proxy.ts	HTTP/SSE forwarding, local endpoints, request overrides
src/optimize.ts	Ordered Responses request policy pipeline
src/compact/	Tool-schema, recoverable-text, and PNG transforms
src/store.ts	Content-addressed archive, TTL, search, cleanup
src/mcp.ts	skein_compress, skein_retrieve, and skein_stats
src/shell.ts	Opt-in Codex hook, RTK delegation, shell-output archive
src/routing.ts	Prompt-complexity effort selection
src/metrics.ts	Content-free optimization event aggregation
src/economics.ts	Provider usage parsing (SSE and JSON) and usage JSONL recording
src/pricing.ts	Dated per-model price catalog and cache-aware cost computation
src/dashboard.ts + src/dashboard/	Loopback dashboard server and metrics/usage aggregation
src/config.ts	Config loading, validation, and environment overrides
src/install.ts	Transactional Codex install, uninstall, and verify
src/codex.ts	Non-mutating Codex integration snippets
src/cli.ts	User-facing command dispatcher

Architecture rationale

• Responses proxy for broad coverage. Request-level savings should not depend on the model remembering to call a tool.
• MCP for exact recovery. A short skein:<hash> reference is useful only if omitted context can be fetched on demand.
• Local-first storage. Exact tool output stays on the machine, gzip-compressed with 0700 directories and 0600 files. This is access protection, not encryption.
• Recent-tail preservation. New observations are more likely to affect the immediate next action and are left untouched.
• Text lane before vision lane. Text remains searchable and exact. Vision is a model-dependent optimization, so it is opt-in and has a break-even gate.
• Deterministic transforms first. Schema cleanup, line selection, identifier retention, and routing are inspectable and testable. Semantic summarization can be added later behind evaluation gates.
• Strict shell allowlist. Hook rewriting is limited to recognized read/verification commands. Destructive, compound, redirected, or ambiguous commands are not rewritten or auto-approved.
• Measurement without inflated claims. Savings statistics are tokenizer-based estimates. Dollar figures use provider-reported usage priced through a dated per-model catalog with cache-aware input rates; models absent from the catalog report cost as unknown instead of guessing.

Dashboard

The dashboard reads the metrics JSONL used by token-skein stats plus the provider usage JSONL, and renders estimated token savings, provider-reported cost and latency where recorded, cache hit/miss counts, and an hourly series, broken down by optimization mode and model. Start it with bun run dashboard (equivalent to bun run src/dashboard.ts); it listens on 127.0.0.1:8790 by default, overridable with TOKEN_SKEIN_DASHBOARD_PORT. It binds to loopback only and is not reachable over the network.

What was ported

The implementation is a TypeScript/Bun reimplementation of selected concepts, not a concatenation of the original projects.

Source	Useful idea	TokenSkein status
pxpipe	Render dense text into image pages	Implemented, opt-in
pxpipe	Model allowlist and profitability threshold	Implemented
pxpipe	Preserve a recent exact-text tail	Implemented for Responses input items
Headroom	Compress-cache-retrieve lifecycle	Implemented with content-addressed local storage
Headroom	TTL, cleanup, targeted retrieval, stats	Implemented
Headroom	Compact verbose tool schemas	Implemented recursively
Headroom	MCP compression/retrieval interface	Implemented with structured output
RTK	Rewrite supported commands before execution	Implemented as optional rtk rewrite delegation
RTK	Reduce noisy shell output	Implemented with exact archive recovery
Caveman	Short answers without losing technical substance	Implemented as a configurable instruction
Caveman	Bypass terse style for safety and irreversible actions	Preserved in the style policy
Global Claude setup	Progressive disclosure and task-based model effort	Implemented as recovery-on-demand plus effort routing
Global Claude setup	Durable global instruction layer	Optional Codex AGENTS.md fragment included

The optional prompt fragment lives at integrations/codex/AGENTS.token-skein.md. It is intentionally small: moving a large global CLAUDE.md wholesale would spend the context this project is meant to save.

Planned ports and extensions

The prioritized backlog is tracked in PLAN.md. The main planned work is:

• build a repeatable A/B harness for correctness, exact identifiers, latency, and cost;
• add streaming-aware response accounting without buffering SSE;
• add session-aware history compaction instead of only item-age rules;
• tune visual pages per model and reject OCR-hostile or exactness-critical content;
• add native shell reducers so RTK remains optional rather than required;
• add MCP resources and indexed retrieval for large archives;
• evaluate other provider protocols only after the Responses path is stable.

Quick start

Requirements: Bun 1.3+ and an OpenAI API key for the default upstream.

cd ~/projects/token-skein
bun install
bun run check
bun run proxy

The default proxy listens on http://127.0.0.1:8788 and forwards to https://api.openai.com. It does not copy or store API keys; authorization headers are forwarded in memory.

For a generic Responses client, point its OpenAI base URL at:

http://127.0.0.1:8788/v1

Codex integration

Print integration snippets without changing global configuration:

bun run src/cli.ts codex-snippet

Then:

1. merge the provider and MCP blocks into ~/.codex/config.toml;
2. select model_provider = "token_skein" globally or in the profile where it should run;
3. merge the hook block into ~/.codex/hooks.json only if the experimental shell lane is wanted;
4. after reviewing its strict allowlist, set shell.enabled to true for that opt-in lane;
5. start bun run proxy before starting that Codex profile;
6. ensure OPENAI_API_KEY is available to Codex.

This path uses OpenAI API-key billing. It does not claim to tunnel ChatGPT/Codex subscription authentication through a custom provider. WebSocket Responses transport is currently disabled; HTTP and SSE are forwarded.

The Codex hook adapter targets the Bash PreToolUse shape. Codex builds that expose shell execution only through another tool name will still benefit from the proxy and MCP lanes, but will not use the shell rewrite until an adapter for that tool shape is added.

Optional global instruction fragment

Review integrations/codex/AGENTS.token-skein.md, then merge only the rules you want into ~/.codex/AGENTS.md. TokenSkein never edits that global file automatically.

Configuration

The default config path is ~/.config/token-skein/config.json. Start from token-skein.config.example.json. Nested values are merged over defaults.

Environment overrides:

Variable	Purpose
TOKEN_SKEIN_CONFIG	Alternate JSON config path
TOKEN_SKEIN_HOST	Proxy bind host
TOKEN_SKEIN_PORT	Proxy port
TOKEN_SKEIN_UPSTREAM	Upstream API origin
TOKEN_SKEIN_STORE_DIR	Local archive directory
TOKEN_SKEIN_EVENTS_PATH	Metrics JSONL path
TOKEN_SKEIN_BIN	Exact command used by the hook wrapper
TOKEN_SKEIN_DASHBOARD_PORT	Dashboard bind port (default 8790), read directly by bun run dashboard

The archive quota is a config file field, not an environment variable: archive.maxBytes (default 200 MiB) caps the local context store, and once exceeded the least-recently-touched entries are evicted first.

Request-level controls:

Header	Accepted values	Effect
x-token-skein-bypass	true / false	Skip all request transforms
x-token-skein-vision	true / false	Override the vision switch for this request
x-token-skein-style	true / false	Override concise-style injection
x-token-skein-effort	low, medium, high, xhigh	Explicit reasoning effort

Vision defaults to off. Enabling it is only a request to evaluate the lane: content must still exceed minimumBytes, the model must match vision.models, and estimated text tokens divided by estimated image tokens must exceed minimumSavingsRatio.

CLI

token-skein proxy                 start the local Responses proxy
token-skein mcp                   start the stdio MCP server
token-skein hook codex            process a PreToolUse event from stdin
token-skein shell --encoded DATA  execute a hook-approved command and filter output
token-skein stats                 show estimated savings and archive statistics
token-skein cleanup               remove expired archive entries
token-skein codex-snippet         print Codex integration snippets
token-skein install               install the Codex integration (transactional)
token-skein uninstall             remove the Codex integration
token-skein doctor                check prerequisites for the Codex integration
token-skein verify                validate an existing install without mutating anything
token-skein config                print active configuration

Until the package is linked globally, replace token-skein with bun run src/cli.ts.

The dashboard (bun run dashboard) and the license scanner (bun run license-scan) are separate Bun scripts, not token-skein subcommands.

MCP tools

• skein_compress: archive exact content and return a compact view plus reference;
• skein_retrieve: fetch the exact original or matching lines with context;
• skein_stats: aggregate optimization and archive metrics.

The MCP server uses stdio and emits protocol traffic only on stdout; diagnostics go to stderr.

Data and security boundaries

• Archive objects contain original tool output and may contain secrets. They are local, gzip-compressed, permission-restricted, and TTL-bound, but not encrypted.
• Compacted views redact common API-key, authorization, token, secret, and password forms. The exact archive intentionally retains the original for recovery.
• Metrics record sizes, token estimates, mode, model, references, and non-content metadata; they do not intentionally record request or output bodies.
• The proxy binds to loopback by default. Binding it to a network interface requires your own authentication and transport controls.
• The shell hook is disabled by default. When explicitly enabled, it never rewrites commands containing destructive patterns, command chaining, redirects, heredocs, command substitution, dangerous reducer flags, or newlines. A non-match falls back to normal Codex behavior.

Known limitations

• Savings are estimates from an o200k tokenizer, not invoice-grade measurements.
• Image token cost varies by model, image detail policy, page density, and pricing. PNG can be cheaper, equal, or more expensive than text.
• Text-in-image is lossy for exact punctuation and identifiers. The text sidecar and archive reduce that risk but do not make vision exact.
• Only older function_call_output strings are compacted automatically; arbitrary conversation history is not yet rewritten.
• Visual pages are capped. Omitted material remains available through the archive.
• The current effort router is heuristic and English-biased.
• The proxy has no distributed store, encryption-at-rest, or multi-user isolation.

Verification

bun run typecheck
bun test

Tests cover storage and expiry, recoverable text compaction, tool-schema compaction, effort routing, vision gating, shell safety, and end-to-end proxy transformation through a mock upstream.

License and attribution

TokenSkein is licensed under Apache-2.0. It adapts concepts from pxpipe (MIT), Headroom (Apache-2.0), RTK (Apache-2.0), and the user's global Claude workflow. See NOTICE and LICENSES/README.md.