Read this in: English · Español

Demo Links

• Live gateway: https://gateway.javierzader.com
• GHAGGA integration target: https://github.com/JNZader/ghagga
• OpenCode: https://github.com/anomalyco/opencode

Visuals coming soon.

Quick Portfolio Snapshot

• One service for LLM routing, encrypted credential storage, MCP tooling, and OpenAI-compatible HTTP access.
• 11 provider adapters today: 5 direct API providers plus 6 CLI-backed providers.
• Supports API keys and auth-file workflows, including auth.json and .credentials.json.
• Includes task-aware bridge routing, model routing, project-scoped credentials with global fallback, semantic code search, context compression, and CRDT shared state.
• Ships as a local dev tool, self-hosted HTTP gateway, MCP stdio server, and Docker deployment.

Why It Matters

• Centralizes secrets instead of scattering provider tokens across every project and tool.
• Lets you reuse CLI subscriptions such as OpenCode, Claude, Gemini, Codex, Qwen, and Copilot behind one interface.
• Gives OpenAI-compatible tools a single stable endpoint while preserving provider/model resolution metadata.
• Supports multi-project setups where project-specific credentials override _global defaults cleanly.
• Exposes MCP tools beyond plain generation: vault operations, code search, shared state, usage inspection, and provider-group management.

Quick Start

pnpm install
pnpm run serve

Open http://localhost:3456.

Store a credential and generate text:

curl -X POST http://localhost:3456/v1/credentials \
  -H 'Content-Type: application/json' \
  -d '{"provider":"anthropic","apiKey":"sk-ant-..."}'

curl -X POST http://localhost:3456/v1/generate \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"Explain quicksort in one paragraph"}'

If you set LLM_GATEWAY_AUTH_TOKEN, add Authorization: Bearer <token> to every protected route.

Jump to Technical Docs

• Full API reference: Technical README
• Auth and credential model: Authentication, Credential Management
• MCP integration: MCP Server
• Docker and self-hosting: Docker Deployment

---

Technical README

Table of Contents

1. Quick Start
2. Dashboard
3. API Reference
4. Providers
5. Authentication
6. Credential Management
7. Cross-Model Bridge
8. Context Compression
9. Semantic Code Search
10. CRDT Multi-Agent State
11. Integrations
12. Docker Deployment
13. MCP Server
14. Configuration
15. Security Profiles
16. Approval Flows
17. Three-Part Prompt
18. RTK Output Compression
19. Local LLM Offloading
20. Model Routing
21. HF Auto-Discovery
22. Architecture
23. Security
24. Development
25. License

Quick Start

# Install dependencies
pnpm install

# Start the HTTP server + dashboard
pnpm run serve

# MCP stdio mode only
pnpm run start

Basic HTTP flow:

# Store a global Anthropic key
curl -X POST http://localhost:3456/v1/credentials \
  -H 'Content-Type: application/json' \
  -d '{"provider":"anthropic","apiKey":"sk-ant-..."}'

# Generate text with automatic provider selection
curl -X POST http://localhost:3456/v1/generate \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"Explain quicksort in one paragraph"}'

If auth is enabled:

curl -X POST http://localhost:3456/v1/generate \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{"prompt":"Explain quicksort in one paragraph"}'

Dashboard

The repo currently has two dashboard surfaces:

• Local inline shell at http://localhost:3456/ — legacy local ops surface served directly by the bridge. This remains the source of truth for local credential/auth-file management and quick test generation.
• React admin app under dashboard/ (built into docs/) — admin/observability surface for overview, providers, usage, groups, circuit breakers, settings, and related views.

They intentionally coexist for now and do not have full feature parity.

• Hosted demo: https://gateway.javierzader.com
• Local inline shell: http://localhost:3456

First-Time Setup

1. Start the gateway with pnpm run serve.
2. Open the dashboard.
3. Enter the base URL for your gateway.
4. Enter the bearer token if LLM_GATEWAY_AUTH_TOKEN is configured.
5. Test the connection and save.

Local Inline Shell Capabilities

• Add, list, filter, and delete encrypted API keys.
• Upload auth files for CLI-backed providers.
• Inspect provider availability and available models.
• Send test prompts and inspect returned provider/model metadata.
• Work with project-scoped credentials without exposing raw secrets.

React Admin App Capabilities

• Overview / provider status / usage / groups / circuit breakers / settings
• Admin-facing operational visibility over bridge subsystems
• Hosted separately from the inline shell via the dashboard/ app

Recommended auth-file mappings in the UI and API:

• opencode -> auth.json
• claude -> .credentials.json
• codex -> auth.json
• gemini -> settings.json and oauth_creds.json
• qwen -> settings.json and oauth_creds.json
• copilot -> use token credentials instead of auth files

API Reference

All protected endpoints require:

Authorization: Bearer <your-token>

When LLM_GATEWAY_AUTH_TOKEN is not set, auth is disabled for local development. GET /health always stays public.

Core HTTP Endpoints

Endpoint	Method	Description
/health	GET	Public health check for uptime monitors and platforms like Coolify
/metrics	GET	Prometheus metrics export
/v1/generate	POST	Native generation endpoint
/v1/chat/completions	POST	OpenAI-compatible chat completions
/v1/models	GET	OpenAI-compatible model list
/v1/providers	GET	Provider availability and metadata
/v1/latency	GET	Current latency measurements when latency routing is enabled
/v1/cost/estimate	GET	Cost estimate for a model and token counts
/v1/cost/models	GET	Model pricing table
/v1/usage	GET	Raw usage records
/v1/usage/summary	GET	Aggregated usage summary
/v1/credentials	POST / GET	Store and list encrypted API keys
/v1/credentials/:id	DELETE	Delete a stored credential
/v1/files	POST / GET	Store and list encrypted auth files
/v1/files/:id	DELETE	Delete a stored auth file
/v1/groups	GET / POST	List or create provider groups
/v1/groups/:id	PUT / DELETE	Update or delete a provider group

POST /v1/generate

Native generation endpoint with provider/model selection and project-scoped credential resolution.

# Auto-select provider
curl -X POST http://localhost:3456/v1/generate \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{"prompt":"Explain quicksort in one paragraph"}'

# Explicit provider + model + project
curl -X POST http://localhost:3456/v1/generate \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'X-Project: my-app' \
  -d '{
    "prompt":"Write a haiku about Rust",
    "provider":"groq",
    "model":"llama-3.3-70b-versatile",
    "maxTokens":256,
    "system":"You are a poet.",
    "project":"my-app"
  }'

Request body:

Field	Type	Required	Description
prompt	string	Yes	User prompt
system	string	No	System prompt
provider	string	No	Preferred provider ID
model	string	No	Specific model ID
maxTokens	number	No	Max output tokens
project	string	No	Credential scope
strict	boolean	No	Strict routing behavior when supported

Response:

{
  "text": "Quicksort is a divide-and-conquer...",
  "provider": "anthropic",
  "model": "claude-sonnet-4-20250514",
  "tokensUsed": 150,
  "requestedProvider": null,
  "requestedModel": null,
  "resolvedProvider": "anthropic",
  "resolvedModel": "claude-sonnet-4-20250514",
  "fallbackUsed": false
}

POST /v1/chat/completions

OpenAI-compatible chat endpoint. This is the drop-in path for tools that already speak OpenAI format.

• Non-streaming and streaming requests are supported.
• System messages are collapsed into the system prompt.
• Conversation context is reconstructed from earlier messages.
• Response stays OpenAI-compatible and adds x_gateway metadata.

curl -X POST http://localhost:3456/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "model":"claude-sonnet-4-20250514",
    "messages":[
      {"role":"system","content":"You are a helpful assistant."},
      {"role":"user","content":"What is the capital of France?"}
    ],
    "max_tokens":1024
  }'

Response:

{
  "id": "chatcmpl-<uuid>",
  "object": "chat.completion",
  "created": 1710000000,
  "model": "claude-sonnet-4-20250514",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "The capital of France is Paris." },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 150 },
  "x_gateway": {
    "requestedProvider": null,
    "requestedModel": "claude-sonnet-4-20250514",
    "resolvedProvider": "anthropic",
    "resolvedModel": "claude-sonnet-4-20250514",
    "fallbackUsed": false,
    "tokensUsed": 150
  }
}

GET /v1/models

Lists available models in OpenAI-compatible format.

curl http://localhost:3456/v1/models \
  -H 'Authorization: Bearer YOUR_TOKEN'

{
  "object": "list",
  "data": [
    {
      "id": "claude-sonnet-4-20250514",
      "object": "model",
      "created": 0,
      "owned_by": "llm-gateway",
      "name": "Claude Sonnet 4",
      "provider": "anthropic",
      "max_tokens": 8192
    }
  ]
}

GET /v1/providers

Lists registered providers and their availability.

curl http://localhost:3456/v1/providers \
  -H 'Authorization: Bearer YOUR_TOKEN'

{
  "providers": [
    { "id": "anthropic", "name": "Anthropic", "type": "api", "available": true },
    { "id": "openai", "name": "OpenAI", "type": "api", "available": false },
    { "id": "opencode-cli", "name": "OpenCode CLI", "type": "cli", "available": true }
  ]
}

Credentials API

Store API keys encrypted at rest. Upsert key is (provider, keyName, project).

# Global credential
curl -X POST http://localhost:3456/v1/credentials \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "provider":"anthropic",
    "keyName":"default",
    "apiKey":"sk-ant-api03-..."
  }'

# Project-scoped credential
curl -X POST http://localhost:3456/v1/credentials \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "provider":"openai",
    "keyName":"default",
    "apiKey":"sk-proj-...",
    "project":"my-app"
  }'

{ "id": 1, "provider": "anthropic", "keyName": "default", "project": "_global" }

List credentials:

curl http://localhost:3456/v1/credentials \
  -H 'Authorization: Bearer YOUR_TOKEN'

curl 'http://localhost:3456/v1/credentials?project=my-app' \
  -H 'Authorization: Bearer YOUR_TOKEN'

{
  "credentials": [
    {
      "id": 1,
      "provider": "anthropic",
      "keyName": "default",
      "project": "_global",
      "maskedValue": "sk-ant-...***",
      "createdAt": "2025-01-15 10:30:00",
      "updatedAt": "2025-01-15 10:30:00"
    }
  ]
}

Delete a credential:

curl -X DELETE http://localhost:3456/v1/credentials/1 \
  -H 'Authorization: Bearer YOUR_TOKEN'

Auth Files API

Store auth files for CLI-backed providers encrypted at rest. Upsert key is (provider, fileName, project).

This is the path that preserves the older auth.json and .credentials.json workflows.

# OpenCode auth.json
curl -X POST http://localhost:3456/v1/files \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "provider":"opencode",
    "fileName":"auth.json",
    "content":"{\"token\":\"oc-...\"}",
    "project":"_global"
  }'

# Claude CLI .credentials.json
curl -X POST http://localhost:3456/v1/files \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "provider":"claude",
    "fileName":".credentials.json",
    "content":"{\"claudeAiOauth\":{...}}",
    "project":"my-app"
  }'

{ "id": 1, "provider": "opencode", "fileName": "auth.json", "project": "_global" }

List auth files:

curl http://localhost:3456/v1/files \
  -H 'Authorization: Bearer YOUR_TOKEN'

curl 'http://localhost:3456/v1/files?project=my-app' \
  -H 'Authorization: Bearer YOUR_TOKEN'

{
  "files": [
    {
      "id": 1,
      "provider": "opencode",
      "fileName": "auth.json",
      "project": "_global",
      "createdAt": "2025-01-15"
    }
  ]
}

Delete an auth file:

curl -X DELETE http://localhost:3456/v1/files/1 \
  -H 'Authorization: Bearer YOUR_TOKEN'

Usage, Cost, Metrics, and Health

Usage records:

curl 'http://localhost:3456/v1/usage?project=my-app&limit=50' \
  -H 'Authorization: Bearer YOUR_TOKEN'

Usage summary:

curl 'http://localhost:3456/v1/usage/summary?groupBy=provider&project=my-app' \
  -H 'Authorization: Bearer YOUR_TOKEN'

Cost estimate:

curl 'http://localhost:3456/v1/cost/estimate?model=claude-sonnet-4-20250514&inputTokens=1000&outputTokens=500' \
  -H 'Authorization: Bearer YOUR_TOKEN'

Prometheus metrics:

curl http://localhost:3456/metrics \
  -H 'Authorization: Bearer YOUR_TOKEN'

Health check:

curl http://localhost:3456/health

GET /health returns the runtime VERSION constant (src/core/constants.ts) plus uptime, auth mode, and provider counts:

{
  "status": "ok",
  "version": "0.3.1",
  "timestamp": "2025-01-15T10:30:00.000Z",
  "uptime": 3600,
  "auth": { "enabled": true, "mode": "bearer" },
  "providers": { "total": 11, "available": 3 }
}

Note: the VERSION constant and the version field in package.json are not kept in lockstep — /health reports the former.

Provider Groups

Provider groups let you define logical pools for balancing and failover.

curl -X POST http://localhost:3456/v1/groups \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "name":"fast-models",
    "modelPattern":"gpt-*,claude-*",
    "members":[
      {"provider":"groq","weight":2,"priority":1},
      {"provider":"anthropic","weight":1,"priority":2}
    ],
    "strategy":"weighted",
    "stickyTTL":300
  }'

Providers

API Providers

Provider	ID	Auth	Example Models
Anthropic	anthropic	API key	claude-sonnet-4-20250514, claude-haiku-4-20250414
OpenAI	openai	API key	gpt-4o, gpt-4o-mini, o3-mini
Google	google	API key	gemini-2.5-flash, gemini-2.5-pro
Groq	groq	API key	llama-3.3-70b-versatile, llama-3.1-8b-instant
OpenRouter	openrouter	API key	deepseek/deepseek-chat, anthropic/claude-sonnet-4

CLI Providers

Provider	ID	Auth Material	Notes
OpenCode CLI	opencode-cli	auth.json from vault	Large model catalog via subscription routing
Claude CLI	claude-cli	.credentials.json from vault	Uses Claude Max credentials
Gemini CLI	gemini-cli	CLI auth files	Local CLI-backed execution
Codex CLI	codex-cli	auth.json	OpenAI CLI-backed execution
Qwen CLI	qwen-cli	CLI auth files	Qwen local/subscription access
Copilot CLI	copilot-cli	token credentials	GitHub Copilot-backed routing

OpenCode Model Coverage

OpenCode is the biggest catalog here and is one reason this bridge is useful.

• Free tier models under opencode/*
• OpenCode Go subscription models under opencode-go/*
• Anthropic models under anthropic/*
• GitHub Copilot-routed models under github-copilot/*
• OpenAI-routed models under openai/*

Representative examples from the current adapter list:

• opencode/gpt-5-nano
• anthropic/claude-sonnet-4.5
• github-copilot/gpt-5.4
• openai/gpt-5.4

Provider Priority and Fallback

Default behavior without an explicit provider/model:

1. API providers are tried first.
2. CLI providers follow as fallback.
3. If a model is explicitly requested, the owning provider is preferred.
4. If bridge routing is enabled, the bridge can override the initial provider choice and then walk the configured fallback chain.

Authentication

Bearer Token

Set LLM_GATEWAY_AUTH_TOKEN to protect HTTP routes.

# Generate a secure token
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

export LLM_GATEWAY_AUTH_TOKEN="your-64-char-hex-token"

The token must be at least 32 characters.

Auth Rules

Path	Bearer Auth Required
GET /health	No
OPTIONS * (CORS preflight)	No
/auth/github/*	No
/v1/admin/* (entire admin surface)	No
All other HTTP routes, including dashboard and /metrics, when token is set	Yes

Important behavior:

• The bearer-auth middleware skips the entire /v1/admin/* prefix, not just /v1/admin/auth-config. Admin routes gate themselves with their own dashboard/GitHub-OAuth JWT checks (verifyDashboardJwt) rather than the static bearer token. Keep this in mind when exposing the gateway publicly.
• The dashboard (non-admin routes) is protected when bearer auth is enabled.
• MCP stdio does not use HTTP bearer auth because it runs as a local process.
• Token comparison is constant-time via timingSafeEqual.

Project Scoping

Project scope can be supplied in either place:

1. JSON body field: "project": "my-app"
2. Header: X-Project: my-app

Body field wins over header.

Credential Management

Global vs Project Credentials

Credential resolution follows the same pattern for API keys and auth files:

1. Try the project-specific entry.
2. Fall back to _global.

That lets you keep a shared default while still isolating overrides per app or customer.

API Keys

API keys are encrypted with AES-256-GCM and stored in SQLite.

# Global key
curl -X POST http://localhost:3456/v1/credentials \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{"provider":"anthropic","apiKey":"sk-ant-..."}'

# Project key
curl -X POST http://localhost:3456/v1/credentials \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{"provider":"anthropic","apiKey":"sk-ant-project-...","project":"my-app"}'

Auth Files

CLI adapters use file-based auth where necessary. These files are also encrypted and stored in the vault.

# OpenCode auth.json
curl -X POST http://localhost:3456/v1/files \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "provider":"opencode",
    "fileName":"auth.json",
    "content":"{\"token\":\"oc-...\"}"
  }'

# Claude CLI .credentials.json
curl -X POST http://localhost:3456/v1/files \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "provider":"claude",
    "fileName":".credentials.json",
    "content":"{\"claudeAiOauth\":{...}}"
  }'

Claude and OpenCode Credential Sync Pattern

The vault layer also contains a Claude OAuth integration that:

1. Reads ~/.claude/.credentials.json
2. Refreshes the token when needed
3. Syncs the token into OpenCode-style auth.json

That matters because this bridge can unify Claude CLI and OpenCode auth flows instead of treating them as separate credential silos.

Cross-Model Bridge

The bridge is an optional routing layer driven by ~/.llm-gateway/bridge.yaml.

Flow:

1. Classify the prompt into a task type.
2. Resolve a preferred provider from routes.
3. Try that provider first.
4. Walk fallback_order sequentially if it fails.

Supported Task Types

Task Type	Heuristic	Typical Route
large-context	Very large prompt/context	gemini-cli
code-review	Review/audit/refactor keywords	claude-cli
fast-completion	Short prompt	groq
default	No heuristic matched	configured default

Example bridge.yaml

routes:
  large-context: gemini-cli
  code-review: claude-cli
  fast-completion: groq

default: claude-cli

fallback_order:
  - claude-cli
  - gemini-cli
  - opencode-cli
  - anthropic
  - groq

If the file is missing, the bridge is disabled and the normal router behavior is used.

Bridge Response Metadata

Field	Description
text	Generated text
provider	Provider that answered
model	Model used
taskType	Classified task type
fallbackUsed	Whether a non-primary provider handled it
latencyMs	End-to-end latency

Context Compression

The CompressorService adds background context compression with caching.

Strategies

Strategy	How It Works	Good For
extractive	Keeps high-scoring sentences	general text
structural	Preserves headings and list structure	markdown/docs
token-budget	Cuts to a size budget at sentence boundaries	hard token limits

Usage

import { CompressorService } from './context-compression/index.js';

const compressor = new CompressorService({
  maxCacheSize: 200,
  workerIntervalMs: 5000,
  defaultStrategy: 'extractive',
  defaultRatio: 0.5,
});

compressor.submit(longContext);
const compressed = compressor.getCompressed(longContext);
const immediate = compressor.compressNow(longContext, 'structural');
compressor.destroy();

Operational Characteristics

• LRU cache for repeated content
• Background worker for non-blocking pre-computation
• Synchronous compression when you need the result immediately
• Useful for prompt pipelines where raw context would otherwise blow up token budgets

Semantic Code Search

The code-search subsystem exposes three search modes through MCP:

• keyword (default): exact/prefix/fuzzy matching with inverted index
• vector: semantic similarity via dense embeddings
• hybrid: RRF fusion of keyword + BM25 + vector for best results

It combines:

• regex-based chunking
• trigram fuzzy search
• BM25 keyword scoring (via MiniSearch)
• dense vector similarity (via transformer embeddings)
• Reciprocal Rank Fusion (RRF) for hybrid ranking
• optional multi-hop import following

Supported Languages

DEFAULT_EXTENSIONS (indexed by default) covers:

.ts, .tsx, .js, .jsx, .mjs, .cjs, .py, .go, .rs, .java, .rb, .lua

Dedicated chunk patterns exist for TypeScript/JavaScript, Python, Go, and Rust. Other indexed extensions (.java, .rb, .lua) fall back to the TypeScript/C-family chunk patterns.

MCP Search Tools

index_codebase:

{
  "rootDir": "/path/to/project",
  "extensions": [".ts", ".js"],
  "ignorePatterns": ["node_modules", "dist"]
}

code_search:

{
  "query": "authentication middleware",
  "scope": "/path/to/project",
  "limit": 10,
  "followImports": true,
  "mode": "hybrid"
}

Returned results include file path, symbol name, kind, content, line numbers, score, and related chunks when import following is enabled.

Search Modes

Mode	Description	Best For
keyword	Exact token matching, prefix search, trigram fuzzy fallback	Known symbol names, fast, no model needed
vector	Cosine similarity over 384-dim embeddings	Conceptual queries, synonyms, semantic relatedness
hybrid	RRF fusion of keyword + BM25 + vector	General use — combines precision + recall

Keyword mode is the default and requires no setup. It scores exact name matches highest, then prefix matches, then keyword-in-content, then trigram fuzzy similarity.

Vector mode uses a local embedding model (Xenova/all-MiniLM-L6-v2, a small 384-dimensional model). On first run the model downloads automatically from HuggingFace and caches locally. Vector search finds semantically related code even when keywords don't overlap.

Hybrid mode runs all three strategies in parallel and fuses the rankings with Reciprocal Rank Fusion (RRF). Results include rrfScore (the fused score) and methodCount (how many strategies found the result). Items found by multiple methods rank higher, giving the best overall coverage.

Embedding Model

• Model: Xenova/all-MiniLM-L6-v2 (small, 384-dim)
• Backend: @xenova/transformers (ONNX runtime, runs locally)
• First run: model auto-downloads and caches to ~/.cache/huggingface/
• Fallback: if the local model fails to load, the embedder can fall back to OpenAI API (text-embedding-3-small) when OPENAI_API_KEY is set

Environment Variables

Variable	Default	Description
EMBEDDER_MODE	local	local uses Xenova transformer; api forces OpenAI API
OPENAI_API_KEY	—	Fallback API embedder key (optional)
VOYAGE_API_KEY	—	Alternative API embedder key (optional)
TRANSFORMERS_OFFLINE	—	Set to 1 to use only cached model, skip download

CRDT Multi-Agent State

The shared_state MCP tool gives agents a conflict-free shared state layer.

Supported CRDTs

Type	Merge Semantics	Good For
g-counter	max-per-node counter merge	token/request tracking
lww-register	last-writer-wins by timestamp	status/assignment
or-set	observed-remove set	shared findings or artifacts

Example Operations

{ "op": "write", "key": "tokens", "type": "g-counter", "nodeId": "agent-1", "amount": 150 }
{ "op": "write", "key": "status", "type": "lww-register", "nodeId": "agent-1", "value": "analyzing" }
{ "op": "write", "key": "findings", "type": "or-set", "nodeId": "agent-1", "action": "add", "element": "Issue in auth.ts:42" }
{ "op": "read", "key": "findings" }
{ "op": "snapshot" }
{ "op": "merge", "snapshot": { "entries": {} } }

This is useful when multiple coding or review agents need to coordinate without central locking.

Integrations

OpenCode

Configure OpenCode to treat the gateway as an OpenAI-compatible provider.

{
  "provider": {
    "llm-gateway": {
      "name": "LLM Gateway",
      "api": "openai",
      "apiKey": "env:LLM_GATEWAY_TOKEN",
      "baseURL": "https://llm-gateway.yourdomain.com/v1",
      "models": {
        "gateway-anthropic": {
          "name": "Anthropic via Gateway",
          "id": "claude-sonnet-4-20250514",
          "contextWindow": 200000,
          "maxOutput": 8192
        },
        "gateway-groq": {
          "name": "Groq via Gateway",
          "id": "llama-3.3-70b-versatile",
          "contextWindow": 128000,
          "maxOutput": 4096
        }
      }
    }
  }
}

export LLM_GATEWAY_TOKEN="your-gateway-auth-token"
opencode

GHAGGA

GHAGGA can use the bridge as a provider.

1. Select LLM Gateway in the GHAGGA dashboard.
2. Enter the gateway base URL.
3. Enter the gateway bearer token.
4. Pick a model.

Typical review modes routed through the gateway:

• simple
• workflow
• consensus

Any OpenAI-Compatible Tool

General settings:

Setting	Value
Base URL	https://llm-gateway.yourdomain.com/v1
API Key	your LLM_GATEWAY_AUTH_TOKEN

Works with LangChain, LlamaIndex, Cursor, Continue, and any HTTP client that can call /v1/chat/completions.

LangChain Python example:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="https://llm-gateway.yourdomain.com/v1",
    api_key="your-gateway-token",
    model="claude-sonnet-4-20250514",
)

response = llm.invoke("Explain quicksort")
print(response.content)

LangChain TypeScript example:

import { ChatOpenAI } from '@langchain/openai';

const llm = new ChatOpenAI({
  configuration: {
    baseURL: 'https://llm-gateway.yourdomain.com/v1',
  },
  apiKey: 'your-gateway-token',
  model: 'claude-sonnet-4-20250514',
});

const response = await llm.invoke('Explain quicksort');

Docker Deployment

Docker Compose

services:
  llm-gateway:
    build: .
    ports:
      - "3456:3456"
    volumes:
      - llm-data:/root/.llm-gateway
    environment:
      - LLM_GATEWAY_PORT=3456
      - LLM_GATEWAY_AUTH_TOKEN=your-secure-token-here
      - LLM_GATEWAY_MASTER_KEY=your-64-char-hex-key
volumes:
  llm-data:

docker compose up -d

Docker Build and Run

docker build -t llm-gateway .

docker run -d \
  -p 3456:3456 \
  -v llm-data:/root/.llm-gateway \
  -e LLM_GATEWAY_AUTH_TOKEN="your-token" \
  -e LLM_GATEWAY_MASTER_KEY="your-64-char-hex-key" \
  llm-gateway

What the Image Includes

The Dockerfile currently installs:

• pnpm 9
• OpenCode CLI
• Claude Code CLI
• Gemini CLI
• Codex CLI
• Qwen CLI
• GitHub Copilot CLI

Coolify

1. Create a new service pointing at this repository.
2. Use the Dockerfile build pack.
3. Set environment variables such as LLM_GATEWAY_PORT, LLM_GATEWAY_AUTH_TOKEN, and optionally LLM_GATEWAY_MASTER_KEY.
4. Mount a persistent volume at /root/.llm-gateway.
5. Use /health for health checks.

MCP Server

The project runs as an MCP stdio server by default.

Primary MCP Tools

Tool	Description
llm_generate	Generate text with provider routing and fallback
llm_models	List available models
vault_store, vault_list, vault_delete	API key management
vault_store_file, vault_list_files, vault_delete_file	Auth-file management
code_search, index_codebase	Semantic code search
shared_state	CRDT shared state
list_groups, create_group, delete_group	Provider group management
usage_summary, usage_query	Cost and usage inspection
configure_circuit_breaker, circuit_breaker_stats	Provider failure-control tuning
discover_models	Trigger HuggingFace-enriched model discovery
approval_list, approval_approve, approval_deny	Approval-flow management (see Approval Flows)

PageIndex Conversation Tools

Seven additional static MCP tools (defined in src/pageindex/tools.ts) handle long-conversation pagination and reasoning-based navigation over stored conversation history:

Tool	Description
conversation_paginate	Paginate a stored conversation
conversation_get_page	Fetch a specific page
conversation_context	Retrieve context around a point in the conversation
conversation_navigate	Navigate between pages/sections
conversation_info	Summary/metadata for a conversation
conversation_find_relevant	Find the most relevant pages for a query
conversation_check_compaction	Check whether the conversation should be compacted

These are categorized as read tools, so they are available under both local-dev and restricted security profiles.

Claude Code Config

Add to ~/.config/claude/mcp.json:

{
  "mcpServers": {
    "llm-bridge": {
      "command": "mcp-llm-bridge"
    }
  }
}

For a local source checkout:

{
  "mcpServers": {
    "llm-bridge": {
      "command": "npx",
      "args": ["tsx", "/path/to/mcp-llm-bridge/src/index.ts"]
    }
  }
}

Claude Desktop Config

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "llm-bridge": {
      "command": "mcp-llm-bridge"
    }
  }
}

MCP stdio runs locally and does not use the HTTP bearer-token middleware.

Dynamic MCP Servers

The bridge supports loading external .mcp-server.js plugin files at runtime. This lets you extend the toolset without modifying the core codebase.

What It Is

Any .mcp-server.js file placed in the plugin directory is loaded at startup and its tools are registered alongside the static tools (30 at time of writing: 23 core tools plus 7 PageIndex conversation tools). Plugins export a McpServerDefinition object (or use the builder) with tools, resources, and prompts.

Enable

Set MCP_DYNAMIC_SERVERS=true:

export MCP_DYNAMIC_SERVERS=true
export MCP_SERVERS_DIR=./mcp-servers

Create a Plugin

Create a .mcp-server.js file in the plugin directory:

import { McpServerBuilder } from 'mcp-llm-bridge/mcp-builder';

export default new McpServerBuilder()
  .tool('greet', 'Say hello to someone', { name: { type: 'string' } }, async ({ name }) => {
    return { content: [{ type: 'text', text: `Hello, ${name}!` }] };
  })
  .build();

The builder validates naming conventions, schema completeness, and description quality. Tools are registered on the MCP server and appear in ListTools.

Directory

The default plugin directory is ./mcp-servers. Override with:

export MCP_SERVERS_DIR=./my-custom-plugins

Security

Dynamic tools are registered with the read category by default. This means they are:

• Allowed under local-dev and restricted profiles
• Blocked under the open profile (which only allows generate tools)

The enforcer applies the same category-based filtering to dynamic tools as it does to static tools.

Coexistence with Static Tools

Static tools (vault, search, generate, etc.) and dynamic tools appear together in the ListTools response. There is no namespacing — tool names must be unique across both sets. The approval flow and rate limiting apply uniformly to all tools.

Configuration

Core Environment Variables

Variable	Default	Description
LLM_GATEWAY_PORT	3456	HTTP server port
LLM_GATEWAY_DB_PATH	~/.llm-gateway/vault.db	SQLite vault path
LLM_GATEWAY_MASTER_KEY	auto-generated	64-char hex key, otherwise saved to ~/.llm-gateway/master.key
LLM_GATEWAY_AUTH_TOKEN	unset	Bearer token for HTTP routes
LLM_GATEWAY_AUTH_REQUIRED	unset	Force auth on or off explicitly
LLM_GATEWAY_SECURITY_PROFILE	local-dev	Security profile for MCP tool exposure

Optional Runtime Features

Variable	Effect
FALLBACK_STRATEGY=free-models	enables free-model fallback routing
FREE_MODEL_CATALOG=true	loads the free-model catalog at startup
LATENCY_ROUTING=true	enables latency-based routing
MAX_COMPARISON_COST_USD	caps comparison-service spending

Master Key Priority

1. LLM_GATEWAY_MASTER_KEY
2. existing ~/.llm-gateway/master.key
3. auto-generated new key written with mode 0600

If you lose the master key, stored credentials are unrecoverable. Back it up in production.

Bridge Config Path

~/.llm-gateway/bridge.yaml

If that file does not exist, bridge routing is disabled.

Security Profiles

Security profiles enforce trust-level-based access control on both MCP tools and HTTP endpoints. Three profiles are built-in:

Profile	Allowed Categories	Rate Limit	Sandbox
local-dev	all (destructive, read, generate, admin)	none	false
restricted	read + generate only	100 req / 15 min	false
open	generate only	10 req / 15 min	false

Configuration

Set via environment variable:

LLM_GATEWAY_SECURITY_PROFILE=restricted

Default is local-dev (backward compatible — no restrictions).

Each profile also carries a sandbox flag (default false). Today this flag is best understood as prepared infrastructure, not a guarantee of sandboxed runtime execution: it is exposed on the profile schema and through the admin API, and the repo includes a Docker/process sandbox runner under src/sandbox/, but the active runtime does not yet expose sandboxed execution tools or route normal tool execution through that runner. Also note that the helper falls back to plain process execution with a timeout when Docker is unavailable, so this should not be treated as complete containment.

HTTP Enforcement

Under restricted or open, the gateway blocks destructive HTTP endpoints (e.g., POST /v1/credentials) and returns:

{ "error": "Access denied: endpoint blocked by security profile", "code": "SECURITY_PROFILE_DENIED" }

Read endpoints (GET /v1/providers, GET /v1/models) remain open under restricted.

MCP Enforcement

Under non-local-dev profiles, ListTools returns only tools in the allowed categories. CallTool is authorized before execution. Rate limiting is applied per profile.

Approval Flows

Destructive MCP tools can be paused for explicit human approval when the security profile is not local-dev.

How It Works

1. Client calls a destructive tool (e.g., vault_store).
2. If approval is required, the gateway returns an approvalRequired payload with a requestId.
3. Admin reviews pending requests via GET /v1/approvals or approval_list MCP tool.
4. Admin approves or denies via POST /v1/approvals/:id/approve or approval_approve MCP tool.
5. Original tool executes only after approval.

Auto-Approve List

Read-only tools (file_read, search, list, vault_list) bypass approval automatically.

HTTP Endpoints

Endpoint	Method	Description
/v1/approvals	GET	List pending approval requests
/v1/approvals/:id/approve	POST	Approve a request
/v1/approvals/:id/deny	POST	Deny a request

MCP Tools

Tool	Description
approval_list	List pending requests
approval_approve	Approve by request ID
approval_deny	Deny by request ID

Three-Part Prompt

The three-part prompt pattern separates prompts into system (role/constraints), context (background data), and instruction (the actual task). Research shows measurable quality improvement, especially with smaller models.

HTTP API

Both /v1/generate and /v1/chat/completions accept the three fields:

curl -X POST http://localhost:3456/v1/generate \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "system": "You are a code reviewer.",
    "context": "We use Zod 4 and Hono.",
    "instruction": "Review this schema for edge cases."
  }'

Legacy flat prompt is still accepted and auto-detected when system/context/instruction are absent.

MCP Schema

The llm_generate tool exposes system, context, and instruction as optional fields alongside the legacy prompt:

{
  "system": "You are a helpful assistant.",
  "context": "The project uses TypeScript.",
  "instruction": "Explain strict mode benefits."
}

Enable/Disable

OPTIMIZE_MESSAGES_ENABLED=true   # default: true

RTK Output Compression

RTK-style compression strips redundant content from tool call results before passing them to LLMs. This saves token budget on large structured outputs.

Strategies

1. Filter — remove noise fields (created_at, id, etag, etc.)
2. Group — merge repeated similar entries into count + sample
3. Truncate — enforce max length on string values
4. Deduplicate — remove exact-duplicate array entries

Configuration

ENABLE_OUTPUT_COMPRESSION=true   # default: true

Analytics Endpoint

curl http://localhost:3456/v1/compression/stats \
  -H 'Authorization: Bearer YOUR_TOKEN'

Response:

{ "totalCalls": 42, "compressedCalls": 42, "avgRatio": 0.65, "totalSavingsChars": 15200 }

Local LLM Offloading

Offloadable tasks (summarization, formatting, classification) can be routed to local runtimes (Ollama, LM Studio) instead of cloud providers, cutting API token cost on those deterministic tasks. (The src/local-llm/ module documents an 86–95% design target for token savings on boilerplate tasks; this is a design goal, not a measured benchmark.)

Environment Variables

Variable	Default	Description
LOCAL_LLM_ENABLED	false	Enable local LLM routing
OLLAMA_URL	http://localhost:11434	Ollama API endpoint
LM_STUDIO_URL	http://localhost:1234	LM Studio API endpoint

Detection

At startup, the gateway probes both backends. Models are listed at:

curl http://localhost:3456/v1/local/models \
  -H 'Authorization: Bearer YOUR_TOKEN'

Fallback

If the local LLM fails or the task is not offloadable, the gateway falls back to cloud providers automatically and emits a metric.

MCP Tool

Tool	Description
local_llm_generate	Generate via local LLM with offload detection

Model Routing

Model routing adds task-aware provider selection that classifies each prompt and routes it based on configured rules, preferred endpoint order, cost tiers, and observed quality feedback.

What It Does

• Classifies incoming prompts into runtime task types such as code-review, large-context, fast-completion, summarization, and translation
• Matches the task against routing rules defined in model-routing.json
• Tries preferred endpoints in rule order while enforcing the configured cost cap
• Falls back to more expensive endpoints if quality drops below threshold
• Learns from feedback — records success/failure per endpoint+task for adaptive routing

Enable

MODEL_ROUTING_ENABLED=true

When enabled, the precedence stack becomes:

1. Session stickiness
2. Group-based routing
3. ModelRouter (task-aware selection)
4. Local-LLM offloading (only if ModelRouter is disabled or returns no match)
5. Standard resolution (model match → provider preference → API before CLI)
6. Latency reordering

Configuration

Create model-routing.json in the project root. The gateway loads it at startup.

{
  "enabled": true,
  "endpoints": [...],
  "rules": [...],
  "defaultEndpoint": "opencode-cli-default",
  "qualityThreshold": 0.7,
  "qualityWindowSize": 50
}

Field	Type	Description
enabled	boolean	Whether model routing is active
endpoints	array	Available model endpoints with cost tier and capabilities
rules	array	Task-to-endpoint routing rules (first match wins)
defaultEndpoint	string	Fallback endpoint ID when no rule matches
qualityThreshold	number	Minimum acceptable quality rate (0–1)
qualityWindowSize	number	Number of recent requests to track per endpoint+task

Endpoint fields:

Field	Type	Description
id	string	Unique endpoint identifier
providerId	string	Provider ID (e.g., anthropic, openai, opencode-cli)
model	string	Model ID for API calls
costTier	string	free, cheap, standard, or expensive
capabilities	array	Capability tags (e.g., chat, code, reasoning)
maxTokens	number	Maximum context window in tokens

Rule fields:

Field	Type	Description
id	string	Unique rule identifier
taskType	string	One of large-context, code-review, fast-completion, default, boilerplate, commit-message, format-conversion, style-check, summarization, translation, not-offloadable, or *
preferredEndpoints	array	Ordered list of endpoint IDs to try
maxCostTier	string	Most expensive tier allowed for this task
allowFallback	boolean	Whether to fall back to default endpoint if all preferred fail

Example Task-to-Endpoint Mappings

Task Type	Preferred Endpoints	Cost Cap
code-review	Claude Sonnet → GPT-4.1	expensive
large-context	Claude Sonnet → GPT-4.1	expensive
fast-completion	GPT-4.1-mini → OpenCode CLI	standard
summarization	GPT-4.1 → Claude Sonnet	expensive
* (default)	OpenCode CLI → GPT-4.1-mini	standard

Coexistence with Local-LLM Offloading

Local-LLM offloading and model routing work together with clear precedence:

• ModelRouter runs first. If it selects an endpoint, that provider is promoted to the top of the candidate list.
• Local-LLM offloading runs only when ModelRouter is disabled or returns no match. This prevents conflicts: explicit routing rules always beat heuristic offloading.

If you want local models in your routing mix, register them as endpoints with "costTier": "free" and include them in rule preferredEndpoints.

Example File

See model-routing.example.json in the repository root for a full template with multiple endpoints and routing rules.

HF Auto-Discovery

At startup (when enabled), the gateway scans local backends and enriches detected models with HuggingFace metadata (tags, pipeline type, recommended tasks).

Configuration

AUTO_DISCOVER_MODELS=true   # default: false
HF_TOKEN=hf_xxxxxxxxxx       # optional, for private repos

Admin Endpoint

Trigger discovery on demand:

curl -X POST http://localhost:3456/v1/admin/discover \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{ "hfToken": "optional-override" }'

Response:

{
  "ok": true,
  "backendsScanned": ["ollama", "lm-studio"],
  "models": [...],
  "enrichedCount": 3,
  "unenrichedCount": 1
}

Cache

Enriched metadata is persisted to SQLite (hf_model_cache table) so subsequent startups are fast even without HF API access.

Architecture

Clients (GHAGGA, OpenCode, curl, LangChain, any OpenAI-compatible tool)
    |
    |  POST /v1/chat/completions  |  POST /v1/generate  |  MCP stdio
    v
+-------------------------------------------------------------------+
|                    MCP LLM Bridge (Hono + MCP)                    |
|                                                                   |
|  HTTP Server                       MCP Server                     |
|  - /v1/chat/completions            - llm_generate                 |
|  - /v1/generate                    - vault_*                      |
|  - /v1/models                      - code_search                  |
|  - /v1/providers                   - index_codebase               |
|  - /v1/credentials CRUD            - shared_state                 |
|  - /v1/files CRUD                  - usage_*                      |
|  - /v1/groups CRUD                 - circuit_breaker_*            |
|  - /metrics /health                - group tools                  |
|  - /v1/compression/stats           - approval_*                   |
|  - /v1/local/models                - local_llm_generate           |
|  - /v1/admin/discover              - discover_models              |
+-------------------------------------------------------------------+
|  Bridge routing         | Context compression | Code search        |
|  Provider groups        | Cost tracking       | CRDT state         |
|  Security profiles      | Approval flows      | Local LLM          |
|  HF discovery           | Three-part prompt   | Output compression |
+-------------------------+---------------------+--------------------+
| Router (model -> provider)       | Vault (AES-256-GCM + SQLite)   |
+-------------------------+---------------------+--------------------+
    |                                                  |
    v                                                  v
  API providers                                   CLI providers
  Anthropic, OpenAI, Google, Groq, OpenRouter     OpenCode, Claude,
                                                   Gemini, Codex, Qwen, Copilot

Design Notes

• Hono keeps the HTTP layer small and fast.
• better-sqlite3 keeps the vault single-file and operationally simple.
• SQLite WAL mode improves concurrent read behavior.
• API providers are preferred before CLI providers unless bridge logic says otherwise.
• Vault writes use upsert semantics for repeatable automation.
• CLI adapters materialize auth files into temp homes and clean them up in finally blocks.
• Bridge routing is intentionally optional and file-driven.
• Code search stays in-memory for speed and freshness.
• CRDTs reduce coordination pain for parallel agent workflows.

Experimental Modules

• src/acp/ — Agent Client Protocol implementation (server.ts, translator.ts, types.ts).
  Present in the repo but not wired into the active runtime. There is no import path from src/index.ts, no active HTTP/stdio ACP transport, and no live MCP tool-execution bridge yet. Treat it as a tested protocol prototype that still needs a dedicated ACP integration sprint.

• src/sandbox/ — Docker/process sandbox runner.
  The sandbox flag now exists in security profiles, but the runtime still does not expose sandboxed execution tools like execute_code or shell_command. In other words: the infrastructure is prepared, but the feature is not complete.

Session Systems

The gateway now uses SessionManager for both session-affinity concerns:

1. Router sticky sessions (SessionManager.pinRouterStickySession) — Pins a specific clientId + model to a provider/key with TTL-based expiry.

2. Group/API sessions (src/session/session-manager.ts) — Manages session affinity for multi-turn conversations and dashboard metrics.

They are separate by design inside the same manager instance: router stickiness handles provider selection, while group/API sessions handle conversation continuity.
Do not conflate the two entry kinds.

GET /v1/admin/sessions reports them separately for that reason:

• routerStickySessions comes from SessionManager router-sticky entries and reflects the pins the Router actually uses at request time.
• groupSessions comes from SessionManager and reflects group-level session affinity metrics.
• The endpoint includes a note explaining the split so the dashboard does not imply a single shared session pool.

Security

• AES-256-GCM encryption for stored keys and auth files
• constant-time bearer-token comparison
• master key file stored with mode 0600
• config directory created with mode 0700
• credentials are never returned raw from listing endpoints
• temp auth files are cleaned up after CLI invocations
• minimum 32-character auth token requirement
• public /health endpoint for safe monitoring

Development

pnpm run dev
pnpm run serve
pnpm run start
pnpm test
pnpm run typecheck
pnpm run build

Scripts

Script	Command	Description
start	tsx src/index.ts	MCP stdio mode
dev	tsx watch src/index.ts	local development
serve	tsx src/index.ts serve	HTTP server and dashboard
test	node --import tsx --import ./test/setup/inject-require.mjs --test test/*.test.ts test/**/*.test.ts	test suite
build	tsup src/index.ts --format esm --dts	production build
typecheck	tsc --noEmit	TypeScript checking

Requirements

• Node.js 22+
• pnpm 9+

License

MIT, as declared in package.json ("license": "MIT"). Note: a standalone LICENSE file is not yet committed to the repository.