Local Llm Delegation

An optimized Model Context Protocol (MCP) server designed to dramatically reduce premium token costs by intelligently delegating low-complexity tasks to high-performance local LLMs via LiteLLM.

This server allows you to leverage massive local context windows for "surgical" development tasks (refactoring, unit tests, documentation) while keeping your primary LLM focused on complex architecture and reasoning.

🚀 Key Features

• LiteLLM Integration: Seamlessly connect to Ollama, vLLM, LM Studio, Anthropic, or any OpenAI-compatible provider.
• Externalized Configuration: Fine-tune model parameters (temperature, top_p, max_tokens) via config.yaml.
• Customizable Prompts: Define your own task templates and system messages in prompts.yaml.
• Token Optimization: Automatically offload simple tasks to local models like Qwen 2.5/3.5 Coder.
• Usage Tracking: Monitor your savings with built-in usage logging and statistics.

🛠️ Requirements

• Local LLM Runner (e.g., Ollama, vLLM, LM Studio)
• Python 3.13+

📦 Installation

1. Clone the repository:

   git clone https://github.com/mrrodriguez/local-llm-delegation-mcp.git
   cd local-llm-delegation-mcp
   

2. Set up environment variables:

   cp .env.example .env
   # Edit .env to match your local setup
   

3. Choose your installation method:

   Option A: Global CLI (Recommended for simplicity) Install as a global tool. This adds the local-llm-delegator command to your PATH.

   pip install .
   # Or using pipx (preferred)
   pipx install .
   

   Option B: Local Environment (Recommended for development) Use a virtual environment and point your MCP client directly to a startup script.

   python -m venv .venv
   source .venv/bin/activate
   pip install .
   

⚙️ Configuration

The server uses a flexible configuration system. Settings are loaded from config.yaml and prompts.yaml located in the project root, with key overrides available via Environment Variables (or a .env file).

Configuration Hierarchy

1. Base Defaults: Hardcoded in config.py.
2. YAML Files: config.yaml (model settings) and prompts.yaml (task templates).
3. Environment Variables: Overrides specific settings like LOCAL_MODEL_NAME and OPENAI_BASE_URL.

1. Quick Start: Environment Variables

The fastest way to switch models without editing files:

# In your .env file or shell environment
LOCAL_MODEL_NAME=ollama/llama3:8b  # Overrides config.yaml
OPENAI_BASE_URL=http://localhost:11434/v1

2. Model Tuning (config.yaml)

Update the name to match your local provider's format (e.g., ollama/qwen2.5-coder). The provided template is optimized for high-end hardware (e.g., Apple M-series Max).

model:
  name: "openai/Qwen3-Coder-30B-A3B-Instruct-MLX-8bit" # Example: OMLX/MLX
  temperature: 0.1
  max_tokens: 32768
  extra_params:
    num_ctx: 65536
    num_predict: 32768
    top_p: 0.80

Note: The name must follow the LiteLLM Provider Format.

3. Custom Tasks (prompts.yaml)

Define your own tools by adding entries to prompts.yaml. Each top-level key becomes a sub-task for the query_local_llm_with_context tool.

🔌 Integration

You can add this server to your preferred client using either the global command or a direct path to the project.

1. Gemini Code CLI

If installed globally (Option A):

gemini mcp add local-llm-delegator local-llm-delegator --trust

If using a direct script or path (Option B):

gemini mcp add local-llm-delegator /path/to/your/start-script.sh --trust

2. Claude Code

Add to your ~/.claude.json:

"mcpServers": {
  "local-llm-delegator": {
    "command": "local-llm-delegator" // If installed globally; otherwise any wrapper script you have to start it
  }
}

Note: If using a custom startup script (like for OMLX), set the command to the absolute path of your script.

📊 Usage Tracking

The server logs usage to mcp_usage.jsonl. View stats via:

• get_local_llm_usage_stats (MCP tool)
• show-stats (CLI command - coming soon) or python show_stats.py

📜 License

MIT