Llama Server

MCP server bridging Claude Code to local llama.cpp. Run local LLMs alongside Claude for experimentation, testing, and cost-effective inference.

Requirements

• Node.js 18+
• llama.cpp with llama-server built
• A GGUF model file

Installation

npm install llama-mcp-server

Or clone and build from source:

git clone https://github.com/ahays248/llama-mcp-server
cd llama-mcp-server
npm install
npm run build

Configuration

Configure via environment variables:

Variable	Description	Default
LLAMA_SERVER_URL	URL of llama-server	http://localhost:8080
LLAMA_SERVER_TIMEOUT	Request timeout in ms	30000
LLAMA_MODEL_PATH	Path to GGUF model file	(none)
LLAMA_SERVER_PATH	Path to llama-server binary	llama-server

Usage with Claude Code

Option 1: Plugin Installation (Recommended)

Due to a known bug in Claude Code, non-plugin MCP servers may connect but not expose their tools. The workaround is to install llama-mcp-server as a plugin via a local marketplace.

Step 1: Create the marketplace structure

llama-marketplace/
├── .claude-plugin/
│   └── marketplace.json
└── plugins/
    └── llama/
        ├── .claude-plugin/
        │   └── plugin.json
        └── .mcp.json

Step 2: Create marketplace.json

// llama-marketplace/.claude-plugin/marketplace.json
{
  "name": "llama-marketplace",
  "description": "Local marketplace for llama.cpp MCP plugin",
  "owner": {
    "name": "Your Name"
  },
  "plugins": [
    {
      "name": "llama",
      "description": "llama.cpp MCP server for local LLM inference",
      "source": "./plugins/llama"
    }
  ]
}

Step 3: Create plugin.json

// llama-marketplace/plugins/llama/.claude-plugin/plugin.json
{
  "name": "llama",
  "version": "0.1.0",
  "description": "llama.cpp MCP server for local LLM inference"
}

Step 4: Create .mcp.json

// llama-marketplace/plugins/llama/.mcp.json
{
  "mcpServers": {
    "llama": {
      "command": "npx",
      "args": ["-y", "llama-mcp-server"],
      "env": {
        "LLAMA_SERVER_URL": "http://localhost:8080",
        "LLAMA_MODEL_PATH": "/path/to/your/model.gguf",
        "LLAMA_SERVER_PATH": "/path/to/llama-server"
      }
    }
  }
}

Step 5: Install the plugin

# Add the local marketplace
claude plugin marketplace add /path/to/llama-marketplace

# Install the plugin
claude plugin install llama@llama-marketplace

# Restart Claude Code

After restart, tools will appear as mcp__plugin_llama_llama__*.

Option 2: Direct MCP Configuration

Note: This method may not work due to the bug mentioned above. If tools don't appear after adding the server, use Option 1.

Add to your Claude Code MCP configuration:

claude mcp add llama -e LLAMA_SERVER_URL=http://localhost:8080 -e LLAMA_MODEL_PATH=/path/to/model.gguf -e LLAMA_SERVER_PATH=/path/to/llama-server -- npx -y llama-mcp-server

Or add manually to ~/.claude.json:

{
  "mcpServers": {
    "llama": {
      "command": "npx",
      "args": ["-y", "llama-mcp-server"],
      "env": {
        "LLAMA_SERVER_URL": "http://localhost:8080",
        "LLAMA_MODEL_PATH": "/path/to/your/model.gguf",
        "LLAMA_SERVER_PATH": "/path/to/llama-server"
      }
    }
  }
}

Tools

Server Tools

Tool	Description
llama_health	Check if llama-server is running and get status
llama_props	Get or set server properties
llama_models	List available/loaded models
llama_slots	View current slot processing state
llama_metrics	Get Prometheus-compatible metrics

Token Tools

Tool	Description
llama_tokenize	Convert text to token IDs
llama_detokenize	Convert token IDs back to text
llama_apply_template	Format chat messages using model's template

Inference Tools

Tool	Description
llama_complete	Generate text completion from a prompt
llama_chat	Chat completion (OpenAI-compatible)
llama_embed	Generate embeddings for text
llama_infill	Code completion with prefix and suffix context
llama_rerank	Rerank documents by relevance to a query

Model Management Tools

Tool	Description
llama_load_model	Load a model (router mode only)
llama_unload_model	Unload the current model (router mode only)

LoRA Tools

Tool	Description
llama_lora_list	List loaded LoRA adapters
llama_lora_set	Set LoRA adapter scales

Process Control Tools

Tool	Description
llama_start	Start llama-server as a child process
llama_stop	Stop the llama-server process

Example: Starting llama-server and Running Inference

User: Start llama-server with my local model

Claude: I'll start llama-server for you.
[Uses llama_start tool with model path]

User: Generate a haiku about coding

Claude: Let me use the local model for that.
[Uses llama_complete tool]

Result:
Lines of code cascade
Through the silent morning hours
Bugs flee from the light

Development

# Run tests
npm test

# Type check
npm run typecheck

# Build
npm run build

# Watch mode for development
npm run dev

Troubleshooting

Tools don't appear in Claude Code

Symptom: Server shows "Connected" in claude mcp list but no llama_* tools are available.

Cause: Known bug in Claude Code where non-plugin MCP servers don't expose tools (#12164).

Solution: Use the plugin installation method (Option 1 above).

HTTP 501 errors for certain tools

Some tools require specific server configurations:

Tool	Requirement
llama_metrics	Start llama-server with --metrics flag
llama_embed	Start llama-server with --embedding flag or use an embedding model
llama_infill	Use a model with fill-in-middle support (e.g., CodeLlama, DeepSeek Coder)
llama_rerank	Use a reranker model
llama_load_model / llama_unload_model	llama-server must be in router mode

Connection refused errors

Symptom: Cannot connect to llama-server at http://localhost:8080

Solutions:

1. Use llama_start to start the server, or
2. Start llama-server manually: llama-server -m /path/to/model.gguf
3. Check that LLAMA_SERVER_URL matches where llama-server is running

WSL/Windows path issues

When running in WSL, ensure paths use Linux format:

• ✓ /home/user/models/model.gguf
• ✗ C:\Users\user\models\model.gguf

License

MIT