Local Llm

English · 繁體中文

I gave Claude Code an intern. It works for free and lives on my laptop.

❝ Senior model decides. Local model types. ❞

Why I built this

One afternoon I watched Claude Code reach for a frontier model — the good one, the one my credit card feels — to generate an __init__.py. Then a pytest skeleton. Then a throwaway first draft of a function I was about to rewrite anyway.

That's when it clicked: I was paying senior-engineer rates for a senior engineer's typing. The thinking — reading my repo, planning the change, reviewing the diff, deciding what's safe to ship — that's what the expensive model is for. The boilerplate is not.

So I did the lazy thing. I already had LM Studio running on my Mac, doing nothing between prompts. Why not let the cheap model on my own machine handle the grunt work, and keep the frontier brain for the calls that actually matter?

One snag: my Mac has 24GB of RAM, and local models cheerfully ate all of it until the whole session froze. So the first thing I built wasn't the clever delegation — it was a dumb RAM valve that refuses a local call when memory runs low. Everything else grew around that.

That's all local-llm-mcp is: a small MCP server that hands your coding agent an intern. The intern only ever returns text — it can't read your repo, edit a file, or run a command — and the senior model reviews everything before a single line changes.

The whole idea in one line: senior model decides, local model types.

Try it in three commands

git clone https://github.com/HenryLinyy/local-llm-mcp
cd local-llm-mcp
bash setup.sh        # venv + registers with Claude Code & Codex + smoke test

Then open a fresh Claude Code or Codex session. No API keys needed — local backends work out of the box. setup.sh even picks a RAM threshold based on your machine.

The mental model

Claude Code / Codex   the senior — reads the repo, plans, edits, runs tests, reviews
local-llm-mcp         hands one bounded task to the intern over MCP
Ollama / LM Studio    the intern — drafts boilerplate, tests, docs, summaries, for $0

A local 7B model costs $0 per token. My frontier agent does not. The rest is arithmetic — and I'll get to the arithmetic honestly below.

The intern can't touch your repo. That's not a missing feature; it's the point. I never have to trust the cheap model with anything that matters — worst case, it hands back a bad draft and the senior throws it out.

What I hand the intern / what stays on my desk

My rule of thumb: if a wrong answer is cheap to catch, delegate it. If it's expensive to catch, don't.

How I actually use it

There's no API to learn. I just tell the senior to delegate, in plain English:

Use ask_local_model with backend="ollama" to draft a pytest suite for this module.
Don't apply it — review it first, then edit the repo yourself.

Call local_status. If a local model is up, use it for boilerplate.
Otherwise fall back to backend="deepseek".

More of my go-to prompts are in examples/claude-code-prompts.md.

Bring any intern you like

Local backends need nothing but a running server. Cloud backends are optional — they read their key from an env var or keys.json, never from source.

Every URL and default model is env-overridable. Running something exotic? Add a custom backend — no Python required.

The tools it exposes

The intern is on a short leash

I learned this the hard way, so you don't have to. Two guards:

1. RAM valve — local calls are refused when free memory drops below LOCAL_LLM_MIN_FREE_GB. This is the feature that exists because I froze my own machine one too many times.
2. Exclusive backend — when a heavy local server (ds4 by default) is up, other local backends stand down instead of fighting over memory.

Tune them live, no restart: set_guard(min_free_gb=8), set_guard(exclusive_backend="none").

And secrets never enter git — keys.json, config.json, and custom_backends.json are all gitignored; keys load from env vars or a chmod 600 file. Cloud backends skip the RAM guard, but they send your prompts to a third party and may cost money — read SECURITY.md before pointing one at proprietary code.

So how much does it save?

Here's where most READMEs lie to you with a big number. I'm not going to.

The honest answer is it depends entirely on your workload, so instead of inventing a percentage I shipped a harness to measure your own:

python scripts/benchmark.py --backend ollama --model qwen2.5-coder:7b --out results.jsonl

Compare premium-only vs. delegated mode with BENCHMARK.md and the runbook. Trust your numbers — not mine, and definitely not a number a README made up to get you to star it.

set_backend(name="my_qwen", base_url="http://localhost:9000/v1", default_model="qwen3-coder", local=1, protocol="openai")

python3 -m venv .venv
.venv/bin/python -m pip install -e .

# Claude Code
claude mcp add local-llm -s user -e LOCAL_LLM_MIN_FREE_GB=16 -- "$PWD/.venv/bin/python" "$PWD/server.py"

# Codex
codex mcp add local-llm --env LOCAL_LLM_MIN_FREE_GB=16 -- "$PWD/.venv/bin/python" "$PWD/server.py"

python -m unittest discover -s tests -v
python scripts/smoke_test.py

FAQ

Does the intern touch my files? Never. It returns text; every edit goes through the senior (your main agent).

Do I need a beefy GPU? No. 7B coder models run on modest hardware, and the RAM valve keeps you from OOMing. No local model handy? Point the intern at a cheap cloud backend.

Is this a fusion model or an autonomous agent? Neither. It's a delegation layer — a tool your existing agent calls.

Why not just switch my agent to a cheap model entirely? Because I want the frontier model's judgment and the cheap model's typing. This keeps both.

Windows / Linux? The server is cross-platform (the RAM guard reads vm_stat on macOS, /proc/meminfo on Linux). The shell helpers are macOS/zsh-flavored.

My other experiments in this space

• qwable — a local multi-model gateway and agent runtime for Codex & Claude Code on Apple Silicon.
• Conclava — a council of local LLMs with task-aware routing and multi-model deliberation.

Contributing

Issues and PRs welcome — see CONTRIBUTING.md. If this saved you some tokens, a ⭐ tells me it was worth open-sourcing.

License

MIT — see LICENSE. Built by someone who got tired of paying frontier prices for import os.