Llm D Bench

Give Claude Code the ability to benchmark llm-d from plain English. Point it at this server and it can probe a cluster, propose a benchmark plan you approve, deploy an llm-d stack, run the benchmark, and explain the results, inside the same security sandbox and approval gates as the llm-d-benchmarking-agent app.

Supported now: the claude-agent-sdk provider (no API key) wired into Claude Code (the CLI), the only path the installer sets up and verifies. The server speaks standard MCP, so other providers (anthropic, openai) and clients (Claude Desktop, Cursor, VS Code, OpenAI Codex CLI) are planned for a future release.

It is the agent's toolset re-exposed over the Model Context Protocol: 35 tools, 5 workflow prompts, and the agent's entire knowledge base (60+ playbooks and heuristics) as readable resources, so a generic agent behaves like a benchmarking expert rather than a blank slate.

How it fits together

This repo is the thin MCP adapter (~500 lines: transport + approval/event adapters + the knowledge-exposure surface). The engine (the 35 tools, the security allowlist, and the knowledge/ playbooks) lives in the llm-d-benchmarking-agent repo, which the installer clones at its latest main and installs into the same virtualenv. The engine must run from a real checkout (it reads knowledge/, the allowlist, and the read-only sibling repos from disk at runtime), which is why it is not a pip dependency.

Install (one command)

The installer fetches the engine + sibling repos, builds one venv with both packages, configures the claude-agent-sdk provider, and registers the server with Claude Code (or prints the config to paste yourself):

bash <(curl -fsSL https://raw.githubusercontent.com/TalBenAmii/llm-d-bench-mcp/main/scripts/install.sh)

Prefer to clone first? The same script runs from inside a checkout:

git clone https://github.com/TalBenAmii/llm-d-bench-mcp.git
cd llm-d-bench-mcp
./scripts/install.sh

It is idempotent (safe to re-run); the only prerequisite is being logged in to the claude CLI (no API key — see Requirements; the installer offers to install the CLI if it's missing). Because the engine app lands in the same venv you also get its web UI for free — the installer's final message prints the launch line (./scripts/run.sh --open → http://127.0.0.1:8000).

What your agent gets

Tools (35)

Group	What your agent can do	Examples
Sense & ground (read-only, auto-run)	Inspect the environment, GPUs, catalog, docs, knowledge	probe_environment, advise_accelerators, list_catalog, discover_stack, search_knowledge, read_knowledge, fetch_key_docs, read_repo_doc
Plan before you spend	Map a use case to a validated plan; check it fits	propose_session_plan, check_capacity, estimate_run_duration, write_and_validate_config, generate_doe_experiment
Deploy & run (approval-gated)	Set up repos, run the CLI, orchestrate Jobs & sweeps	ensure_repos, run_setup, execute_llmdbenchmark, orchestrate_benchmark_run, orchestrate_sweep, provision_hf_secret
Make sense of results (read-only)	Parse reports, compare runs/harnesses, track trends	locate_and_parse_report, analyze_results, compare_reports, compare_harness_runs, aggregate_runs, result_history
Observe & manage	Readiness checks, live cluster metrics, run management	check_endpoint_readiness, observe_run_metrics, manage_orchestrated_runs, cancel_run
Trust & reproduce	Provenance bundles, reproduce a run	export_run_bundle, reproduce_run

Numbers are only ever reported from a validated Benchmark Report v0.2, never scraped from logs or invented.

Workflow prompts (5)

Entry points that drop your agent into the right playbook:

Prompt	Arguments	What it sets up
benchmark_this_model	model?, goal?, slo?	The full interview → preconditions → plan → run → explain workflow
pick_deploy_path	model?, accelerator?	Choosing a deploy path + accelerator guidance
interpret_this_report	report_path?	Parsing and explaining a benchmark report
design_a_sweep	objective?	Designing a design-of-experiments sweep
goal_seek_to_slo	slo	Iterative sweep rounds toward an SLO at best goodput

Resources & instructions

Every knowledge file is exposed as a doc://knowledge/<name> resource, so your agent can read the same playbooks the standalone agent reasons over. The server also advertises a role/workflow preamble in its MCP instructions ("probe first, ground in docs, propose a plan, run only with approval") that capable clients fold into their system prompt.

Manual config (Claude Code)

The installer does this for you. By hand, register the console entry point by its absolute path in the agent project's venv (the installer builds everything into that one venv):

claude mcp add llm-d-bench -s user -- /ABS/PATH/llm-d-benchmarking-agent-project/.venv/bin/llm-d-bench-mcp
# verify:  claude mcp list   (or /mcp inside a session)

A gated-model HF_TOKEN is optional; add it with -e HF_TOKEN=hf_xxx. The agent project's .env already carries the LLM provider config and is loaded regardless of how the server is launched. The module form (.../.venv/bin/python -m llm_d_bench_mcp) works too once both packages are installed in the venv. Smoke-test it without a client using the official inspector: npx @modelcontextprotocol/inspector /ABS/PATH/.venv/bin/llm-d-bench-mcp.

Requirements & scope

• Python ≥ 3.11 and git (the installer handles the venv via uv, or python3 -m venv).
• LLM provider: claude-agent-sdk. No API key; authenticated via your claude CLI login.
• Client: Claude Code (the CLI).
• No cluster needed for the advisory tools and knowledge resources. The deploy/run/orchestrate tools need a reachable Kubernetes cluster + kubeconfig (and HF_TOKEN for gated models).
• The engine repo and its read-only siblings (llm-d, llm-d-benchmark, llm-d-skills) — the installer clones them all automatically.

Security & scope

• stdio / local single-user only. The server has no network listener and no per-caller auth; it acts with your own kubeconfig. This is acceptable only for local use. HTTP/remote/shared transport is deliberately deferred, and "who may connect, whose credentials, what blast radius" become blocking questions before any such mode.
• Approval is re-homed to your client. Every tool call is gated by your MCP client's own tool-permission prompt; the richer SessionPlan approval uses MCP elicitation where the client supports it (with a graceful fallback otherwise). Nothing mutating runs without your say-so; never a silent auto-approve.

Design of record and rationale: DESIGN.md. The engine / full agent: llm-d-benchmarking-agent.