ToolSmith Agent Server

CI

A multi-tool task agent whose one tool layer (file search · read-only SQLite/text-to-SQL · safe calculator · report writer) is driven three ways from a single source of truth:

1. a deterministic mock brain → 100% offline, zero secrets, CI-gated;
2. an optional Groq free-tier model (one env var);
3. a real MCP server so Claude Code / Claude Desktop / Cursor can reason over the exact same tools — real NL→tool reasoning, for free.

No paid API is needed to prove the engineering. The mock makes the whole agent reproducible and testable offline; the MCP path shows a real model driving the identical tools + guardrails at zero cost.

Results (offline mock brain, python -m eval.simple_eval)

Metric	Score
Task Success Rate	6/6 = 1.00
Tool-Trajectory accuracy	6/6 = 1.00
Self-correction / recovery (injected tool errors)	2/2 = 1.00
Unit tests (guardrails, loop, MCP parity, matcher)	21 passing

Trajectory is asserted, not just the final answer — a right answer via the wrong tool still fails. See the honesty notes below on what these numbers do and don't mean.

What one run looks like

▶ TASK (mock): List the top 3 products by revenue and save a report
  ├─ step 0 · db_schema()
  │    ↳ CREATE TABLE products ( id INTEGER PRIMARY KEY, name TEXT ... )
  ├─ step 1 · query_db(sql="SELECT p.name, SUM(s.amount) AS revenue ...")
  │    ↳ name | revenue  Gadget | 600.0  Widget | 375.0  Gizmo | 90.0
  ├─ step 2 · write_report(filename="top_products.md", ...)
  │    ↳ Wrote 79 chars to reports/top_products.md.
  └─ FINAL: Saved top_products.md. Gadget leads with 600.0 in revenue.

Self-correction (the count_orders task queries a table that doesn't exist): db_schema → query_db(orders) → ERROR → query_db(sales) → "There are 5 sales records."

Architecture — one tool layer, three brains, two surfaces

                         tools/  ← THE single source of truth (REGISTRY)
              search_files · db_schema · query_db · calculator · write_report
              (sandbox · read-only SQL · AST calc · write-gate guardrails)
                          │                │                    │
        ┌─────────────────┘                │                    └───────────────┐
        ▼                                   ▼                                     ▼
  agent/loop.py (ReAct)              Groq schema export                   mcp_server/server.py
  reason→act→observe                 (same schemas)                       (FastMCP, stdio)
        │                                                                        │
  LLMProvider seam  ── LLM_PROVIDER=mock (default) | groq ──┐          Claude Code / Desktop / Cursor
        │                                                    │          drive the SAME tools (real model)
  mock_llm (offline, CI) ─────────────────────────────────── groq_llm (free tier)

• Hand-written ReAct loop (no create_react_agent): reason → tool-select → validate args → execute → observe → repeat, under a max-steps cap with identical-action loop detection. Self-correction is emergent: a ToolError becomes an ERROR: observation the model re-plans from.
• Two-tier memory (scratchpad + persistent SQLite thread store) and a JSONL trace of every step.
• Guardrails as tested code: path-sandbox, read-only SQL (sqlglot + mode=ro), AST calculator (no eval), write-gate, and <untrusted> wrapping of tool output (prompt-injection defense). See DECISIONS.md.

Quickstart (offline — no API key)

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt -r requirements-dev.txt

pytest -q                          # 21 tests, guardrail attacks included
python -m eval.simple_eval         # the agent eval gate (offline, deterministic)

python -m agent.run "What is the 8% sales commission on our total revenue?"
python scripts/render_trace.py     # pretty-print the latest ReAct trace

Use it from Claude Code / Claude Desktop / Cursor (real model, free)

The MCP server exposes the same tools. Point a real client at it:

Claude Code (from the project dir):

claude mcp add toolsmith -- /absolute/path/to/toolsmith-agent/.venv/bin/python \
                             /absolute/path/to/toolsmith-agent/mcp_server/server.py
# then, inside Claude Code:  /mcp   (and ask a multi-tool question)

A committed .mcp.json (uv-based) also works automatically if you have uv installed.

Claude Desktop — add to ~/Library/Application Support/Claude/claude_desktop_config.json, then restart:

{
  "mcpServers": {
    "toolsmith": {
      "command": "/absolute/path/to/toolsmith-agent/.venv/bin/python",
      "args": ["/absolute/path/to/toolsmith-agent/mcp_server/server.py"]
    }
  }
}

Cursor — same block in .cursor/mcp.json.

Inspect the server (Tools / Resources / Prompts UI):

npx @modelcontextprotocol/inspector .venv/bin/python mcp_server/server.py

Try calling query_db with DROP TABLE sales and watch it come back a clean, guardrailed error.

Optional: drive the standalone loop with a real model (Groq free tier)

pip install -r requirements-groq.txt
cp .env.example .env     # set GROQ_API_KEY, LLM_PROVIDER=groq
python -m agent.run "Which product earned the most, and what's 8% of it?"

The provider seam swaps with zero changes to the loop.

Honest notes (because measuring is the point)

• The mock proves the loop's control flow, tool selection/dispatch, arg validation, termination + loop-detection, that every guardrail fires, that an ERROR observation triggers recovery, MCP tool parity, tracing, and full offline CI — not that a model reasons or generalizes.
• Real reasoning is covered, for free, by the MCP-in-Claude-Code path (identical tools + guardrails) and by the optional Groq provider.
• pass^k is trivially 1.0 under the deterministic mock; it's only meaningful re-run over a real model. This README does not headline it as a reliability number.

Tech

Python 3.12 · MCP (official SDK / FastMCP) · Pydantic · sqlglot · SQLite · stdlib (ast, pathlib) · pytest · GitHub Actions · Docker · optional Groq. Deliberately torch-free. Sibling project: GroundedQA (RAG) — github.com/e-akgul/groundedqa.