Promptest

An MCP server that grades your AI prompts against Promptest's validated 5-category rubric, tracks what actually happened when you used them, and gives you guidance grounded in the CRG-RIS study of what really predicts good outcomes — not just what looks good on paper.

Why this exists

Promptest's rubric reliably scores prompt quality. But a CRG-RIS evaluation of that rubric found something more specific and more useful: overall score didn't predict cost or scope discipline on a real coding task — but whether the prompt named the bug's root cause and exact fix location did, by a wide margin (37–41% lower cost, and the only variant that avoided a security-flagged over-broad fix).

This MCP server bakes that finding directly into its guidance, and — because it's an MCP tool that lives inside your actual coding sessions — it can track your own real outcomes over time, not just score prompts in isolation the way the web app does.

Tools

• grade_prompt — scores a prompt against the 5-category rubric, returns a grade, category breakdown, and guidance. Returns a taskId.
• log_outcome — record what actually happened: did it work, how many turns did it take, did the fix stay in scope, did you name the root cause/location. Pass the taskId from grade_prompt.
• get_prompting_history — your personal history and stats, including the CRG-RIS-validated comparison: average turns when you named the root cause/location vs. when you didn't.
• explain_rubric — explains the rubric and the CRG-RIS findings without grading anything.

Install

1. Build

git clone https://github.com/cortexresearch/promptest-mcp.git
cd promptest-mcp
npm install
npm run build

2. Add to your MCP client config

Claude Code (.claude/settings.json or via claude mcp add):

{
  "mcpServers": {
    "promptest": {
      "command": "node",
      "args": ["/absolute/path/to/promptest-mcp/dist/index.js"],
      "env": {
        "ANTHROPIC_API_KEY": "sk-ant-..."
      }
    }
  }
}

Claude Desktop (claude_desktop_config.json): same shape, under mcpServers.

3. Set your API key

Set exactly one of these in the env block above, matching whichever provider you want as the judge:

Env var	Provider
ANTHROPIC_API_KEY	Anthropic (Claude)
OPENAI_API_KEY	OpenAI
GOOGLE_API_KEY	Google (Gemini)
OPENROUTER_API_KEY	OpenRouter (any model)

The server auto-detects which provider to use from whichever key is set. To force a specific provider or model, set PROMPTEST_PROVIDER (anthropic | openai | google | openrouter) and/or PROMPTEST_MODEL.

Your key never leaves your machine except to call the provider you chose. This server has no backend, sends nothing to Promptest or Cortex Research Group, and doesn't phone home.

Where your data lives

History is stored locally at ~/.promptest/history.json — plain JSON, nothing hidden, delete it any time to start fresh.

Background

• Promptest — the web app this rubric was built for
• CRG-RIS study — the research this MCP's guidance is grounded in, including the honestly-reported finding that the rubric's overall score alone didn't predict real-world cost, and what did

License

MIT