Playtest Agent
БесплатноНе проверенAutonomous game playtesting agent: plays a game, hunts bugs with generic oracles + LLM, writes reproducible reports. Includes seeded-bug eval harness and MCP se
Описание
Autonomous game playtesting agent: plays a game, hunts bugs with generic oracles + LLM, writes reproducible reports. Includes seeded-bug eval harness and MCP server.
README
An autonomous QA agent that plays a game, hunts for bugs, and writes a reproducible bug report.
It drives a game purely through an observation/action interface (in-process or over a TCP socket, the way a real engine build would be driven), runs a set of generic bug oracles on every state transition, and produces a Markdown report with severity, root-cause hints, and an exact action sequence that reproduces each finding from reset.
It ships with:
- a small deterministic dungeon game with 8 switchable, seeded bugs (soft lock, wall clipping, economy exploit, invulnerability, crash, unwinnable level, ...)
- a hybrid scripted + LLM explorer with rotating "personas" (cautious explorer, reckless fighter, item hoarder)
- an evaluation harness that measures precision and recall of the agent against the seeded bugs
- an MCP server so the agent can be driven from Claude Code, Claude Desktop, or any MCP client
- a test suite and a CI workflow
Results
Scripted-only (no LLM, no API key, fully deterministic), 3 episodes x 250 steps per seed:
| seeded bug | detected | steps to detect |
|---|---|---|
| door_ignores_key | yes | 64 |
| east_wall_missing | yes | 124 |
| potion_no_consume_check | yes | 1 |
| shop_allows_negative_gold | yes | 112 |
| player_invulnerable | yes | 283 |
| softlock_after_turns | yes | 46 |
| key_overflow_crash | yes | 320 |
| goal_does_not_win | yes | 160 |
| (clean game) | 0 false positives | - |
recall = 1.00, precision = 1.00 on seeds 0-2 (and on held-out seeds 3-4). See examples/eval.md and a sample report in examples/report_seeded_bugs.md.
Quick start
pip install -e ".[dev]" # core has zero runtime deps
playtest run # play the clean game, print a report
playtest run --bug softlock_after_turns --bug shop_allows_negative_gold --out report.md
playtest eval # precision/recall table against all seeded bugs
pytest # 29 tests, ~6 s
Drive a game over a socket, the way you would drive a real build:
playtest serve --port 7777 --bug east_wall_missing # terminal 1
playtest run --socket 127.0.0.1:7777 # terminal 2
playtest run exits non-zero when a critical finding exists, so it can gate a CI pipeline.
With an LLM
pip install -e ".[llm]"
export ANTHROPIC_API_KEY=...
playtest run --llm
The model is consulted only when the scripted explorer is stuck or at a periodic check-in, and once more at the end to triage findings (rank severity, write a root-cause hypothesis, drop duplicates). Cost is bounded by max_calls, with one call always reserved for triage. Without a key everything degrades to the scripted policy, so evaluation stays free and reproducible.
Run with claude-sonnet-5, 20 calls per run: recall and precision stay at 1.00 (the scripted policy already finds everything in this game), and each finding gets a one-line root cause. From examples/report_llm.md:
| finding | model's hypothesis |
|---|---|
| Goal tile reached but game did not end | Win-condition check is not triggered on tile entry (missing goal-tile collision/event handler) |
| Door opened without consuming a key | Door-open logic fails to decrement key count or checks wrong door state before consuming inventory |
| Purchase succeeded with insufficient gold | buy_potion lacks a gold>=cost guard before executing the transaction |
All three are correct: they name the exact line each seeded bug removes. The first live run also exposed a bug in the agent itself: exploration consumed the whole call budget and triage silently got nothing, so every hypothesis was empty. Now fixed and covered by a test.
As an MCP server
pip install -e ".[mcp]"
playtest-mcp
Add to Claude Code (.mcp.json):
{ "mcpServers": { "playtest": { "command": "playtest-mcp" } } }
Tools: run_playtest, evaluate_agent, list_bug_flags, play_reset, play_step, play_view.
How it works
game (Dungeon | TCP server)
| observation: ASCII map + JSON state
v
Explorer ──── policy each step ────────────────────────────────
| 1. survival (drink potion when low) [off in reckless persona]
| 2. pending actions from an LLM plan
| 3. probes: use potion at full HP, buy with no gold, push every
| map-border tile and locked door, hunt enemies, hoard items
| 4. coverage: BFS to nearest unvisited reachable tile
| 5. goal: path to the exit once the map is exhausted
| 6. ask LLM planner for an experiment, else random
v
Detectors (run on every (before, action, after) transition)
| crash · numeric invariants · clip-through-wall / teleport ·
| door-without-key · wasted consumable / purchase-without-funds ·
| damage-without-effect · goal-without-win · soft lock
v
Findings ──> (optional LLM triage) ──> Markdown + JSON report
Detectors are generic. They encode invariants like positions stay in bounds, resources never go negative, consuming a thing changes something, the game responds to input, a damage message means HP went down. None of them knows about the specific seeded bugs. GameProfile holds the few game-specific knobs (which messages mean damage or healing, which tile is the goal).
Personas matter more than cleverness. The first version had a single cautious policy and found 3 of 8 bugs. Adding a reckless persona (fights enemies, never heals) found invulnerability, and a hoarder persona (grabs every pickup before opening doors) found the key-overflow crash. That took recall from 0.38 to 1.0 with no LLM involved.
Every finding carries a repro. The action list from reset is stored with the finding, and tests/test_agent.py::test_repro_replays_to_same_state checks that replaying it lands in the reported state.
Design notes and tradeoffs
- Why a bundled toy game? So the evaluation is honest: the bugs are known, seeded one at a time, and the clean game is a real false-positive check. Porting to another game means implementing
GameClient(reset/step/actions/bounds) and adjustingGameProfile. - Why scripted first, LLM second? Scripted coverage is free, fast, and deterministic. The LLM adds value exactly where scripting runs out: hypothesising unusual inputs and explaining findings. Keeping it optional keeps CI reproducible.
- Same-turn confounds. An early version of the damage detector flagged "drink potion, then get hit" as damage-without-effect. The fix was making the game report the whole turn's events and teaching the detector about heal messages. This is the general lesson: oracles must understand the game's event granularity.
- Dedup is per kind, not per instance. The first occurrence of each finding kind is kept. A production version would key on (kind, location) and cluster.
Roadmap
- Screenshot-based observation (vision model) for engines without a state API
- Coverage-guided fuzzing of action sequences near a finding to minimise repros
- Per-location finding clustering and a diff mode ("new since last build")
- Unity/Godot adapter over the same line-JSON protocol
Layout
src/playtest_agent/
game/ dungeon.py (game + BugFlags), server.py (TCP)
agent/ client.py, explorer.py, detectors.py, llm.py
eval/ harness.py (precision/recall)
report.py Markdown rendering
cli.py `playtest run|eval|serve`
mcp_server.py
tests/ game, detectors, agent, socket, eval
examples/ sample reports + eval output
MIT licensed.
Установить Playtest Agent в Claude Desktop, Claude Code, Cursor
unyly install playtest-agentСтавит в Claude Desktop, Claude Code, Cursor и VS Code — сам разбирается с npx, uvx и сборкой из исходников.
Впервые? Поставь CLI: curl -fsSL https://unyly.org/install | sh
Или настроить вручную
Выполни в терминале:
claude mcp add playtest-agent -- uvx --from git+https://github.com/kevinnie2003/playtest-agent playtest-agentПошаговые гайды: как установить Playtest Agent
FAQ
Playtest Agent MCP бесплатный?
Да, Playtest Agent MCP бесплатный — установка в пару кликов через Unyly без оплаты.
Нужен ли API-ключ для Playtest Agent?
Нет, Playtest Agent работает без API-ключей и переменных окружения.
Playtest Agent — hosted или self-hosted?
Self-hosted: сервер запускается локально на твоей машине командой из раздела установки.
Как установить Playtest Agent в Claude Desktop, Claude Code или Cursor?
Открой Playtest Agent на unyly.org, выбери вкладку своего клиента (Claude Desktop, Claude Code, Cursor) и нажми Install — конфиг сгенерируется автоматически, без правки JSON.
Похожие MCP
Opencode Omniroute Plugin
OpenCode plugin for the OmniRoute AI Gateway. Drives dynamic model discovery, /connect auth flow, and multi-instance OmniRoute providers via the official @openc
автор: GitHub ActionsRoblox Studio
Enables AI coding tools to control Roblox Studio for workspace exploration, instance manipulation, and script management. It provides tools for playtesting, sce
автор: paralovAntigravity Drawio
MCP server that enables AI assistants to create, parse, render, and validate Draw.io diagrams programmatically.
автор: S-SUJAN-SFetch
Web content fetching and conversion for efficient LLM usage.
AWS KB Retrieval
Retrieval from AWS Knowledge Base using Bedrock Agent Runtime.
автор: modelcontextprotocolSpring AI MCP Server
Provides auto-configuration for setting up an MCP server in Spring Boot applications.
llm-analysis-assistant
A very streamlined mcp client that supports calling and monitoring stdio/sse/streamableHttp, and can also view request responses through the /logs page. It also
автор: xuzexin-hzMCP-Agent
A simple, composable framework to build agents using Model Context Protocol by [LastMile AI](https://www.lastmileai.dev)
автор: lastmile-aiSpring AI MCP Client
Provides auto-configuration for MCP client functionality in Spring Boot applications.
mcp.natoma.ai
A Hosted MCP Platform to discover, install, manage and deploy MCP servers by [Natoma Labs](https://www.natoma.ai)
Compare Playtest Agent with
Не уверен что выбрать?
Найди свой стек за 60 секунд
Автор?
Embed-бейдж для README
Похожее
Все в категории ai
