Command Palette

Search for a command to run...

UnylyUnyly
Весь каталог

Playtest Agent

БесплатноНе проверен

Autonomous game playtesting agent: plays a game, hunts bugs with generic oracles + LLM, writes reproducible reports. Includes seeded-bug eval harness and MCP se

GitHubEmbed

Описание

Autonomous game playtesting agent: plays a game, hunts bugs with generic oracles + LLM, writes reproducible reports. Includes seeded-bug eval harness and MCP server.

README

ci python license

An autonomous QA agent that plays a game, hunts for bugs, and writes a reproducible bug report.

It drives a game purely through an observation/action interface (in-process or over a TCP socket, the way a real engine build would be driven), runs a set of generic bug oracles on every state transition, and produces a Markdown report with severity, root-cause hints, and an exact action sequence that reproduces each finding from reset.

It ships with:

  • a small deterministic dungeon game with 8 switchable, seeded bugs (soft lock, wall clipping, economy exploit, invulnerability, crash, unwinnable level, ...)
  • a hybrid scripted + LLM explorer with rotating "personas" (cautious explorer, reckless fighter, item hoarder)
  • an evaluation harness that measures precision and recall of the agent against the seeded bugs
  • an MCP server so the agent can be driven from Claude Code, Claude Desktop, or any MCP client
  • a test suite and a CI workflow

Results

Scripted-only (no LLM, no API key, fully deterministic), 3 episodes x 250 steps per seed:

seeded bug detected steps to detect
door_ignores_key yes 64
east_wall_missing yes 124
potion_no_consume_check yes 1
shop_allows_negative_gold yes 112
player_invulnerable yes 283
softlock_after_turns yes 46
key_overflow_crash yes 320
goal_does_not_win yes 160
(clean game) 0 false positives -

recall = 1.00, precision = 1.00 on seeds 0-2 (and on held-out seeds 3-4). See examples/eval.md and a sample report in examples/report_seeded_bugs.md.

Quick start

pip install -e ".[dev]"          # core has zero runtime deps
playtest run                     # play the clean game, print a report
playtest run --bug softlock_after_turns --bug shop_allows_negative_gold --out report.md
playtest eval                    # precision/recall table against all seeded bugs
pytest                           # 29 tests, ~6 s

Drive a game over a socket, the way you would drive a real build:

playtest serve --port 7777 --bug east_wall_missing     # terminal 1
playtest run --socket 127.0.0.1:7777                    # terminal 2

playtest run exits non-zero when a critical finding exists, so it can gate a CI pipeline.

With an LLM

pip install -e ".[llm]"
export ANTHROPIC_API_KEY=...
playtest run --llm

The model is consulted only when the scripted explorer is stuck or at a periodic check-in, and once more at the end to triage findings (rank severity, write a root-cause hypothesis, drop duplicates). Cost is bounded by max_calls, with one call always reserved for triage. Without a key everything degrades to the scripted policy, so evaluation stays free and reproducible.

Run with claude-sonnet-5, 20 calls per run: recall and precision stay at 1.00 (the scripted policy already finds everything in this game), and each finding gets a one-line root cause. From examples/report_llm.md:

finding model's hypothesis
Goal tile reached but game did not end Win-condition check is not triggered on tile entry (missing goal-tile collision/event handler)
Door opened without consuming a key Door-open logic fails to decrement key count or checks wrong door state before consuming inventory
Purchase succeeded with insufficient gold buy_potion lacks a gold>=cost guard before executing the transaction

All three are correct: they name the exact line each seeded bug removes. The first live run also exposed a bug in the agent itself: exploration consumed the whole call budget and triage silently got nothing, so every hypothesis was empty. Now fixed and covered by a test.

As an MCP server

pip install -e ".[mcp]"
playtest-mcp

Add to Claude Code (.mcp.json):

{ "mcpServers": { "playtest": { "command": "playtest-mcp" } } }

Tools: run_playtest, evaluate_agent, list_bug_flags, play_reset, play_step, play_view.

How it works

 game (Dungeon | TCP server)
        |  observation: ASCII map + JSON state
        v
 Explorer  ──── policy each step ────────────────────────────────
        |   1. survival (drink potion when low)          [off in reckless persona]
        |   2. pending actions from an LLM plan
        |   3. probes: use potion at full HP, buy with no gold, push every
        |      map-border tile and locked door, hunt enemies, hoard items
        |   4. coverage: BFS to nearest unvisited reachable tile
        |   5. goal: path to the exit once the map is exhausted
        |   6. ask LLM planner for an experiment, else random
        v
 Detectors (run on every (before, action, after) transition)
        |   crash · numeric invariants · clip-through-wall / teleport ·
        |   door-without-key · wasted consumable / purchase-without-funds ·
        |   damage-without-effect · goal-without-win · soft lock
        v
 Findings  ──>  (optional LLM triage)  ──>  Markdown + JSON report

Detectors are generic. They encode invariants like positions stay in bounds, resources never go negative, consuming a thing changes something, the game responds to input, a damage message means HP went down. None of them knows about the specific seeded bugs. GameProfile holds the few game-specific knobs (which messages mean damage or healing, which tile is the goal).

Personas matter more than cleverness. The first version had a single cautious policy and found 3 of 8 bugs. Adding a reckless persona (fights enemies, never heals) found invulnerability, and a hoarder persona (grabs every pickup before opening doors) found the key-overflow crash. That took recall from 0.38 to 1.0 with no LLM involved.

Every finding carries a repro. The action list from reset is stored with the finding, and tests/test_agent.py::test_repro_replays_to_same_state checks that replaying it lands in the reported state.

Design notes and tradeoffs

  • Why a bundled toy game? So the evaluation is honest: the bugs are known, seeded one at a time, and the clean game is a real false-positive check. Porting to another game means implementing GameClient (reset/step/actions/bounds) and adjusting GameProfile.
  • Why scripted first, LLM second? Scripted coverage is free, fast, and deterministic. The LLM adds value exactly where scripting runs out: hypothesising unusual inputs and explaining findings. Keeping it optional keeps CI reproducible.
  • Same-turn confounds. An early version of the damage detector flagged "drink potion, then get hit" as damage-without-effect. The fix was making the game report the whole turn's events and teaching the detector about heal messages. This is the general lesson: oracles must understand the game's event granularity.
  • Dedup is per kind, not per instance. The first occurrence of each finding kind is kept. A production version would key on (kind, location) and cluster.

Roadmap

  • Screenshot-based observation (vision model) for engines without a state API
  • Coverage-guided fuzzing of action sequences near a finding to minimise repros
  • Per-location finding clustering and a diff mode ("new since last build")
  • Unity/Godot adapter over the same line-JSON protocol

Layout

src/playtest_agent/
  game/       dungeon.py (game + BugFlags), server.py (TCP)
  agent/      client.py, explorer.py, detectors.py, llm.py
  eval/       harness.py (precision/recall)
  report.py   Markdown rendering
  cli.py      `playtest run|eval|serve`
  mcp_server.py
tests/        game, detectors, agent, socket, eval
examples/     sample reports + eval output

MIT licensed.

from github.com/kevinnie2003/playtest-agent

Установить Playtest Agent в Claude Desktop, Claude Code, Cursor

Рекомендуется · одна команда, все IDE
unyly install playtest-agent

Ставит в Claude Desktop, Claude Code, Cursor и VS Code — сам разбирается с npx, uvx и сборкой из исходников.

Впервые? Поставь CLI: curl -fsSL https://unyly.org/install | sh

Или настроить вручную

Выполни в терминале:

claude mcp add playtest-agent -- uvx --from git+https://github.com/kevinnie2003/playtest-agent playtest-agent

Пошаговые гайды: как установить Playtest Agent

FAQ

Playtest Agent MCP бесплатный?

Да, Playtest Agent MCP бесплатный — установка в пару кликов через Unyly без оплаты.

Нужен ли API-ключ для Playtest Agent?

Нет, Playtest Agent работает без API-ключей и переменных окружения.

Playtest Agent — hosted или self-hosted?

Self-hosted: сервер запускается локально на твоей машине командой из раздела установки.

Как установить Playtest Agent в Claude Desktop, Claude Code или Cursor?

Открой Playtest Agent на unyly.org, выбери вкладку своего клиента (Claude Desktop, Claude Code, Cursor) и нажми Install — конфиг сгенерируется автоматически, без правки JSON.

Похожие MCP

Opencode Omniroute Plugin

OpenCode plugin for the OmniRoute AI Gateway. Drives dynamic model discovery, /connect auth flow, and multi-instance OmniRoute providers via the official @openc

GitHub Actionsавтор: GitHub Actions
85

Roblox Studio

Enables AI coding tools to control Roblox Studio for workspace exploration, instance manipulation, and script management. It provides tools for playtesting, sce

paralovавтор: paralov

Antigravity Drawio

MCP server that enables AI assistants to create, parse, render, and validate Draw.io diagrams programmatically.

S-SUJAN-Sавтор: S-SUJAN-S

Fetch

Web content fetching and conversion for efficient LLM usage.

автор: Community

AWS KB Retrieval

Retrieval from AWS Knowledge Base using Bedrock Agent Runtime.

modelcontextprotocolавтор: modelcontextprotocol

Spring AI MCP Server

Provides auto-configuration for setting up an MCP server in Spring Boot applications.

автор: Community

llm-analysis-assistant

A very streamlined mcp client that supports calling and monitoring stdio/sse/streamableHttp, and can also view request responses through the /logs page. It also

xuzexin-hzавтор: xuzexin-hz

MCP-Agent

A simple, composable framework to build agents using Model Context Protocol by [LastMile AI](https://www.lastmileai.dev)

lastmile-aiавтор: lastmile-ai

Spring AI MCP Client

Provides auto-configuration for MCP client functionality in Spring Boot applications.

автор: Community

mcp.natoma.ai

A Hosted MCP Platform to discover, install, manage and deploy MCP servers by [Natoma Labs](https://www.natoma.ai)

автор: Community

Compare Playtest Agent with

Не уверен что выбрать?

Найди свой стек за 60 секунд

Автор?

Embed-бейдж для README

Похожее

Все в категории ai