Command Palette

Search for a command to run...

UnylyUnyly
Весь каталог

Capability Value Test

БесплатноНе проверен

Agent Skill for visual, traceable A/B tests of newly integrated AI capabilities, models, MCP servers, browsers, and workflows

GitHubEmbed

Описание

Agent Skill for visual, traceable A/B tests of newly integrated AI capabilities, models, MCP servers, browsers, and workflows

README

English | 简体中文 | 繁體中文 | 日本語 | 한국어 | Русский | हिन्दी | Español | Français | Deutsch | Português | العربية

Make the use of a new capability visible, and make the value it creates measurable.

Capability Value Test is an agent-agnostic Skill for verifying whether a newly integrated capability (such as a Skill, model, plugin, MCP server, browser, open-source component, or agent workflow) was actually invoked and whether it creates measurable value compared with the previous method.

It designs a fair A/B test between the previous method and the method enabled by the new capability. A human reviews the complete design before either arm runs. After execution, it produces a standalone, visual, traceable HTML report.

Why it exists

After integrating a new capability, people often cannot answer:

  1. Was it actually invoked, and where is the evidence?
  2. Did it create value, and can that value be quantified?
  3. Can its use process and concrete value be understood visually?

The problem is not only whether an integration works. Its process is often invisible and its value cannot be measured against the previous way of working.

How it works

  1. Define the new capability and the methods before and after integration.
  2. Choose one real, representative task the user can understand without learning unrelated technical jargon.
  3. Keep inputs, environment, constraints, timing scope, and evaluation criteria aligned.
  4. Present the complete A/B test design for human review; revise it when rejected.
  5. Run both arms only after explicit approval.
  6. Record the process, outputs, metrics, and invocation evidence for both arms.
  7. Separate what the experiment proved from what public sources say, then synthesize concrete usage scenarios and a recommendation.
  8. Render a standalone HTML report in the user's conversation language by default, with a top-right language dropdown.

The goal is not to make the new capability win. It is to establish what changed and whether the evidence supports adoption.

What the HTML report shows

1. Final recommendation

The report uses exactly one verdict:

  • Adopt
  • Do not adopt
  • Inconclusive — more evidence is required

It explains the basis, strengths, problems, concrete suitable and unsuitable scenarios, and experimental limits. A scenario identifies who is doing what task, in what environment, and why the capability fits or does not fit.

2. Value and benefits

The report can compare result quality, end-to-end elapsed time, token usage, cost, steps, errors, and retries. It interprets the differences instead of showing raw numbers alone. Metrics unavailable from the host are displayed as “Not available,” never as zero.

3. Proof of invocation

The final section keeps separate evidence for both arms, such as commands, tool or API calls, key inputs and outputs, logs, timestamps, task IDs, status values, errors, and retries. Evidence must be real and secrets must be redacted.

4. Experiment evidence and external research

The report visibly separates observations from the approved A/B test, findings supported by cited public sources, and the final synthesis. Research adds decision context when one experiment cannot establish general strengths, tradeoffs, or usage scenarios; it never overrides a failed experimental result.

5. Report language

The report initially uses the user's conversation language. A language button in the top-right opens a dropdown for switching the complete narrative report offline. Raw commands, logs, errors, and tool output stay in their original form so the evidence remains auditable.

Human approval is required

Previous method + method enabled by the new capability
                         ↓
                  Experiment design
                         ↓
        Human review ── rejected → revise and review again
                         ↓ approved
                     Run both arms
                         ↓
             Visual HTML report + verdict

Approval applies only to the exact design shown. Consequential actions such as login, payment, submission, communication, or deletion still require their own authorization.

Fair-test principles

Both arms should use the same task, inputs, constraints, timing scope, quality criteria, and evidence level. Any unavoidable difference and its possible effect must be disclosed in the report.

Example uses

  • Compare Chrome and Lightpanda on the same web task.
  • Compare two models on quality, time, and token usage.
  • Determine whether a new Skill reduces steps and elapsed time.
  • Compare an MCP server with an existing API or manual workflow.

Install

Install to supported agents with the Skills CLI:

npx -y skills add EthanZhu-07/capability-value-test -g --all

Or clone the repository and follow your agent's Skill-directory convention:

git clone https://github.com/EthanZhu-07/capability-value-test.git

Use

Use Capability Value Test to compare my existing browser method with the method enabled by Lightpanda. Show me the test design before running it.

The agent must return the design first and wait for human approval.

Render a report manually

The renderer uses only the Python standard library:

python3 scripts/render_report.py tests/sample-experiment.json --output report.html

The result is a self-contained HTML file that opens offline in a modern browser. The fixtures are demonstration data, not claims about real products.

The HTML layout is a versioned, fixed report format. Every report keeps the same section order and includes a short explanation below each major heading so readers know what the section means and how to interpret its evidence. Experiment-specific content changes through experiment.json; individual runs do not create custom layouts.

Limits and privacy

  • One run per arm is directional evidence, not a statistical guarantee.
  • Both methods must be able to complete the same representative task.
  • Tokens and cost can be compared only when the host exposes them.
  • If invocation cannot be verified, the verdict must be Inconclusive.
  • Headless execution is shown as an execution record, never misrepresented as a live screen recording.
  • Secrets, cookies, authorization headers, private tokens, and unrelated personal data must not enter the report.

Repository structure

.
├── SKILL.md                       # Agent-agnostic instructions and workflow
├── agents/openai.yaml             # Optional host UI metadata
├── assets/report-template.html    # Offline visual report template
├── scripts/render_report.py       # JSON validator and HTML renderer
└── tests/                         # Valid and invalid demonstration fixtures

Development

python3 -m py_compile scripts/render_report.py
python3 scripts/render_report.py tests/sample-experiment.json --output /tmp/capability-value-report.html
python3 scripts/render_report.py tests/invalid-experiment.json --output /tmp/invalid.html

The last command should fail with exit code 2. GitHub Actions runs the core checks on pushes and pull requests.

License

No license file is currently included. Public visibility does not itself grant permission to copy, modify, or redistribute the project.

from github.com/EthanZhu-07/capability-value-test

Установка Capability Value Test

У этого сервера нет опубликованного пакета — он собирается из исходников. Открой репозиторий и следуй инструкции в README.

▸ github.com/EthanZhu-07/capability-value-test

FAQ

Capability Value Test MCP бесплатный?

Да, Capability Value Test MCP бесплатный — установка в пару кликов через Unyly без оплаты.

Нужен ли API-ключ для Capability Value Test?

Нет, Capability Value Test работает без API-ключей и переменных окружения.

Capability Value Test — hosted или self-hosted?

Self-hosted: сервер запускается локально на твоей машине командой из раздела установки.

Как установить Capability Value Test в Claude Desktop, Claude Code или Cursor?

Открой Capability Value Test на unyly.org, выбери вкладку своего клиента (Claude Desktop, Claude Code, Cursor) и нажми Install — конфиг сгенерируется автоматически, без правки JSON.

Похожие MCP

Opencode Omniroute Plugin

OpenCode plugin for the OmniRoute AI Gateway. Drives dynamic model discovery, /connect auth flow, and multi-instance OmniRoute providers via the official @openc

GitHub Actionsавтор: GitHub Actions
83

Roblox Studio

Enables AI coding tools to control Roblox Studio for workspace exploration, instance manipulation, and script management. It provides tools for playtesting, sce

paralovавтор: paralov

Antigravity Drawio

MCP server that enables AI assistants to create, parse, render, and validate Draw.io diagrams programmatically.

S-SUJAN-Sавтор: S-SUJAN-S

Fetch

Web content fetching and conversion for efficient LLM usage.

автор: Community

AWS KB Retrieval

Retrieval from AWS Knowledge Base using Bedrock Agent Runtime.

modelcontextprotocolавтор: modelcontextprotocol

Spring AI MCP Server

Provides auto-configuration for setting up an MCP server in Spring Boot applications.

автор: Community

llm-analysis-assistant

A very streamlined mcp client that supports calling and monitoring stdio/sse/streamableHttp, and can also view request responses through the /logs page. It also

xuzexin-hzавтор: xuzexin-hz

MCP-Agent

A simple, composable framework to build agents using Model Context Protocol by [LastMile AI](https://www.lastmileai.dev)

lastmile-aiавтор: lastmile-ai

Spring AI MCP Client

Provides auto-configuration for MCP client functionality in Spring Boot applications.

автор: Community

mcp.natoma.ai

A Hosted MCP Platform to discover, install, manage and deploy MCP servers by [Natoma Labs](https://www.natoma.ai)

автор: Community

Compare Capability Value Test with

Не уверен что выбрать?

Найди свой стек за 60 секунд

Автор?

Embed-бейдж для README

Похожее

Все в категории ai