BrowserControl

Give your AI agent real browser superpowers.
_{Vision-first browser automation for any MCP-compatible AI agent — local, private, and zero-cost.}

What it is • How it looks • Quick start • Connect your AI • Tools • Examples • Why • Contributing

✨ What it is

BrowserControl is an MCP server that gives your AI agent a real browser it can see, click, type, and debug — using a vision-first approach based on the Set of Marks (SoM) pattern.

Instead of fragile CSS selectors, XPath, or DOM trees, the agent sees an annotated screenshot with numbered red boxes over every interactive element, then just calls:

click(5)        →  clicks the 5th element
type_text(3, "hello world")  →  types into the 3rd element
upload_file(7, "/path/to/file.pdf")  →  uploads a file via the native browser input

No selectors to guess. No selectors to break when the page changes. Just point at numbers.

🖼️ How it looks

Every tool that interacts with the page returns an annotated screenshot plus a textual element map:

┌─────────────────────────────────────────────────┐
│  GitHub                          [Sign in]      │
├─────────────────────────────────────────────────┤
│                                                 │
│   ┌──────────────────────┐    ┌──────────────┐  │
│   │ 1  🔍 Search…        │    │ 2  Pulls     │  │
│   └──────────────────────┘    │ 3  Issues    │  │
│                               │ 4  Codespace │  │
│   ┌───────────────────────┐   └──────────────┘  │
│   │ 5  Sign in            │                     │
│   └───────────────────────┘                     │
└─────────────────────────────────────────────────┘

Found 5 interactive elements:
  [1] input - Search or jump to...
  [2] a - Pulls
  [3] a - Issues
  [4] a - Codespaces
  [5] button - Sign in

The model receives the image and the list, so it can either reason over the pixels or read the text — whichever is cheaper.

🚀 Quick start

1 · Install

# pip
pip install browsercontrol

# uv (recommended — much faster)
uv add browsercontrol

No extra setup needed. Chromium auto-installs on first run. If the auto-install fails for any reason, run python -m playwright install chromium once and you're set.

2 · Run the server

# As a CLI
browsercontrol

# As a module
python -m browsercontrol

# With FastMCP (for `fastmcp dev` / inspector UI)
fastmcp run browsercontrol.server:mcp

3 · Connect it to your AI

Pick your client — full configs in Connect your AI below.

// Claude Desktop — claude_desktop_config.json
{
  "mcpServers": {
    "browsercontrol": {
      "command": "browsercontrol"
    }
  }
}

Restart Claude Desktop, then try:

"Open Hacker News and tell me the top story."

That's it — your agent now has a browser.

🧠 Why BrowserControl?

A dozen MCP servers can open a web page. Here's what BrowserControl does that most don't:

🧩 Connect your AI

BrowserControl speaks MCP stdio, so it works with anything that speaks MCP.

{
  "mcpServers": {
    "browsercontrol": {
      "command": "browsercontrol"
    }
  }
}

export MCP_SERVERS='{"browsercontrol": {"command": "browsercontrol"}}'

{
  "browsercontrol": {
    "command": "browsercontrol"
  }
}

{
  "mcpServers": [
    { "name": "browsercontrol", "command": "browsercontrol" }
  ]
}

{ "browsercontrol": { "command": "browsercontrol" } }

{
  "context_servers": {
    "browsercontrol": {
      "command": { "path": "browsercontrol" }
    }
  }
}

import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client


async def main() -> None:
    params = StdioServerParameters(command="browsercontrol", args=[])
    async with stdio_client(params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            tools = await session.list_tools()
            result = await session.call_tool(
                "navigate_to", {"url": "https://news.ycombinator.com"}
            )


asyncio.run(main())

{
  "mcpServers": {
    "browsercontrol": { "command": "uvx",  "args": ["browsercontrol"] }
  }
}

{
  "mcpServers": {
    "browsercontrol": { "command": "pipx", "args": ["run", "browsercontrol"] }
  }
}

{
  "mcpServers": {
    "browsercontrol": {
      "command": "browsercontrol",
      "env": {
        "BROWSER_HEADLESS": "false",
        "BROWSER_VIEWPORT_WIDTH": "1920",
        "BROWSER_VIEWPORT_HEIGHT": "1080",
        "LOG_LEVEL": "DEBUG"
      }
    }
  }
}

🛠️ Tools

Every action returns an annotated screenshot + element map (when relevant), so the model always has the latest visual context.

Navigation

Interaction

Tabs

Forms

Content

DevTools

Recording

View recordings with: npx playwright show-trace ~/.browsercontrol/recordings/<name>.zip

⚙️ Configuration

All settings are environment variables. Sensible defaults out of the box.

# Headful + phone-sized viewport for mobile testing
BROWSER_HEADLESS=false BROWSER_VIEWPORT_WIDTH=375 BROWSER_VIEWPORT_HEIGHT=812 browsercontrol

# Verbose logs for debugging the MCP server itself
LOG_LEVEL=DEBUG browsercontrol

📚 Examples

🔍 Web research

"Find the Wikipedia article on Python and tell me who created it."

→ navigate_to("https://wikipedia.org")
→ type_text(2, "Python programming language")
→ press_key("Enter")
→ get_page_content()
→ "Python was created by Guido van Rossum, first released in 1991."

🐛 Debug a web app

"Check localhost:3000 for any errors and tell me what's broken."

→ navigate_to("http://localhost:3000")    # auto-fallback to 127.0.0.1 if needed
→ get_console_logs()
→ "2 errors:
   [ERROR] Uncaught TypeError: Cannot read property 'map' of undefined (app.js:42)
   [ERROR] Failed to load resource: 404 /api/users"
→ get_network_requests()
→ "GET /api/users -> 404 (12ms)  ← check your API route"

🧪 Record a test run

"Walk through the login flow on staging.example.com while recording."

→ start_recording("login_flow")
→ navigate_to("https://staging.example.com/login")
→ type_text(3, "[email protected]")
→ type_text(4, "hunter2")
→ click(5)             # Sign In button
→ take_snapshot("after_login")
→ stop_recording()
→ "Saved ~/.browsercontrol/recordings/login_flow.zip"

📝 Fill out a form

"Fill out the contact form at example.com/contact with a polite message."

→ navigate_to("https://example.com/contact")
→ type_text(2, "Alex Doe")
→ type_text(3, "[email protected]")
→ type_text(4, "Hi! Just saying hello.")
→ click(5)             # Submit

🗂️ Multi-tab research

"Open GitHub in one tab and Hacker News in another, then tell me the top repo and the top story."

→ create_tab("https://github.com/trending")
→ create_tab("https://news.ycombinator.com")
→ list_tabs()          # → returns both tabs with indices
→ switch_tab(0)        # focus GitHub
→ get_page_content()
→ switch_tab(1)        # focus HN
→ get_page_content()

📂 Upload a file

"Upload my resume to the job application form."

→ navigate_to("https://jobs.example.com/apply")
→ upload_file(8, "/Users/me/Documents/resume.pdf")

📱 Mobile responsive check

"Switch to iPhone 14 viewport and screenshot the homepage."

→ set_viewport(390, 844)
→ navigate_to("https://my-site.com")
→ screenshot(annotate=False, full_page=True)

🏗️ Architecture

┌─────────────────┐     ┌────────────────────┐     ┌─────────────┐
│   AI Agent      │────▶│  BrowserControl    │────▶│  Chromium   │
│ (Claude, Gemini,│◀────│   MCP Server       │◀────│ (Playwright)│
│  GPT, local…)   │     │                    │     │             │
└─────────────────┘     └────────────────────┘     └─────────────┘
        │                        │                       │
        │  call_tool("click",5)  │  page.click(handle)    │
        │  ◀─────────────────────│  ◀─────────────────────│
        │  annotated PNG +       │  screenshot.png        │
        │  element map           │  + SoM overlay         │

Action loop

1. Agent calls a tool — e.g. click(5).
2. Server resolves the element — Looks up element 5 from the last SoM pass, then calls document.elementFromPoint(x, y) to get the actual DOM element (handles overlays, sticky headers, and shadow boundaries).
3. Browser executes — Playwright drives the real Chromium.
4. Server re-screenshots — New screenshot, fresh element detection (with shadow DOM + iframe recursion), numbered boxes drawn with Pillow.
5. Annotated result returns — Both image and textual map are sent back so the model can pick its preferred representation.

Why SoM over selectors?

• No DOM drift — If the site gets redesigned, the numbered elements still point at the right pixels.
• No hallucinated selectors — The model never invents div.flex-container > button.btn-primary:nth-child(3).
• Tiny tokens — A list of {"id": 7, "tag": "button", "text": "Submit"} is far cheaper than a full DOM tree.
• Vision + text — The model can use either or both. Vision for visual reasoning, text for fast selection.

🆚 How it compares

A few notes, kept honest:

⚖️ We try to be fair: Playwright MCP is excellent if you're fine with the accessibility tree approach. Stagehand and Browser-Use are great if you specifically want an LLM in the loop. BrowserControl's niche is vision-first, fully local, zero-LLM-API browser control with built-in devtools.

🧪 Troubleshooting

xvfb-run browsercontrol          # Linux without a display
BROWSER_HEADLESS=false browsercontrol   # with a display

python -m playwright install chromium --with-deps

ls -la ~/.browsercontrol/

pkill -f browsercontrol
browsercontrol

npx playwright show-trace ~/.browsercontrol/recordings/session_20260228.zip

📁 Project structure

browsercontrol/
├── __init__.py              # Public exports (`mcp`)
├── __main__.py              # `python -m browsercontrol` entry point
├── server.py                # FastMCP server definition + tool registration
├── browser.py               # BrowserManager — Playwright lifecycle + SoM renderer
├── config.py                # Env-var configuration
└── tools/
    ├── navigation.py        # navigate_to, go_back, go_forward, refresh, scroll
    ├── interaction.py       # click, click_at, type_text, press_key, hover, scroll_to_element, wait
    ├── forms.py             # select_option, check_checkbox, upload_file
    ├── content.py           # get_page_content, get_text, get_page_info, run_javascript, screenshot
    ├── devtools.py          # console, network, errors, inspect, perf, cookies, viewport
    ├── recording.py         # start/stop_recording, take_snapshot, list_recordings
    └── tabs.py              # create_tab, switch_tab, close_tab, list_tabs

tests/                       # 1,664 lines of pytest tests (mocked Playwright)
├── conftest.py              # Shared fixtures: mock_page, mock_context, mock_browser_manager…
└── test_*.py                # One file per module

🛠️ Development

git clone https://github.com/adityasasidhar/browsercontrol
cd browsercontrol

# Install deps + Playwright + Chromium
uv sync
uv run playwright install chromium --with-deps

# Run the server in dev mode (FastMCP inspector)
uv run fastmcp dev browsercontrol/server.py

# Run tests
uv run pytest

# Lint, format, security checks
uv run ruff check .
uv run ruff format .
uv run pre-commit run --all-files

CI runs the full suite on Ubuntu, Windows, and macOS with ruff, mypy --strict, bandit, and pytest.

🤝 Contributing

Contributions welcome! See CONTRIBUTING.md for the workflow and CODE_QUALITY.md for the toolchain.

Good first contributions:

• Firefox / WebKit support
• DOM diffing (detect changes between snapshots)
• Accessibility audit tools
• Mobile emulation presets (iPhone, Pixel, iPad)
• Cookie import/export (Netscape format)
• --record-video MP4 output (in addition to traces)
• Network request blocking / mocking

When adding a tool, please add a matching tests/test_<your_tool>.py covering the happy path and error path.

📄 License

MIT — do whatever you want, just keep the copyright.

🙏 Acknowledgments

• Built on FastMCP and Playwright.
• The Set of Marks idea comes from the Set-of-Marks prompting literature for visual tool use — adapted here as the agent's primary interface rather than a fallback.
• Thanks to the MCP community for making AI-tool integration this pleasant.

Built for AI agents that need to see the web.

⭐ Star on GitHub  •  🐛 Report a bug  •  💡 Request a feature