WebControl

Headless browser automation service for LLM agents. Navigate pages, read structured content, click elements, fill forms — all through a REST API or MCP (Model Context Protocol) tools.

How It Works

┌──────────────────────────┐
│       LLM Agent          │
│  (Claude, custom, etc.)  │
└────────────┬─────────────┘
             │  1. navigate("https://example.com")
             │  2. receives PageContent with element refs (e1, e2, e3...)
             │  3. fill("e3", "[email protected]")
             │  4. click("e7")
             │  5. receives updated PageContent
             ▼
┌──────────────────────────┐
│      WebControl          │
│  REST API + MCP Server   │
│  (single process/port)   │
└────────────┬─────────────┘
             │
             ▼
┌──────────────────────────┐
│  Playwright (headless)   │
│  Chromium browser        │
└──────────────────────────┘

The LLM reads a compact structured representation of the page (interactive elements, forms, links — not raw HTML), decides what to do, and sends an action. WebControl executes it and returns the new page state.

Quick Start

Local

# Install
python3 -m venv .venv && source .venv/bin/activate
pip install ".[dev]"
playwright install chromium

# Run
webcontrol serve

Server starts at http://localhost:8080. Try the health check:

curl http://localhost:8080/health

Docker

docker compose up --build

Usage

REST API

# Create a session
curl -X POST http://localhost:8080/api/v1/sessions \
  -H "Content-Type: application/json" \
  -d '{"name": "my-task"}'
# Returns: {"id": "abc-123", "name": "my-task", ...}

# Navigate
curl -X POST http://localhost:8080/api/v1/sessions/abc-123/navigate \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
# Returns: {"success": true, "page_content": {"elements": [...], "forms": [...], ...}}

# Fill a form field (using ref from page_content)
curl -X POST http://localhost:8080/api/v1/sessions/abc-123/fill \
  -H "Content-Type: application/json" \
  -d '{"ref": "e3", "value": "[email protected]"}'

# Click an element
curl -X POST http://localhost:8080/api/v1/sessions/abc-123/click \
  -H "Content-Type: application/json" \
  -d '{"ref": "e7"}'

# Close session when done
curl -X DELETE http://localhost:8080/api/v1/sessions/abc-123

MCP (Model Context Protocol)

Add to your Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "webcontrol": {
      "command": "webcontrol",
      "args": ["mcp-stdio"]
    }
  }
}

For Cursor, add to .cursor/mcp.json in your project. See mcp-configs/ for more examples.

Once configured, the LLM gets these tools: create_session, navigate, get_page_content, click, fill, select, submit, screenshot, execute_js, close_session.

Anti-Bot Resilience

Sites with anti-bot walls (Amazon, Cloudflare) often serve a block page with HTTP 200, so a naive browser thinks navigation succeeded. WebControl detects blocks and auto-escalates through robustness tiers — stealth fingerprint → human-like behavior → proxy (if configured) — and reports honestly which tier worked (blocked, tier_used, block_reason on every navigate response). If all browser tiers are blocked it raises a 409 Blocked error recommending the search tool; pass fallback_to_search: true to instead return read-only results from a pre-crawled search index (Tier S). See docs/robustness.md.

Configuration

All settings via environment variables prefixed WC_:

Authentication

Set WC_API_KEY to enable API key authentication on REST endpoints:

export WC_API_KEY="your-secret-key"
webcontrol serve

Clients must include the key in requests:

curl -H "x-api-key: your-secret-key" http://localhost:8080/api/v1/sessions

The /health endpoint is always unauthenticated.

MCP stdio mode does not use HTTP auth (it runs as a local subprocess).

Development

# Run tests
pytest tests/ -v

# Run a single test
pytest tests/test_api/test_routes.py::test_navigate_and_get_content -v

# Lint
ruff check src/ tests/

# Format
ruff format src/ tests/

License

MIT