WebControl

Headless browser automation service for LLM agents. Navigate pages, read structured content, click elements, fill forms — all through a REST API or MCP (Model Context Protocol) tools.

How It Works

┌──────────────────────────┐
│       LLM Agent          │
│  (Claude, custom, etc.)  │
└────────────┬─────────────┘
             │  1. navigate("https://example.com")
             │  2. receives PageContent with element refs (e1, e2, e3...)
             │  3. fill("e3", "[email protected]")
             │  4. click("e7")
             │  5. receives updated PageContent
             ▼
┌──────────────────────────┐
│      WebControl          │
│  REST API + MCP Server   │
│  (single process/port)   │
└────────────┬─────────────┘
             │
             ▼
┌──────────────────────────┐
│  Playwright (headless)   │
│  Chromium browser        │
└──────────────────────────┘

The LLM reads a compact structured representation of the page (interactive elements, forms, links — not raw HTML), decides what to do, and sends an action. WebControl executes it and returns the new page state.

Quick Start

Local

# Install
python3 -m venv .venv && source .venv/bin/activate
pip install ".[dev]"
playwright install chromium

# Run
webcontrol serve

Server starts at http://localhost:8080. Try the health check:

curl http://localhost:8080/health

Docker

docker compose up --build

Usage

REST API

# Create a session
curl -X POST http://localhost:8080/api/v1/sessions \
  -H "Content-Type: application/json" \
  -d '{"name": "my-task"}'
# Returns: {"id": "abc-123", "name": "my-task", ...}

# Navigate
curl -X POST http://localhost:8080/api/v1/sessions/abc-123/navigate \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
# Returns: {"success": true, "page_content": {"elements": [...], "forms": [...], ...}}

# Fill a form field (using ref from page_content)
curl -X POST http://localhost:8080/api/v1/sessions/abc-123/fill \
  -H "Content-Type: application/json" \
  -d '{"ref": "e3", "value": "[email protected]"}'

# Click an element
curl -X POST http://localhost:8080/api/v1/sessions/abc-123/click \
  -H "Content-Type: application/json" \
  -d '{"ref": "e7"}'

# Close session when done
curl -X DELETE http://localhost:8080/api/v1/sessions/abc-123

MCP (Model Context Protocol)

Add to your Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "webcontrol": {
      "command": "webcontrol",
      "args": ["mcp-stdio"]
    }
  }
}

For Cursor, add to .cursor/mcp.json in your project. See mcp-configs/ for more examples.

Once configured, the LLM gets these tools: create_session, navigate, get_page_content, click, fill, select, submit, screenshot, execute_js, close_session.

Anti-Bot Resilience

Sites with anti-bot walls (Amazon, Cloudflare) often serve a block page with HTTP 200, so a naive browser thinks navigation succeeded. WebControl detects blocks and auto-escalates through robustness tiers — stealth fingerprint → human-like behavior → proxy (if configured) — and reports honestly which tier worked (blocked, tier_used, block_reason on every navigate response). If all browser tiers are blocked it raises a 409 Blocked error recommending the search tool; pass fallback_to_search: true to instead return read-only results from a pre-crawled search index (Tier S). See docs/robustness.md.

Configuration

All settings via environment variables prefixed WC_:

Variable	Default	Description
WC_PORT	8080	Server port
WC_HOST	0.0.0.0	Bind address
WC_HEADLESS	true	Run browser headless
WC_BROWSER_TYPE	chromium	Browser engine (chromium, firefox, webkit)
WC_MAX_SESSIONS	10	Maximum concurrent browser sessions
WC_DEFAULT_SESSION_TTL_SECONDS	1800	Session idle timeout (30 min)
WC_VIEWPORT_WIDTH	1280	Default viewport width
WC_VIEWPORT_HEIGHT	720	Default viewport height
WC_NAVIGATION_TIMEOUT_MS	30000	Navigation timeout
WC_ACTION_TIMEOUT_MS	10000	Action timeout (click, fill, etc.)
WC_API_KEY	(empty)	API key for REST auth; empty = no auth
WC_PROXY_SERVER	(empty)	HTTP proxy (e.g., http://proxy:8080); also enables the proxy escalation tier
WC_PROXY_USERNAME	(empty)	Proxy auth username
WC_PROXY_PASSWORD	(empty)	Proxy auth password
WC_STEALTH_ENABLED	true	Mask headless browser fingerprint (anti-bot tier 0)
WC_USER_AGENT	(empty)	Override the stealth default user-agent
WC_LOCALE	en-US	Browser locale + Accept-Language header
WC_TIMEZONE_ID	(empty)	Override browser timezone
WC_NAVIGATION_ESCALATION	true	Auto-escalate through robustness tiers when a site blocks
WC_BEHAVIORAL_JITTER_MS	800	Max random delay for the behavioral escalation tier
WC_SEARCH_TIER_ENABLED	false	Enable the search-index fallback (Tier S)
WC_SEARCH_PROVIDER	exa	Search provider for Tier S (exa/brave)
WC_SEARCH_API_KEY	(empty)	API key for the search provider
WC_NAVIGATION_RETRIES	2	Retry attempts for navigation
WC_ACTION_RETRIES	1	Retry attempts for actions
WC_RETRY_DELAY_MS	500	Delay between retries
WC_LOG_LEVEL	INFO	Log level
WC_LOG_JSON	false	Output structured JSON logs

Authentication

Set WC_API_KEY to enable API key authentication on REST endpoints:

export WC_API_KEY="your-secret-key"
webcontrol serve

Clients must include the key in requests:

curl -H "x-api-key: your-secret-key" http://localhost:8080/api/v1/sessions

The /health endpoint is always unauthenticated.

MCP stdio mode does not use HTTP auth (it runs as a local subprocess).

Development

# Run tests
pytest tests/ -v

# Run a single test
pytest tests/test_api/test_routes.py::test_navigate_and_get_content -v

# Lint
ruff check src/ tests/

# Format
ruff format src/ tests/

License

MIT