Bright Data Server

Free-first web access for AI agents. Bright Data only when a site actually blocks you.

Scout is a Model Context Protocol server that gives your AI agent a full browser and scraping toolkit. Most pages are fetched for free with a plain HTTP request. The paid Bright Data tier only kicks in when an anti-bot wall (Cloudflare, Akamai, CAPTCHA, etc.) is detected — which means most research workflows cost nothing.

No API key required to get started. Free tools work out of the box. Configure Bright Data only when you need it.

---

How routing works

Every fetch follows a cost ladder and stops at the first rung that works:

1. Free direct fetch  →  realistic browser headers, no cost
2. Bright Data        →  paid proxy + unlocker, only on detected block

An in-memory skip-list (persisted to disk) remembers which domains blocked the free tier so repeat calls go straight to Bright Data — no wasted attempt. Skip-list entries expire after 30 minutes so the server re-probes in case a site drops its protection.

---

Tools

Smart (cost-aware) tools

Tool	Description
smart_scrape	Start here. Free-first fetch with automatic Bright Data fallback. Supports content_only format (main article text, strips ads/nav), disk caching, robots.txt checking, per-domain rate limiting, and PDF auto-escalation.
smart_scrape_batch	Fetch up to 50 URLs in parallel with the same free-first routing. Returns a cost summary. Much faster than looping smart_scrape.
smart_crawl	Start at a URL and follow links up to a configurable depth and page limit. Stays on the same host by default; accepts a URL regex filter.
smart_diff	Fetch a page and compare it to the last cached version. Stores a baseline on first call; detects changes on every subsequent call.
smart_extract	Extract structured metadata from a page: JSON-LD schema.org objects, Open Graph tags, Twitter card meta, and standard meta tags. Returns clean JSON.
parse_feed	Fetch and parse an RSS 2.0 feed, Atom 1.0 feed, or XML sitemap. Returns structured JSON — no HTML parsing needed.
check_robots	Check whether a URL is permitted by the site's robots.txt. Result is cached for 24 hours.
smart_scrape_skiplist	Dump the current skip-list of hard-blocked domains for cost debugging.
serp_search	Web search. DuckDuckGo queries try a free direct fetch first. Google, Bing, and Yandex always use the paid Bright Data SERP API.

Bright Data tools (always paid)

Tool	Description
unlocker_scrape	Force a fetch through Bright Data's Web Unlocker. Use when you specifically need a screenshot, or when smart_scrape has already confirmed a site is hard-blocked.
unlocker_scrape_async	Start a long-running unlock job. Returns a response_id.
unlocker_get_async_result	Poll for and retrieve the result of an async unlock job.
unlocker_success_rate	Per-domain success-rate statistics from the last 7 days.
web_scraper_trigger	Trigger a Bright Data dataset / Web Scraper crawl job over one or more URLs.
web_scraper_get_results	Check progress and download results for a crawl job by snapshot_id.
browser_scrape	Drive a real remote cloud browser: navigate, click, type, scroll, run JS, auto-solve CAPTCHAs, emulate devices, block ads, return rendered HTML / text / screenshot. Requires BRIGHTDATA_BROWSER_AUTH.
web_data_* (35 tools)	Pre-built structured scrapers for Amazon, LinkedIn, Google Maps, Instagram, Reddit, and more. Returns clean JSON — no setup.

---

Quickstart

git clone https://github.com/AndrewEstopinan/browser-mcp
cd browser-mcp
npm install && npm run build
node dist/index.js   # works immediately — no API key needed for free tools

To unlock paid tools, set BRIGHTDATA_API_KEY:

node --env-file=.env dist/index.js

---

Configuration

Copy .env.example to .env and fill in what you need.

Variable	Required	Default	Description
BRIGHTDATA_API_KEY	No	—	Enables paid tools. Free tools work without it.
BRIGHTDATA_UNLOCKER_ZONE	No	web_unlocker1	Web Unlocker zone name.
BRIGHTDATA_SERP_ZONE	No	serp_api1	SERP API zone name.
BRIGHTDATA_BROWSER_AUTH	For browser_scrape	—	USER:PASS from the Browser API zone.
BRIGHTDATA_BROWSER_HOST	No	brd.superproxy.io:9222	CDP host.
BRIGHTDATA_DATASET_ID	For crawl tools	—	Default dataset id (gd_...).
BRIGHTDATA_API_BASE_URL	No	https://api.brightdata.com	Override API base.
SMART_SKIP_THRESHOLD	No	1	Blocks before a domain is skip-listed.
SMART_SKIP_TTL_MS	No	1800000	Skip-list entry lifetime in ms (30 min).
SMART_SKIP_FILE	No	./smart_skip_list.json	Persisted skip-list path.
SMART_CACHE_DIR	No	./smart_cache	Response cache directory.
SMART_CACHE_TTL_MS	No	3600000	Default cache TTL in ms (1 hour).

---

Add to your MCP client

Add this to your Claude Desktop, Cursor, or other MCP client config:

{
  "mcpServers": {
    "scout": {
      "command": "node",
      "args": ["/absolute/path/to/browser-mcp/dist/index.js"],
      "env": {
        "BRIGHTDATA_API_KEY": "your_api_key_here",
        "BRIGHTDATA_UNLOCKER_ZONE": "web_unlocker1",
        "BRIGHTDATA_SERP_ZONE": "serp_api1",
        "BRIGHTDATA_BROWSER_AUTH": "brd-customer-XXXX-zone-YYYY:zzzz"
      }
    }
  }
}

For free-only use, omit the env block entirely.

---

Example calls

Fetch an article (content only, no nav/ads):

{ "tool": "smart_scrape", "arguments": { "url": "https://example.com/article", "data_format": "content_only" } }

Fetch 10 URLs in parallel:

{ "tool": "smart_scrape_batch", "arguments": { "urls": ["https://a.com", "https://b.com"], "concurrency": 5 } }

Crawl a docs site up to 20 pages:

{ "tool": "smart_crawl", "arguments": { "start_url": "https://docs.example.com", "max_pages": 20, "max_depth": 3 } }

Check if a page changed since last time:

{ "tool": "smart_diff", "arguments": { "url": "https://example.com/pricing" } }

Extract JSON-LD and Open Graph metadata:

{ "tool": "smart_extract", "arguments": { "url": "https://example.com/product/123" } }

Parse an RSS feed:

{ "tool": "parse_feed", "arguments": { "url": "https://example.com/feed.xml" } }

Drive a cloud browser through a CAPTCHA:

{
  "tool": "browser_scrape",
  "arguments": {
    "url": "https://hard-site.example",
    "solve_captcha": true,
    "screenshot": true,
    "actions": [
      { "type": "wait_for_selector", "selector": "#results" },
      { "type": "click", "selector": "button.load-more" }
    ]
  }
}

---

Responsible use

Scout accesses public web data through a paid, compliant commercial service. Respect each target site's Terms of Service and robots.txt (use the built-in check_robots tool), applicable laws, and Bright Data's Acceptable Use Policy. Don't use it to access content behind logins you aren't authorized to use, or to collect personal data without a lawful basis.

---

License

MIT