Proxy Browser Server

An MCP server that lets Claude browse the web and make GET / POST / PUT / PATCH / DELETE / HEAD requests while looking like a real Chrome browser at the network layer — so you don't get anti-bot flagged on sites like AliExpress.

• Real Chrome TLS/HTTP2 fingerprint (via curl_cffi), not just a spoofed User-Agent
• Headless-browser render tier — falls back to a real, undetected Chromium (patchright) that executes JavaScript and passes JS anti-bot challenges (Cloudflare, DataDome, AliExpress)
• Anti-bot challenge detection — flags captcha/punish walls instead of returning the stub as content
• Persistent cookie session — multi-step browsing looks like one continuous browser
• Token-efficient output — HTML is cleaned into markdown by default (raw available)
• Streamed, byte-capped body reads — an oversized/binary response can't blow up memory
• Automatic retries with backoff + jitter (honors Retry-After) for idempotent methods
• Randomized think-time + a hard 1–2 request cap per call — a constant, bursty cadence is itself a bot tell
• Optional upstream proxy — plug in a residential/rotating proxy for IP rotation
• Full verb set with form or JSON bodies

---

Why not just requests + a Chrome User-Agent?

Because a UA string fools nobody serious. Cloudflare, Akamai, and AliExpress fingerprint your TLS handshake (JA3/JA4) and HTTP/2 frame ordering — a stock Python HTTP client is instantly identifiable as a bot no matter what User-Agent header it sends.

This server uses curl_cffi, which drives libcurl with a real Chrome TLS + HTTP/2 fingerprint. The connection itself looks like Chrome — verified against tls.peet.ws (JA4 t13d1516h2, Chrome's Akamai HTTP/2 fingerprint, over h2).

---

Requirements

• Python 3.9+ (developed/tested on 3.11, Windows 11)

• pip

• Claude Code CLI (to consume the MCP server)

• Python packages (installed via requirements.txt):

  Package	Role
  mcp	Official MCP Python SDK (FastMCP server)
  curl_cffi	Chrome TLS/HTTP2 impersonation engine (bundled libcurl)
  trafilatura	Main-content extraction → markdown (token-efficient)
  beautifulsoup4	Full-page DOM cleanup
  markdownify	HTML → markdown conversion

curl_cffi ships its own libcurl build, so there is no external C dependency to install.

---

Installation

# 1. Clone the repo
git clone https://github.com/nullsection/headless-proxy-mcp.git
cd headless-proxy-mcp

# 2. (recommended) create a virtual environment
python -m venv .venv
.\.venv\Scripts\Activate.ps1        # PowerShell
# source .venv/bin/activate         # macOS/Linux

# 3. Install dependencies
python -m pip install -r requirements.txt

---

Setup — register with Claude Code

# User scope = available in all your projects
claude mcp add -s user proxy-browser -- python C:/path/to/headless-proxy-mcp/proxy_browser_server.py

# Verify
claude mcp get proxy-browser        # should show: Status: Connected

Use forward slashes in the path — a backslash path can be mangled by the shell.

To pass configuration (e.g. a proxy), add -e flags:

claude mcp add -s user proxy-browser \
  -e PROXY_URL=http://user:pass@host:port \
  -e IMPERSONATE=chrome124 \
  -- python C:/path/to/headless-proxy-mcp/proxy_browser_server.py

Restart Claude Code after registering so it loads the server.

---

Usage

Once registered and restarted, just ask Claude in natural language — e.g. "browse aliexpress for usb cables and summarize the top results." Claude selects the right tool automatically.

Tools

Tool	Description
browse	HTTP GET a URL, with an optional headless-browser fallback (render policy).
render	Load a URL in a real headless browser (executes JS; passes JS challenges).
http_post	HTTP POST with form data or json body.
http_put	HTTP PUT with form data or json body.
http_patch	HTTP PATCH with form data or json body.
http_delete	HTTP DELETE (optional data or json body).
http_head	HTTP HEAD — response headers only (liveness / type / redirect).
session_reset	Clear cookies / start a fresh browsing identity.
session_info	Show impersonation target, proxy, config, and current cookies.

Retries & idempotency: browse, http_put, http_delete, and http_head are idempotent, so transient 429/5xx/network failures are retried automatically with exponential backoff (honoring Retry-After). http_post and http_patch are not retried, to avoid duplicate side effects.

Parameters

• url — full URL including http:// or https://.
• extract — how the body is returned:
  • "auto" (default) — main content as markdown (best for pages)
  • "full" — the whole page converted to markdown
  • "text" — plain text only
  • "raw" — exact HTML / response bytes
• headers — optional extra request headers (merged over sane Chrome defaults).
• proxy — optional per-request proxy URL (overrides PROXY_URL for that call).
• timeout — request timeout in seconds (default 30).
• data / json (POST/PUT only) — form fields or a JSON body (use one).

---

Configuration

Set via environment variables (or -e flags at registration). See .env.example.

Variable	Default	Description
IMPERSONATE	chrome131	Chrome build to mimic (e.g. chrome124, chrome131, chrome136). Keep it recent.
PROXY_URL	(empty)	Upstream proxy: http://user:pass@host:port or socks5://host:port. Empty = direct.
MAX_CHARS	60000	Max characters of the extracted body returned to Claude (protects context).
MAX_RESPONSE_BYTES	10000000	Hard cap on raw bytes read off the wire; body is streamed and reading stops past this.
PER_HOST_DELAY	0	Minimum seconds between requests to the same host (politeness / anti-flag).
PER_HOST_JITTER	0.3	Randomize the per-host delay by ±this fraction (0.3 = ±30%) so cadence isn't constant.
MAX_RETRIES	2	Additional attempts after the first on 429/5xx/network errors (bounded by MAX_REQUESTS_PER_CALL).
RETRY_BACKOFF	0.5	Base retry backoff in seconds, grown exponentially with jitter.
MAX_RETRY_WAIT	30	Cap on any single retry sleep, even if Retry-After asks for more.
REQUEST_JITTER_MIN / REQUEST_JITTER_MAX	0.5 / 2.0	Randomized think-time (s) before every request — fires even on a single hit, so nothing goes out with bot-like zero latency. MAX=0 disables.
MAX_REQUESTS_PER_CALL	2	Hard ceiling on total HTTP hits per tool call (initial + retries), clamped to 1–2 to stay under anti-bot thresholds.
RENDER_MODE	auto	Browser-fallback policy for browse: auto | never | always.
RENDER_WAIT_UNTIL	domcontentloaded	Load state renders wait for; use networkidle for XHR-driven grids.
RENDER_HEADLESS	1	0 = headful window (max stealth, needs a display).
RENDER_CHANNEL	(bundled)	chrome/msedge to drive the real installed browser (new headless, real GPU). Recommended for hard walls.
RENDER_USER_DATA_DIR	(none)	Persistent Chrome profile dir — banks anti-bot cookies + logins across runs (the PerimeterX lever).
RENDER_EXTRA_ARGS	(none)	Extra Chromium flags, comma-separated (e.g. --use-angle=d3d11 for real-GPU WebGL on Windows).
RENDER_LOCALE / RENDER_TIMEZONE / RENDER_VIEWPORT / RENDER_UA	en-US / America/New_York / 1280x800 / (auto)	Realistic browser context. UA defaults to the engine's real UA with the HeadlessChrome tell stripped.

See .env.example for the full render knob list (RENDER_TIMEOUT, RENDER_WAIT_MS, RENDER_MIN_CONTENT_CHARS).

---

Troubleshooting

• Failed to connect — the path was likely mangled by backslashes. Re-add using forward slashes (see setup above), and confirm python is on your PATH.
• Server connects but a site still blocks you — you're probably being flagged by IP, not fingerprint. Set PROXY_URL to a residential/rotating proxy and raise PER_HOST_DELAY.
• Page comes back blank / missing content — the site renders via JavaScript. curl_cffi does not execute JS (see Limitations).
• Response truncated — raise MAX_CHARS, or request a tighter extract mode.

---

Headless browser render tier

The HTTP engine mimics Chrome's network identity but can't run JavaScript, so JS-rendered pages and JS challenge walls defeat it. The optional render tier fixes that by driving a real, undetected Chromium (patchright, a patched Playwright drop-in) that executes JS, passes challenges, and hides the automation tells (navigator.webdriver, the HeadlessChrome UA).

pip install -r requirements-render.txt
patchright install chromium          # one-time browser download (or: playwright install chromium)

• browse(url, render="auto") (default) — tries fast HTTP first, then transparently escalates to the browser when the page looks challenge-walled or JS-thin and an engine is installed. render="never" = HTTP only; render="always" = browser only.
• render(url, wait_selector=".product-card", wait_ms=4000) — explicit browser load. Use wait_selector/wait_ms (or RENDER_WAIT_UNTIL=networkidle) for grids that populate via XHR after load.

Verified live: this tier passes AliExpress's _tmd_/punish captcha wall that blocks the HTTP client, and renders the JS product grid. navigator.webdriver is hidden and the UA is cleaned to a real Chrome string.

Getting past hard walls (PerimeterX / DataDome / Facebook)

For aggressive bot managers, the render tier has a persistent-profile + organic-navigation mode. Verified live getting past Carsales' PerimeterX wall (which 403s a cold session):

• RENDER_CHANNEL=chrome + RENDER_USER_DATA_DIR=<dedicated dir> + RENDER_EXTRA_ARGS=--use-angle=d3d11
• render(url, warm=[...homepage, category...]) — step down to the deep page so the anti-bot cookie matures instead of cold-linking.
• render(url, press=["Escape"]) — dismiss Facebook Marketplace's login overlay.

See docs/anti-bot-bypass.md for the full playbook — the wall tiers, the detection-signal table, the exact PerimeterX recipe, pacing/cooldown notes, and when to reach for a managed unblocker instead.

Testing

Unit tests are hermetic — they stub curl_cffi/mcp and mock the browser, so no native libs, browser, or network are needed. They cover the request/retry loop, the streamed byte cap, Retry-After handling, challenge detection, extraction dispatch, the tiered render fallback, and the render engine (including its asyncio-safety) — ~92% line coverage across both modules.

python -m pip install -r requirements-dev.txt
python -m pytest tests/                       # fast, offline
python -m coverage run -m pytest tests/ ; python -m coverage report proxy_browser_server.py

Live smoke test (AliExpress)

tests/smoke_aliexpress.py drives the real server against AliExpress — exercising the full Chrome-impersonation path, retries, byte cap, and HTML→markdown extraction end to end. Run it from the same env the MCP runs in (needs the runtime deps installed):

python tests/smoke_aliexpress.py                       # "moza racing steering wheel", cheapest-first
python tests/smoke_aliexpress.py "moza r9 wheelbase"   # custom query
python tests/smoke_aliexpress.py --raw "moza ks wheel" # raw HTML instead of markdown

The same check runs under pytest when opted in:

$env:RUN_SMOKE=1 ; python -m pytest tests/test_smoke.py::test_live_aliexpress_moza

AliExpress is heavily JS-rendered and anti-bot defended. The HTTP path gets a real 200 but thin/challenge HTML; the render tier passes the wall and returns the full product grid.

Live render test (opt-in)

With the render deps installed, exercise the real browser end to end:

$env:RUN_RENDER=1 ; python -m pytest tests/test_browser_engine.py::test_live_render_executes_js_and_returns_html

Limitations

• JavaScript is handled by the render tier, not the HTTP engine. Plain curl_cffi can't run JS; install the render deps (patchright + Chromium) and use render="auto"/render="always" for JS-rendered or challenge-walled sites. Without the render deps installed, JS-heavy pages come back thin (and are flagged when it's a known anti-bot wall).
• Render tier is heavier. A browser launch costs ~1–3 s and real memory; it only fires on demand (challenge/thin page in auto, or explicit render), not on every request.
• IP reputation still matters. Even the browser can be IP-flagged after abuse; for the most aggressive sites pair with a residential PROXY_URL and/or RENDER_HEADLESS=0.
• Respect Terms of Service and rate limits. This tool is for legitimate automation and research; the built-in per-host throttle, request jitter, and 1–2 request cap keep it polite — don't hammer sites.

---

Project layout

headless-proxy-mcp/
├── proxy_browser_server.py   # the MCP server — HTTP engine, tools, tiered fallback
├── browser_engine.py         # headless-browser render engine (patchright/playwright)
├── requirements.txt          # runtime dependencies (HTTP engine)
├── requirements-render.txt   # optional: headless browser render tier
├── requirements-dev.txt      # test dependencies (pytest, coverage)
├── .env.example              # configuration reference
├── .gitignore
├── README.md
├── docs/
│   └── anti-bot-bypass.md    # playbook: wall tiers, PerimeterX recipe, FB trick
└── tests/
    ├── conftest.py            # stubs curl_cffi/mcp so tests run offline
    ├── test_proxy_browser.py  # hermetic unit tests — HTTP engine + tiered fallback
    ├── test_browser_engine.py # hermetic unit tests — render engine (mocked browser)
    ├── smoke_aliexpress.py    # live smoke test runner (real network)
    └── test_smoke.py          # smoke-test unit coverage + opt-in live check

License

MIT (or your preference).