Ai Browser Gateway

Fetch web pages for AI agents with HTTP first, a headless browser when needed, and a provider ladder for JavaScript rendering and Cloudflare challenges.

License: MIT Container: GHCR

Why

Use the cheapest sufficient way to read a page: start with HTTP and browser headers, then launch a browser when the response needs it. The initial benchmark found that browsers added value through JavaScript rendering; they did not improve coverage over HTTP with browser headers on that set of live targets. A later Scrapling measurement added a Cloudflare target that Patchright could not fetch. See Benchmarks for the evidence and its limits.

The one-shot container runs the provider ladder locally. It needs no gateway API, token, or Docker daemon inside the container.

When a plain fetch is not enough

A one-line HTTP client is enough when the response already contains the page. It is not enough when the body is a refusal or a shell. The signs below are the ones this repository already records.

• An HTTP 403, often with a challenge signal. The ladder treats HTTP 403 and a suspected challenge as a reason to try a browser, when browsers are allowed and budget remains. That next step is an attempt, not a guarantee.
• An interstitial instead of the page. A Cloudflare interstitial recorded in the research notes is titled Just a moment. HTTP 200 is not proof of success: the Russian notes say a Cloudflare stub can also arrive as 200, and it can be longer than the real page.
• A response that arrived without the page text, because that text is produced by JavaScript. The Russian notes call this content_missing: the bytes came back, and the content was not in them. The initial benchmark found the browser's value in that rendering, not in extra coverage over HTTP that already sent browser headers.
• A page behind a login wall. This gateway does not log in. Responsible use states the same limit for interactive challenges: no CAPTCHA-solving service is integrated, and a challenge that needs a person stops here.

Quick start

Read a page as Markdown:

docker run --rm ghcr.io/stufently/ai-browser-gateway-oneshot:latest https://example.com/ --format markdown

For a site behind Cloudflare, allow a larger budget; the default 30 seconds leaves little room for the browser steps:

docker run --rm ghcr.io/stufently/ai-browser-gateway-oneshot:latest https://cloudflare-protected.example/ --format markdown --budget-ms 90000

Or build and run from the repository root:

docker build -f deploy/Dockerfile.oneshot -t ai-browser-gateway-oneshot .
docker run --rm ai-browser-gateway-oneshot https://example.com/ --format markdown

CLI reference

The container entry point runs python3 -m gateway.oneshot:

python3 -m gateway.oneshot URL [--format MODE] [--expected-text TEXT] [--budget-ms N] [--no-browser]

URL is required. Running the module outside the image requires the image's dependencies and probe setup.

Flag	Default	Meaning
--format	text	Output content as text, html, markdown, links, or meta.
--expected-text	Unset (null)	Require this nonblank text in the page HTML or text.
--budget-ms	30000	Total request budget in milliseconds, from 1 to 180000.
--no-browser	Off (browsers allowed)	Use only the HTTP provider.

For exit codes 0 and 1, stdout contains one JSON line with fields ok, url, final_url, provider, age_hours, error_type, step, elapsed_ms, format, content, and attempts. The selected format changes content; the surrounding JSON remains. attempts records provider outcomes and ladder decisions.

Exit code	Meaning	stderr
0	ok=true: page accepted.	No CLI error.
1	ok=false: page not delivered. Inspect the JSON outcome.	No CLI error.
2	Invalid arguments.	{"error": "invalid_request"}
3	Internal error.	{"error": "internal_error"}
4	Interrupted by SIGTERM, SIGINT, or SIGHUP during the ladder.	{"error": "interrupted"}

For codes 2 through 4, stdout is empty and stderr contains the single JSON error line shown above. --help is not supported: it returns code 2 and invalid_request instead of usage text.

How the provider ladder works

The one-shot ladder stops as soon as a page is accepted, or when policy or the budget prevents further attempts. An HTTP 200 alone does not establish success: the gateway also checks content and challenge signals.

Provider	What it does	When it runs
curl_cffi	HTTP with browser headers and TLS impersonation.	First; its attempt is capped at 15000 ms within the total budget.
patchright	Browser rendering for JavaScript pages; on a JavaScript challenge (Qrator's 401 loader) it waits up to 15 s for the page to reload.	After an eligible HTTP failure, such as missing content, a suspected challenge, a JavaScript challenge, HTTP 403, or an HTTP timeout, if browsers are allowed and budget remains.
scrapling	Stealthy browser session with solve_cloudflare=True.	After an eligible Patchright failure, if budget remains.

Some outcomes stop escalation: interactive challenges need human action; HTTP 429 skips browser attempts, unless the body is a PerimeterX block page (then it is a suspected challenge, and a browser is tried); Akamai's edge "Access Denied" is reported as ip_blocked (it routes like any HTTP 403); connection errors and browser timeouts do not automatically advance to the next browser. The ladder does not guarantee access to every Cloudflare-protected page.

--no-browser removes both browser providers. The one-shot image makes direct requests, with no RSS/Wayback entrances or cache. Environment variables http_proxy, https_proxy, all_proxy, ftp_proxy, and no_proxy are ignored in every letter case. A proxy is used only when ABG_EGRESS_PROXY holds its URL (http, https, socks5, socks5h): the ladder then ends with one curl_cffi attempt through it (profile proxy), for sites that block the host's address. Pass it as docker run -e ABG_EGRESS_PROXY so the credentials stay out of the command line.

Reusing a solved challenge

When a browser step passes a challenge, the cookies and User-Agent it ended with can let the cheap HTTP step read the same host next time. Measured on 2026-10-09: bizprofile.net (Cloudflare cf_clearance) took 21.5 s through Scrapling, then 1.2–1.4 s through curl_cffi alone; rbc.ru (Qrator) took 6.2 s, then about 1 s. The self-hosted service keeps these sessions in memory. The one-shot image keeps them only in a directory you mount and name in ABG_SESSION_DIR:

mkdir -m 700 -p ~/.cache/abg-sessions
docker run --rm --user "$(id -u):$(id -g)" -v ~/.cache/abg-sessions:/sessions \
  -e ABG_SESSION_DIR=/sessions ghcr.io/stufently/ai-browser-gateway-oneshot:latest \
  https://cloudflare-protected.example/ --format markdown --budget-ms 90000

A session belongs to one host and lasts at most 30 minutes. Only direct https:// steps use one, because cookies are bound to the address that solved them and the stored cookies keep no Secure flag. A page reached by redirect to another host leaves no session. A replay that does not pass discards the session, and the ladder continues as usual. Each file is 0600 and holds cookies of an anonymous visit; a directory that another user owns or can write to is ignored. Between gateway and provider the session travels in an environment variable, the same way as the proxy URL: anyone who can run docker inspect on the host can read it while the provider container lives.

MCP server for AI agents

scripts/abg-mcp (Python 3.12+, stdlib only) exposes one MCP tool, fetch_page, over stdio. It calls the existing gateway API and its full provider ladder. Add a stdio server to your MCP client's configuration, using absolute paths:

{
  "mcpServers": {
    "ai-browser-gateway": {
      "command": "/path/to/ai-browser-gateway/scripts/abg-mcp",
      "env": {
        "ABG_URL": "http://127.0.0.1:8765/v1/fetch",
        "ABG_TOKEN_FILE": "/path/to/abg/client-token"
      }
    }
  }
}

From the checkout, python3 -m gateway.mcp_stdio is equivalent. The API must already be running; the adapter opens no listening socket. ABG_TOKEN takes precedence over ABG_TOKEN_FILE, whose default is ~/.config/abg/client-token. The API URL above is the default. HTTP uses no system proxy or redirects.

fetch_page requires url; optional arguments are format (text|html|markdown|links|meta, default text), expected_text (default null), budget_ms (1–180000, default 30000), allow_browser (default true), and max_age_hours (default 0). Fetch options come from tool arguments. Results include page content plus provider/attempt metadata; gateway and transport failures are tool errors. Credentials are redacted from results. Tool calls run concurrently (up to 8), while ping is answered immediately. Invalid tool arguments are reported as tool errors with isError: true. Page text (or error text) appears in both the text content block and structuredContent.content, alongside the available provider/attempt metadata. The server negotiates MCP 2025-11-25 or 2025-06-18, falling back to 2025-11-25 for an unsupported version. Stdout carries only JSON-RPC lines.

Self-hosted HTTP API

The gateway service exposes POST /v1/fetch with Bearer authentication and a JSON request containing url and optional fetch settings. Use scripts/abg-fetch URL [text|html|markdown|links|meta] to call it from the shell. The MCP adapter uses this service too. See the Russian deployment documentation for setup and operations.

How it compares

This table compares only clients the repository runs. The first row is the ordinary HTTP client that sits below the ladder. The other three rows are the ladder itself, in the order How the provider ladder works already states. No success rate is claimed here; the samples and their limits are in Benchmarks.

Approach	How it reaches the page	What it is for	When it runs	What you pay
Ordinary HTTP client	HTTP with no TLS-fingerprint impersonation and no browser.	The lower bound. It reads a page only when the server already returns that page to a normal client. A challenge can end as HTTP 403 or an interstitial.	Not a ladder step. It is the baseline the ladder is measured against.	One short request, and no browser process.
curl_cffi	HTTP with browser headers and TLS impersonation.	The first attempt inside the ladder.	First. That attempt is capped at 15000 ms within the total budget.	The cheapest step that still speaks HTTP.
patchright	A browser session that renders JavaScript.	An HTTP result that was not accepted: missing content, a suspected challenge, HTTP 403, or an HTTP timeout.	After an eligible HTTP failure, if browsers are allowed and budget remains.	A browser process, taken from the time that remains.
scrapling	A stealthier browser session with solve_cloudflare=True.	A page the Patchright attempt did not accept.	After an eligible Patchright failure, if budget remains.	A second browser session, and only while budget remains.

Some outcomes never reach the next row. An interactive challenge needs a person. HTTP 429 skips the browser steps, except for a PerimeterX block page. A connection error or a browser timeout does not by itself move to the next browser. --no-browser removes both browser providers and leaves the HTTP provider. The ladder does not guarantee access to every Cloudflare-protected page.

Benchmarks

• In the initial five-target live benchmark, HTTP with browser headers fetched four targets, and none of the tested browsers added another. The controlled scenarios showed the browser's value in JavaScript rendering. See the phase 1 verdict.
• In the later six-target comparison, Scrapling with solve_cloudflare=True fetched the Cloudflare-protected Bizprofile target that Patchright failed to fetch, adding one unique target. See the Scrapling measurements.
• A class-level run on 18 targets, 12 of them kept off the repository: pages without protection were fetched by every provider; of the two challenged Cloudflare targets, Scrapling took both, curl_cffi and Patchright one each and plain curl none; login-wall pages (public profile and post metadata) were taken by both browsers but only three or four of six by HTTP clients. Median time per request was 0.4 s for curl_cffi, 2.1 s for Patchright and 7.4 s for Scrapling. See the class-level results.

Class-level summary of that run (targets fetched / targets measured):

Class	Targets	curl	curl_cffi	patchright	scrapling	Any provider
cloudflare	2	0/2	1/2	1/2	2/2	2/2
cloudflare+spa	1	1/1	1/1	1/1	1/1	1/1
cloudflare-passive	1	1/1	1/1	1/1	1/1	1/1
login-wall	6	3/6	4/6	6/6	6/6	6/6
none	8	8/8	8/8	8/8	8/8	8/8

These are small samples from specific runs and network conditions, not a general success-rate promise.

FAQ

Why does a plain HTTP client get 403 when a browser does not?

A plain client sends non-browser headers, shows its own TLS fingerprint and does not run JavaScript, so a check can answer it with HTTP 403 or an interstitial. On the public set, browser headers made the difference: TLS impersonation added no target on top of them. A browser is not an automatic win either: there Patchright added no coverage over HTTP with browser headers, and Scrapling added one challenged Cloudflare target. The later class-level run found browsers ahead on login walls (6/6 against 4/6) and Scrapling ahead on challenged Cloudflare (2/2 against 1/2), at a cost of seconds per request. See Benchmarks.

Does this project solve CAPTCHA and interactive challenges?

No. No CAPTCHA-solving service is integrated. An interactive challenge stops the ladder, because it needs a person. The same limit is stated in Responsible use.

Do I need a server and a token just to read a page?

No. The one-shot image in Quick start runs the provider ladder locally. It needs no gateway API and no token. A token is only for the self-hosted POST /v1/fetch API, and for the MCP adapter that calls it.

How do I connect this to an MCP client?

Add scripts/abg-mcp as a stdio server. It exposes one tool, fetch_page, and the gateway API has to be running already. Arguments, defaults, and protocol versions 2025-11-25 and 2025-06-18 are in MCP server for AI agents.

Can it log in and read pages behind a login wall?

No. The gateway does not log in and does not read a page that requires an authenticated session. A challenge that needs a person is the same kind of stop, described in Responsible use.

How long should I wait for a page behind Cloudflare?

The default budget is 30 seconds (--budget-ms 30000). Once the ladder reaches a browser, that is often too short. Quick start uses 90 seconds (--budget-ms 90000) for a Cloudflare-protected page. The allowed range is 1 to 180000 milliseconds.

Responsible use

Use this gateway for your own sites, monitoring, and page reading by AI agents. Respect each site's terms and robots.txt. No CAPTCHA-solving services are integrated; interactive challenges can require human action.

Development

Run the unit suite and frozen probes from the repository root with the official Python image pinned by digest. Both containers run without network access:

docker run --rm --network none --user 1002:1002 -v "$PWD":/work:ro -w /work -e HOME=/tmp -e PYTHONDONTWRITEBYTECODE=1 -e PYTHONHASHSEED=0 python:3.14.7-slim-bookworm@sha256:82bc3c539b8813ada9d68c63b40158fa002f7f33de9bf3312a3dfdc0620dff56 python3 -m unittest discover -q -s tests -t .
docker run --rm --network none --user 1002:1002 -v "$PWD":/work:ro -w /work -e HOME=/tmp -e PYTHONDONTWRITEBYTECODE=1 -e PYTHONHASHSEED=0 python:3.14.7-slim-bookworm@sha256:82bc3c539b8813ada9d68c63b40158fa002f7f33de9bf3312a3dfdc0620dff56 python3 -m unittest -q tests.probe_m9_transport tests.probe_m10_product tests.probe_m11_api_cli tests.probe_m12_service tests.probe_m12_service_regressions tests.probe_m13_late_container

Documentation

• Russian README: deployment and operations notes.
• Research: the phase-1 verdict, the Scrapling measurements, and the other write-ups already in the tree.
• Changelog: what changed in the project.
• Security: how to report a vulnerability.

License

The project is licensed under the MIT License. Google Chrome bundled in the container image is distributed under Google's terms.