Cloudflare Crawl

MCP server for Cloudflare Browser Rendering Crawl API. Fetches and crawls web pages, returning clean Markdown optimized for LLM consumption.

Built with FastMCP and uv.

Tools

Tool	Description
scrape_url	Fetch a single page as Markdown. The primary tool — use this when you know the URL.
map_url	Discover URLs on a site without fetching content. Use to find the right page first.
crawl_url	Crawl multiple pages and return all content as Markdown.

Typical workflow: map_url to find pages → scrape_url to read the right one.

Prerequisites

1. A Cloudflare account with Browser Rendering enabled
2. An API token with Account > Browser Rendering > Edit permission (create one here)
3. uv installed

Configuration

Variable	Required	Description
CF_API_TOKEN	Yes	Cloudflare API token
CF_ACCOUNT_ID	Yes	Cloudflare Account ID
CF_RATE_LIMIT	No	API requests per minute (default: 6 for Free, set to 600 for Paid)

Setup

Claude Code

claude mcp add cloudflare-crawl \
  -e CF_API_TOKEN=your_api_token \
  -e CF_ACCOUNT_ID=your_account_id \
  -- uv run --directory /path/to/cloudflare-crawl-mcp python server.py

Codex

codex mcp add cloudflare-crawl \
  -- env CF_API_TOKEN="your_api_token" CF_ACCOUNT_ID="your_account_id" \
  uv run --directory /path/to/cloudflare-crawl-mcp python server.py

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "cloudflare-crawl": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/cloudflare-crawl-mcp", "python", "server.py"],
      "env": {
        "CF_API_TOKEN": "your_api_token",
        "CF_ACCOUNT_ID": "your_account_id"
      }
    }
  }
}

Cursor

Add to .cursor/mcp.json in your project or ~/.cursor/mcp.json globally:

{
  "mcpServers": {
    "cloudflare-crawl": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/cloudflare-crawl-mcp", "python", "server.py"],
      "env": {
        "CF_API_TOKEN": "your_api_token",
        "CF_ACCOUNT_ID": "your_account_id"
      }
    }
  }
}

Tool Reference

scrape_url

Fetch a single page and return its content as Markdown.

url (string, required)    — The URL to fetch
render (boolean, optional) — Render JavaScript with headless browser (default: true, set false for static pages)

map_url

Discover URLs on a website without fetching full content.

url (string, required)                  — Starting URL
limit (number, optional)                — Max URLs to discover (default: 50)
depth (number, optional)                — Link depth to follow (default: 2)
include_subdomains (boolean, optional)  — Follow subdomain links (default: false)
include_external_links (boolean, optional) — Follow external links (default: false)
include_patterns (string[], optional)   — Only visit matching URLs (e.g. "https://example.com/docs/**")
exclude_patterns (string[], optional)   — Skip matching URLs

crawl_url

Crawl multiple pages and return all content as Markdown.

url (string, required)                  — Starting URL
limit (number, optional)                — Max pages to crawl (default: 10)
depth (number, optional)                — Link depth (default: 1)
include_subdomains (boolean, optional)  — Follow subdomain links (default: false)
include_external_links (boolean, optional) — Follow external links (default: false)
include_patterns (string[], optional)   — Only visit matching URLs
exclude_patterns (string[], optional)   — Skip matching URLs
render (boolean, optional)              — Render JavaScript (default: true)

Cloudflare Plan Limits

	Free	Paid
Browser time	10 min/day	10 hrs/month
API rate limit	6 req/min	600 req/min
Concurrent browsers	3	10
Max pages per job	100,000	100,000
Max job duration	7 days	7 days
Results available	14 days	14 days

render: false crawls run on Workers instead of a headless browser and do not consume browser time.

License

MIT