Turbowebfetch

npm version npm downloads License GitHub stars

Real browsers. Real content. Full fidelity.

Your AI agents need to read web pages. Documentation, product info, articles, research. But standard fetch tools use plain HTTP - they cannot handle modern client-side rendering or bot mitigation layers, and return empty shells.

TurboWebFetch runs actual Chrome browsers. Your agents see what users see.

14 parallel browsers. Zero API keys. Runs locally.

---

Prerequisites

Before installing, verify you have:

node --version    # Need 18+
python3 --version # Need 3.8+

Google Chrome must be installed (not Chromium).

---

Quick Start

claude mcp add turbowebfetch npx turbowebfetch

That's it. Your agents now have access to the fetch and fetch_batch tools.

---

What This Is (And Isn't)

TurboWebFetch helps your AI agents access content you have the right to access. It renders JavaScript-heavy pages that standard tools cannot handle.

It is for:

• Fetching documentation that requires JS rendering (React, Stripe, etc.)
• Product research on e-commerce sites
• Reading articles and news behind JS walls
• Multi-source research for your AI agents

It is not for:

• Circumventing paywalls
• Scraping data you don't have permission to collect
• High-volume data harvesting (rate-limited by design)
• Violating websites' Terms of Service

The challenge-handling exists because many legitimate sites use broad bot mitigation that affects even authorized access. If a site restricts access and you don't have permission, respect that.

---

WebFetch vs TurboWebFetch

Scenario	WebFetch	TurboWebFetch
Static HTML pages	Works	Works (overkill)
JavaScript SPAs	Empty content	Full render
Sites with JS challenges	Fails	Negotiates automatically
Bot mitigation layers	Fails	Negotiates automatically
Parallel agents	One at a time	14 simultaneous browsers
JS-heavy sites (docs, e-commerce)	Blocked or empty	Works

Rule of thumb: Use WebFetch for simple pages. Use TurboWebFetch when that fails.

---

Usage

Single page:

mcp__turbowebfetch__fetch(url: "https://react.dev/learn", format: "markdown")

Response:

{
  "success": true,
  "url": "https://react.dev/learn",
  "title": "Quick Start - React",
  "content": "# Quick Start\n\nWelcome to the React documentation...",
  "status": 200
}

Batch (parallel):

mcp__turbowebfetch__fetch_batch(
  urls: [
    "https://react.dev/learn",
    "https://nextjs.org/docs",
    "https://www.target.com/p/some-product"
  ],
  format: "text"
)

All three fetch simultaneously in separate browsers.

---

Parameters

Parameter	Default	Description
url	required	The URL to fetch
format	"text"	"text", "markdown", or "html"
timeout	60000	Milliseconds. Increase to 90000+ for slow sites
wait_for	-	CSS selector to wait for (rarely needed)

The tool auto-detects when content has loaded. Use wait_for only if auto-detection fails on a specific site.

---

Known Limitations

Sites that don't work:

• Login-required content - This tool doesn't handle authentication
• Interactive CAPTCHAs - It handles JS challenges, not image selection tasks
• Zillow - Requires interactive verification
• Bloomberg - Requires interactive verification

Performance:

• Adds 5-10 seconds per page (browser startup + rendering + human-like behavior)
• Memory usage: ~200-400MB per browser instance
• For 14 parallel fetches, expect ~4GB RAM usage

Not for scale: This is a user assistant, not a scraping service. Rate-limited to 60 requests/minute per domain.

---

Configuration

Optional environment variables:

Variable	Default	Description
TURBOFETCH_MAX_PROCESSES	14	Max concurrent browsers
TURBOFETCH_HUMAN_MODE	true	Human-like scrolling/delays
TURBOFETCH_HEADLESS	true	Headless mode (auto-switches if blocked)

Most users won't need to change these.

---

Troubleshooting

"Python not found"

# macOS
brew install python3

# Ubuntu/Debian
sudo apt install python3 python3-venv

"Chrome not launching"

Install Google Chrome from https://google.com/chrome (not Chromium).

"Content is empty"

Some heavily lazy-loaded sites need an explicit selector:

mcp__turbowebfetch__fetch(
  url: "https://www.bestbuy.com/site/searchpage.jsp?st=laptop",
  wait_for: "[class*=\"product\"]",
  timeout: 90000
)

"Page not loading on [site]"

Some sites require interactive verification that automated browsers cannot complete. Open an issue with the URL.

---

How It Works

1. Your agent calls the MCP tool
2. TurboWebFetch spawns a Python process with Chrome (via nodriver)
3. Chrome loads the page, executes JavaScript, negotiates any browser challenges
4. Content is extracted and returned as clean text/markdown/HTML
5. Browser closes, process exits

Each fetch is isolated. No cookies or state persist between requests.

---

Development

Clone and build locally:

git clone https://github.com/aza-ali/turbowebfetch.git
cd turbowebfetch
npm install
npm run build

Python setup (virtual environment + nodriver) runs automatically during npm install. If it fails, run manually:

npm run setup:python

Then register with Claude Code:

claude mcp add turbowebfetch node /path/to/turbowebfetch/dist/index.js

---

License

MIT License. See LICENSE for details.

Copyright (c) 2026 Mourtaza Ali