WebFetch Server

An MCP (Model Context Protocol) server that fetches webpages by URL, renders JavaScript using Puppeteer with stealth plugins to avoid bot detection, converts the HTML content to clean Markdown, and returns it for LLM analysis.

Features

• JavaScript Rendering: Uses Puppeteer with stealth plugin to render dynamic content
• Bot Evasion: Stealth plugins prevent detection as a bot
• Markdown Conversion: Converts HTML to clean, readable Markdown using Turndown
• Content Filtering: Removes ads, scripts, styles, and other non-essential elements
• Link Preservation: Maintains links with their URLs for context
• Image References: Preserves image alt text and sources as markdown references

Quick Start

Using Docker with Streamable Transport

Build the image (no server needed to be running):

make build
# or: docker build -t webfetch-mcp-server .

Configure your MCP client to use streamable transport with Docker. The container will be started on-demand by the MCP client.

Using Make

A Makefile is included for easy management:

make build      # Build the Docker image
make clean      # Remove container and image
make logs       # Follow container logs (if running)
make debug      # Run in foreground to see output

Running from GitHub (npx)

Run directly from your GitHub repository without cloning:

npx git+https://github.com/Sh1d0w/webfetch-mcp.git -- -i

Or with the package name:

npx webfetch-mcp-server@latest -- -i

Running Locally

Install dependencies:

npm install

Run in development mode:

npm run dev

Or build and run:

npm run build
npm start

Configuration

Environment Variables

Variable	Description	Default
PUPPETEER_SKIP_CHROMIUM_DOWNLOAD	Skip Chromium download (use system)	true
PUPPETEER_EXECUTABLE_PATH	Path to Chromium executable	/usr/bin/chromium-browser

MCP Client Configuration

Docker Streamable Transport

Configure your MCP client to use streamable transport. The container starts on-demand when the MCP client connects.

Docker run command (for reference):

docker run --rm -i webfetch-mcp-server node dist/index.js

Cursor

Add to your .cursorrules or Cursor MCP settings:

{
  "mcpServers": {
    "webfetch": {
      "command": "docker",
      "args": ["run", "--rm", "-i", "webfetch-mcp-server", "node", "dist/index.js"]
    }
  }
}

Claude Desktop (macOS)

Edit ~/Library/Application Support/Claude/claude_config.json:

{
  "mcpServers": {
    "webfetch": {
      "command": "docker",
      "args": ["run", "--rm", "-i", "webfetch-mcp-server", "node", "dist/index.js"]
    }
  }
}

Claude Desktop (Linux)

Edit ~/.config/claude/claude_config.json:

{
  "mcpServers": {
    "webfetch": {
      "command": "docker",
      "args": ["run", "--rm", "-i", "webfetch-mcp-server", "node", "dist/index.js"]
    }
  }
}

LM Studio

In LM Studio MCP settings:

• Name: webfetch
• Type: stdio (streamable transport)
• Command: docker
• Arguments: run --rm -i webfetch-mcp-server node dist/index.js

Why this works:

• --rm: Automatically removes container after exit (clean state each time)
• -i: Keeps stdin open for MCP communication
• Container starts on-demand when LM Studio connects

opencode

Add to your opencode configuration:

import { StreamableHTTPServerTransport } from '@modelcontextprotocol/sdk/server/streamableHttp.js';

// In your config file
{
  mcpServers: {
    webfetch: {
      command: 'docker',
      args: ['run', '--rm', '-i', 'webfetch-mcp-server', 'node', 'dist/index.js']
    }
  }
}

Why Xvfb is Required

Even in "headless" mode, Chromium/Puppeteer requires an X server for rendering. The Dockerfile includes:

• Xvfb (X Virtual Framebuffer): Provides a virtual display
• DISPLAY=:99: Tells Chromium to use the virtual display
• Startup script: Starts Xvfb before running Node.js

Without Xvfb, Puppeteer will crash immediately after starting, causing the container to restart in a loop.

Available Tools

fetch_url

Fetches a webpage and converts it to Markdown.

Parameters:

• url (string, required): The URL of the webpage to fetch

Returns:

• Clean Markdown content with title and URL header
• Error message if fetching fails

Example Usage:

User: Please analyze this article about MCP protocol
Assistant: [calls tool] fetch_url with url="https://modelcontextprotocol.io"

Output Format

The server returns content in the following format:

# Page Title

https://example.com

---

[Clean markdown content here...]

Content Processing

• Removes <script>, <style>, nav, footer, header, aside elements
• Strips advertisement containers (.ad, .advertisement, #ads)
• Removes cookie consent banners
• Converts links to include their URLs in parentheses
• Preserves images as markdown references with alt text
• Maintains code blocks formatting
• Normalizes whitespace and removes excessive line breaks

Building for Production

Build with Make (recommended):

make build

Or directly with Docker:

docker build -t webfetch-mcp-server .

Configure your MCP client to use the built image with streamable transport. No persistent server needed.

Troubleshooting

Container keeps restarting / "Container is not running"

Cause: Puppeteer needs a virtual display (Xvfb) even in headless mode. Chromium's headless mode still requires an X server for rendering.

Solution: The Dockerfile includes Xvfb automatically. If you're using a custom setup, ensure:

1. Install Xvfb: apk add xvfb (Alpine) or apt-get install xvfb (Debian/Ubuntu)
2. Start Xvfb before running Chromium: Xvfb :99 -screen 0 1024x768x24 &
3. Set DISPLAY env var: export DISPLAY=:99

"Connection closed" / MCP error -32000

Cause: Container crashed when the MCP client tried to connect.

Solution: Ensure you're using docker run --rm in your MCP configuration. This starts a fresh container for each connection, avoiding issues with stale containers.

Check container logs

make logs
# or: docker logs -f webfetch-mcp-server

Connection Timeout

The default timeout is 30 seconds. For slower websites, you may need to increase this in the source code.

Stealth Detection Bypassed

Some websites have advanced bot detection. The stealth plugin helps but cannot guarantee bypass for all sites.

Development

Run tests:

npm test

Lint and type-check:

npm run lint
npm run typecheck

License

MIT License - See LICENSE file for details.