Spider Server

The fastest web crawling, scraping, and browser automation server for AI agents. Gives Claude direct access to the web through 22 tools — crawl sites at 100K+ pages/sec, extract structured data with AI, and control remote browsers with built-in anti-bot bypass.

Why Spider

• Speed — Crawl 100K+ pages per second. Smart request routing picks HTTP or headless Chrome automatically. Streaming responses start delivering data immediately.
• Cost — Pay-per-use credits with no subscription required for core tools. Check your balance anytime with spider_get_credits. AI tools available with an AI subscription.
• Reliability — Anti-bot bypass with fingerprinting and proxy rotation. Browser fleet with automatic fallback across Chrome, Firefox, and more. Built-in retry and stealth escalation.

Quick Start

Claude Code

claude mcp add spider -- npx -y spider-cloud-mcp

Set your API key:

export SPIDER_API_KEY="your-api-key"

Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "spider": {
      "command": "npx",
      "args": ["-y", "spider-cloud-mcp"],
      "env": {
        "SPIDER_API_KEY": "your-api-key"
      }
    }
  }
}

Cursor

Add to .cursor/mcp.json:

{
  "mcpServers": {
    "spider": {
      "command": "npx",
      "args": ["-y", "spider-cloud-mcp"],
      "env": {
        "SPIDER_API_KEY": "your-api-key"
      }
    }
  }
}

Windsurf

Add to ~/.codeium/windsurf/mcp_config.json:

{
  "mcpServers": {
    "spider": {
      "command": "npx",
      "args": ["-y", "spider-cloud-mcp"],
      "env": {
        "SPIDER_API_KEY": "your-api-key"
      }
    }
  }
}

Configuration

Variable	Required	Description
SPIDER_API_KEY	Yes	Your Spider API key. Get one at spider.cloud/api-keys

Tools

Core Tools (8)

These work on pay-per-use credits with no subscription required.

Tool	Description
spider_crawl	Crawl a website and extract content from multiple pages. Follows links up to a depth/limit.
spider_scrape	Scrape a single page. Faster and cheaper than crawling when you need one URL.
spider_search	Search the web. Optionally fetch full page content from results.
spider_links	Extract all links from a page without fetching content.
spider_screenshot	Capture a page screenshot as base64 PNG.
spider_unblocker	Access bot-protected content with advanced anti-bot bypass.
spider_transform	Convert HTML to markdown or text without making web requests.
spider_get_credits	Check your API credit balance.

AI Tools (5)

Natural language web interaction. Describe what you want in plain English.

Requires an AI subscription.

Tool	Description
spider_ai_crawl	AI-guided crawling — describe what content to find.
spider_ai_scrape	Extract structured data with plain English — no CSS selectors.
spider_ai_search	AI-enhanced search with intent understanding.
spider_ai_browser	Automate browser actions with natural language.
spider_ai_links	Find and filter links by description.

Browser Automation Tools (9)

Direct browser control via spider-browser. Browsers run in Spider's cloud with smart retry, browser switching, anti-bot protection, proxy rotation, and automatic stealth escalation. Supports CDP and BiDi protocols.

Tool	Description
spider_browser_open	Open a remote browser session. Returns a session_id.
spider_browser_navigate	Navigate to a URL and wait for load.
spider_browser_click	Click an element by CSS selector.
spider_browser_fill	Fill a form field with text.
spider_browser_screenshot	Take a screenshot of the current page. Returns base64 PNG.
spider_browser_content	Get page HTML or visible text.
spider_browser_evaluate	Execute JavaScript in the page context.
spider_browser_wait_for	Wait for an element, navigation, or network idle.
spider_browser_close	Close the session and stop billing.

Browser sessions auto-close after 5 minutes of inactivity. Always call spider_browser_close when done.

Examples

Research and RAG

"Crawl the React documentation and summarize the hooks API"

Uses spider_crawl to fetch 50+ pages in seconds and return clean markdown ready for context.

spider_crawl: {
  url: "https://react.dev/reference/react",
  limit: 50,
  return_format: "markdown",
  filter_output_main_only: true
}

Structured Data Extraction

"Get all product names and prices from this e-commerce page"

Uses spider_ai_scrape to extract structured JSON with zero CSS selectors.

spider_ai_scrape: {
  url: "https://example-store.com/products",
  prompt: "Extract every product name, price, and availability status as JSON"
}

Multi-Step Browser Automation

"Log into the dashboard, go to reports, and screenshot the monthly summary"

Uses spider_browser_* tools to drive a remote browser with full anti-bot protection.

1. spider_browser_open: { browser: "auto" }
2. spider_browser_navigate: { url: "https://app.example.com/login" }
3. spider_browser_fill: { selector: "input[name='email']", value: "[email protected]" }
4. spider_browser_fill: { selector: "input[name='password']", value: "..." }
5. spider_browser_click: { selector: "button[type='submit']" }
6. spider_browser_wait_for: { navigation: true }
7. spider_browser_navigate: { url: "https://app.example.com/reports/monthly" }
8. spider_browser_screenshot: {}
9. spider_browser_close: {}

Competitive Intelligence

"Search for recent AI startup funding rounds and get the details"

Uses spider_search with time filtering, then spider_scrape for details.

spider_search: {
  search: "AI startup Series A funding 2025",
  num: 10,
  fetch_page_content: true,
  return_format: "markdown",
  tbs: "qdr:m"
}

API Reference

All tools map directly to the Spider API. Core tools accept the same parameters as their API counterparts:

• Crawl/Scrape: url, return_format, request, readability, root_selector, proxy_enabled, cache, and many more
• Search: search, num, fetch_page_content, country, language, tbs
• AI tools: Add a prompt parameter describing what you want in natural language
• Browser tools: Use session_id from spider_browser_open for all operations

Full parameter reference: spider.cloud/docs/api

Links

• Website
• Documentation
• API Reference
• Get API Key
• Credit Pricing
• AI Subscription
• Discord
• Support

License

MIT