Scrapling Extended

PyPI version Python versions License: MIT

An extended Model Context Protocol (MCP) server that builds on top of the official Scrapling MCP to add interactive browser automation capabilities. This server combines the best of both worlds: Scrapling's powerful web scraping engine with Playwright-style browser interaction tools.

Why Extended?

The official Scrapling MCP provides excellent fetch-and-extract capabilities with anti-bot bypass, but lacks interactive browser automation. This extended version adds:

• Browser Interaction: Click, type, hover, press keys, select options
• Navigation: Navigate to URLs, go back in history
• JavaScript Execution: Evaluate arbitrary JS expressions
• Page Snapshots: Get structured page content for further parsing
• Advanced Parsing: CSS selectors, XPath, find by text/regex, find similar elements

All while preserving the official tools: get, fetch, stealthy_fetch, bulk_get, bulk_fetch, bulk_stealthy_fetch, open_session, close_session, list_sessions, and screenshot.

Features

🚀 Official Scrapling Tools (10)

All tools from the official Scrapling MCP with full parameter support:

• get: Fast HTTP requests with browser fingerprint impersonation, TLS fingerprinting, HTTP/3, SSRF protection, retries, cookies, auth
• bulk_get: Concurrent multi-URL HTTP requests
• fetch: Dynamic content fetching with Chromium browser (Playwright)
• bulk_fetch: Concurrent multi-URL dynamic fetching
• stealthy_fetch: Stealth browser with Cloudflare Turnstile/Interstitial bypass
• bulk_stealthy_fetch: Concurrent multi-URL stealthy fetching
• open_session: Create persistent browser session (dynamic or stealthy)
• close_session: Close browser session and free resources
• list_sessions: List all active browser sessions
• screenshot: Capture PNG/JPEG screenshots (returns native ImageContent)

🎮 Interactive Browser Tools (10) - EXTENDED

Tools not available in the official Scrapling MCP:

• browser_navigate: Navigate to URL in existing session
• browser_navigate_back: Go back in browser history
• browser_click: Click elements by CSS selector
• browser_type: Type text into input fields
• browser_press_key: Press keyboard keys (Enter, Tab, Escape, etc.)
• browser_hover: Hover over elements
• browser_select_option: Select dropdown options
• browser_evaluate: Execute JavaScript expressions
• browser_wait: Wait for elements or text to appear
• browser_snapshot: Get structured page snapshot (URL, title, text content)

🔍 Parsing Tools (7) - EXTENDED

Advanced parsing capabilities using Scrapling's Selector API:

• parse_raw_html: Parse raw HTML content directly
• css: Find elements using CSS selectors
• xpath: Find elements using XPath expressions
• find: Find elements by tag name and/or text regex
• find_text: Find elements by exact text match
• find_regex: Find elements by regex pattern
• similar: Find structurally similar elements using Scrapling's intelligent algorithms

Installation

From PyPI (recommended)

pip install scrapling-mcp

From source

git clone https://github.com/iscodev0/scrapling-mcp.git
cd scrapling-mcp
pip install -e .

Browser dependencies

After installation, install browser dependencies:

scrapling install

Usage

Stdio Transport (for Claude Desktop, Cursor, etc.)

Add to your MCP client configuration:

{
  "mcpServers": {
    "scrapling-mcp": {
      "command": "scrapling-mcp"
    }
  }
}

HTTP Transport (for remote clients)

Start the server:

scrapling-mcp --http --port 4891

Configure your MCP client:

{
  "mcpServers": {
    "scrapling-mcp": {
      "url": "http://localhost:4891/mcp"
    }
  }
}

Examples

Basic Scraping

# Fetch a page with HTTP request (returns content directly)
result = get(
    url="https://example.com",
    impersonate="chrome",
    extraction_type="markdown",  # or "html" or "text"
    css_selector=".article-content",  # optional: extract only specific elements
    main_content_only=True  # optional: extract only main content
)
# result.content contains the extracted markdown/html/text

Anti-Bot Bypass

# Fetch Cloudflare-protected site with stealthy browser
result = stealthy_fetch(
    url="https://protected-site.com",
    solve_cloudflare=True,
    headless=True,
    extraction_type="markdown"
)
# result.content contains the extracted content

Interactive Browser Automation

# Open browser session
session = open_session(session_type="dynamic", headless=True)
session_id = session.session_id

# Navigate and interact
browser_navigate(url="https://example.com/login", session_id=session_id)
browser_type(selector="#username", text="[email protected]", session_id=session_id)
browser_type(selector="#password", text="secret", session_id=session_id)
browser_click(selector="button[type='submit']", session_id=session_id)

# Execute JavaScript
result = browser_evaluate(expression="document.title", session_id=session_id)

# Take screenshot (returns native ImageContent)
screenshot(url="https://example.com/dashboard", session_id=session_id)

# Close session
close_session(session_id=session_id)

Bulk Operations

# Fetch multiple URLs concurrently
results = bulk_get(
    urls=[
        "https://example.com/page1",
        "https://example.com/page2",
        "https://example.com/page3"
    ],
    impersonate="chrome",
    extraction_type="markdown"
)
# results is a list of ResponseModel objects

Advanced Parsing

# Parse HTML and use CSS/XPath selectors
parse_raw_html(html="<html>...</html>")

# Find elements with CSS
elements = css(selector=".product-card", limit=10)

# Find similar elements
similar_items = similar(css_selector=".product-card:first-child")

# Find by text or regex
exact_match = find_text(text="Add to Cart", tag="button")
regex_match = find_regex(pattern=r"\$\d+\.\d{2}")

Test Results

All tools have been tested and verified to work correctly:

✅ Official Scrapling Tools (10/10)

• get - HTTP requests with CSS selector extraction and markdown output
• bulk_get - Concurrent multi-URL fetching
• fetch - Dynamic browser fetching
• bulk_fetch - Concurrent dynamic fetching
• stealthy_fetch - Anti-bot bypass fetching
• bulk_stealthy_fetch - Concurrent stealthy fetching
• open_session - Create persistent browser sessions
• close_session - Close browser sessions
• list_sessions - List active sessions
• screenshot - Capture screenshots (returns native ImageContent)

✅ Interactive Browser Tools (9/10)

• browser_navigate - Navigate to URLs ✅
• browser_click - Click elements ✅
• browser_type - Type text into inputs ✅
• browser_press_key - Press keyboard keys ✅
• browser_hover - Hover over elements ✅
• browser_select_option - Select dropdown options ✅
• browser_evaluate - Execute JavaScript ✅
• browser_wait - Wait for specified time ✅
• browser_snapshot - Capture page state (URL, title, content) ✅
• browser_navigate_back - Go back in history ⚠️ (edge case: page may close during navigation)

✅ Cloudflare Bypass Tools (2/2)

• open_session_with_bypass - Create stealthy session with Cloudflare solver ✅
• close_session_with_bypass - Close bypass session ✅

✅ Parsing Tools (7/7)

• parse_raw_html - Parse HTML content ✅
• css - CSS selector queries ✅
• xpath - XPath queries ✅
• find - Find by tag and regex ✅
• find_text - Find by exact text ✅
• find_regex - Find by regex pattern ✅
• similar - Find similar elements ✅

Overall: 28/29 tools working correctly (97%)

Cloudflare Bypass

The open_session_with_bypass tool provides full Cloudflare Turnstile bypass for interactive sessions:

# Create a session with Cloudflare bypass
session = open_session_with_bypass(
    session_id="my_session",
    headless=True,
    solve_cloudflare=True
)

# Navigate to Cloudflare-protected sites
browser_navigate(
    session_id="my_session",
    url="https://protected-site.com"
)
# Cloudflare challenge is automatically solved

# Interact with the page normally
browser_click(session_id="my_session", selector=".button")
browser_type(session_id="my_session", selector="#search", text="query")

# Close when done
close_session_with_bypass(session_id="my_session")

Features:

• Automatic Cloudflare Turnstile challenge detection and solving
• Supports non-interactive and interactive challenge types
• Canvas noise injection for fingerprint protection
• WebRTC blocking to prevent IP leaks
• WebGL support for modern sites

Note: For simple one-time fetches without interaction, use stealthy_fetch(solve_cloudflare=True) instead.

Architecture

The server combines two powerful approaches:

1. Playwright MCP Architecture: Interactive browser automation with persistent sessions, navigation, clicking, typing, screenshots, and JavaScript evaluation
2. Scrapling Engine: Anti-bot bypass, CSS pre-filtering, adaptive element tracking, and intelligent similarity algorithms

This combination provides both the interactivity of a full browser automation tool and the precision of a web scraping framework.

Configuration

CLI Options

scrapling-mcp --help

Options:
  --http          Use Streamable HTTP transport instead of stdio
  --host HOST     Host to bind to when using HTTP (default: 0.0.0.0)
  --port PORT     Port to listen on when using HTTP (default: 4891)

Environment Variables

The server respects Scrapling's environment variables for proxy configuration, browser settings, and more. See Scrapling documentation for details.

Development

Setup

git clone https://github.com/iscodev0/scrapling-mcp.git
cd scrapling-mcp
pip install -e ".[dev]"

Code Quality

# Format code
black src/

# Lint code
ruff check src/

# Type checking
mypy src/

# Run tests
pytest

Comparison with Other MCP Servers

Feature	Scrapling MCP Extended	Playwright MCP	Scrapling Official MCP
HTTP fetching	✅	❌	✅
Dynamic browser	✅	✅	✅
Anti-bot bypass	✅	❌	✅
CSS pre-filtering	✅	❌	✅
Browser interaction	✅	✅	❌
Screenshots (native)	✅	✅	✅
JavaScript evaluation	✅	✅	❌
Bulk operations	✅	❌	✅
Adaptive tracking	✅	❌	✅
Prompt injection protection	✅	❌	✅
SSRF protection	✅	❌	✅
Total tools	27	~50	10

Publishing to PyPI

This project uses PyPI Trusted Publishers with GitHub Actions for secure, token-free publishing.

How It Works

1. GitHub Actions Workflow: The .github/workflows/publish.yml workflow runs when a new release is published
2. OIDC Authentication: GitHub Actions uses OpenID Connect (OIDC) to prove its identity to PyPI
3. Trusted Publisher: PyPI verifies that the request comes from the authorized GitHub repository and workflow
4. Automatic Publish: The package is built and uploaded to PyPI without needing API tokens

Setup (One-Time)

1. Configure Pending Publisher on PyPI:

   • Go to https://pypi.org/manage/account/publishing/
   • Fill in the form:
     • PyPI Project Name: scrapling-mcp
     • Owner: iscodev0
     • Repository name: scrapling-mcp
     • Workflow name: publish.yml
     • Environment name: pypi

2. Create GitHub Environment:

   • Go to your repository Settings → Environments
   • Click "New environment"
   • Name it pypi
   • (Optional) Add protection rules like required reviewers

Publishing a New Version

1. Update version in pyproject.toml:

   version = "0.3.0"
   

2. Update CHANGELOG.md with the new version

3. Commit and push:

   git add -A
   git commit -m "chore: bump version to 0.3.0"
   git push origin main
   

4. Create a GitHub Release:

   gh release create v0.3.0 --title "v0.3.0" --notes "Release notes here"
   

   Or use the GitHub UI: https://github.com/iscodev0/scrapling-mcp/releases/new

5. Automatic Publishing: The workflow will automatically:

   • Build the package (sdist and wheel)
   • Publish to PyPI using OIDC authentication
   • The new version will be available at https://pypi.org/project/scrapling-mcp/

Why Trusted Publishing?

• No API tokens: Eliminates the risk of token leakage
• Secure: Uses OIDC for cryptographic proof of identity
• Automated: No manual upload steps
• Auditable: All publishes are tied to specific GitHub releases

For more information, see the PyPI Trusted Publishers documentation.

License

MIT License - see LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Acknowledgments

• Scrapling - The powerful web scraping framework this server is built on
• Playwright MCP - Inspiration for the browser automation architecture
• Model Context Protocol - The protocol that makes this possible