WebScrape

English · Español

---

English

MCP server that lets AI agents search the web and extract clean Markdown content — no ads, no clutter, just the text your LLM needs.

What's New in v2.0.0

New features:

• JS rendering — Render JavaScript-heavy sites with Playwright (Chromium headless). Auto-fallback when httpx gets 403 or empty content
• Structured data extraction — Extract JSON-LD, Open Graph, Twitter Cards, meta tags, canonical URLs, and hreflang links with extract_schema=True
• Screenshots — New webscrape_screenshot tool captures page screenshots with configurable viewport, full-page mode, and PNG/JPEG format
• Multi-engine search — DuckDuckGo primary, automatic fallback to Google and Bing if DDGS is unavailable
• Smart truncation — Content is truncated at paragraph/sentence boundaries instead of mid-word

Improvements:

• Enhanced cache: 500 entries with 15-minute TTL (was 200, no expiry)
• Better error handling with specific messages for 403, 404, 429, timeouts
• Updated Dockerfile with Chromium dependencies for Playwright

Tools

Tool	Description
webscrape_fetch_url	Fetch a single URL and return clean Markdown. Supports use_readability, js_render, extract_schema, and auto-detects PDFs
webscrape_batch_fetch	Fetch up to 5 URLs in parallel. Supports PDF auto-detection, JS rendering, and structured data
webscrape_search	Search the web (DuckDuckGo → Google → Bing fallback) and scrape results into Markdown
webscrape_screenshot	Capture a screenshot of any web page with headless Chromium. Supports PNG/JPEG, viewport sizing, and full-page capture

Features

• PDF support: URLs ending in .pdf or with application/pdf content-type are auto-detected and text is extracted page by page
• Readability mode: Pass use_readability=True to webscrape_fetch_url for cleaner article extraction using Mozilla Readability (removes nav, sidebars, ads, comments)
• JS rendering: Pass js_render=True to render JavaScript-heavy sites with Playwright (headless Chromium). Auto-fallback when httpx gets 403 or empty content
• Structured data extraction: Pass extract_schema=True to extract JSON-LD, Open Graph, Twitter Cards, meta tags, canonical URLs, and hreflang links
• Multi-engine search: DuckDuckGo primary, automatic fallback to Google and Bing if DDGS is unavailable
• Screenshots: Capture page screenshots with configurable viewport, full-page mode, and PNG/JPEG format
• Built-in cache: 500-entry cache with TTL-based eviction (15 min) for repeated URLs
• Batch fetching: Up to 5 URLs in parallel
• Smart truncation: Content is truncated at paragraph/sentence boundaries instead of mid-word

How to use

Option 1 — MCPize (recommended)

1. Go to https://mcpize.com/marketplace
2. Search Web Scrape and click Start Free
3. You'll get an API key
4. Configure in your AI client:

{
  "mcpServers": {
    "webscrape": {
      "url": "https://webscrape.mcpize.run",
      "headers": {
        "Authorization": "Bearer your-api-key"
      }
    }
  }
}

Option 2 — Render (dev)

{
  "mcpServers": {
    "webscrape": {
      "url": "https://webscrape-mcp.onrender.com"
    }
  }
}

Option 3 — Local

git clone https://github.com/carrasquelalex1/webscrape-mcp.git
cd webscrape-mcp
pip install -r requirements.txt
playwright install chromium
python webscrape_mcp.py

Official Registry

io.github.carrasquelalex1/webscrape-mcp

Dependencies

mcp, httpx, beautifulsoup4, markdownify, pydantic, ddgs, readability-lxml, PyMuPDF, playwright

License

MIT

---

Español

Servidor MCP que permite a agentes de IA buscar en la web y extraer contenido limpio en Markdown — sin anuncios, sin navegación, solo el texto que tu LLM necesita.

Novedades en v2.0.0

Nuevas características:

• Rendering JS — Renderiza sitios con JavaScript usando Playwright (Chromium headless). Fallback automático cuando httpx recibe 403 o contenido vacío
• Extracción de datos estructurados — Extrae JSON-LD, Open Graph, Twitter Cards, meta tags, URLs canónicas y links hreflang con extract_schema=True
• Capturas de pantalla — Nueva herramienta webscrape_screenshot que captura screenshots con viewport configurable, modo full-page y formato PNG/JPEG
• Búsqueda multi-motor — DuckDuckGo primario, fallback automático a Google y Bing si DDGS no está disponible
• Truncado inteligente — El contenido se trunca en límites de párrafo/oración en vez de cortar palabras a la mitad

Mejoras:

• Caché mejorada: 500 entradas con TTL de 15 minutos (antes 200, sin expiración)
• Mejor manejo de errores con mensajes específicos para 403, 404, 429, timeouts
• Dockerfile actualizado con dependencias de Chromium para Playwright

Tools

Tool	Descripción
webscrape_fetch_url	Obtiene una URL y la convierte a Markdown limpio. Soporta use_readability, js_render, extract_schema, y detecta PDFs automáticamente
webscrape_batch_fetch	Obtiene hasta 5 URLs en paralelo. Soporta detección de PDFs, rendering JS, y datos estructurados
webscrape_search	Busca en la web (DuckDuckGo → Google → Bing como fallback) y extrae los resultados a Markdown
webscrape_screenshot	Captura una captura de pantalla de cualquier página web con Chromium headless. Soporta PNG/JPEG, tamaño de viewport, y captura completa

Características

• Soporte PDF: URLs que terminan en .pdf o con content-type application/pdf se detectan automáticamente y se extrae el texto página por página
• Modo Readability: Usá use_readability=True en webscrape_fetch_url para extraer artículos de forma más limpia (elimina navegación, barras laterales, anuncios, comentarios)
• Rendering JS: Usá js_render=True para renderizar sitios con JavaScript usando Playwright (Chromium headless). Fallback automático cuando httpx recibe 403 o contenido vacío
• Extracción de datos estructurados: Usá extract_schema=True para extraer JSON-LD, Open Graph, Twitter Cards, meta tags, URLs canónicas, y links hreflang
• Búsqueda multi-motor: DuckDuckGo primario, fallback automático a Google y Bing si DDGS no está disponible
• Capturas de pantalla: Captura de páginas con viewport configurable, modo full-page, y formato PNG/JPEG
• Caché integrada: 500 entradas con evicción por TTL (15 min) para URLs repetidas
• Batch fetching: Hasta 5 URLs en paralelo
• Truncado inteligente: El contenido se trunca en límites de párrafo/oración en vez de cortar palabras a la mitad

Cómo usarlo

Opción 1 — MCPize (recomendada)

1. Ve a https://mcpize.com/marketplace
2. Busca Web Scrape y haz clic en Start Free
3. Obtendrás una API key
4. Configura en tu cliente de IA:

{
  "mcpServers": {
    "webscrape": {
      "url": "https://webscrape.mcpize.run",
      "headers": {
        "Authorization": "Bearer tu-api-key"
      }
    }
  }
}

Opción 2 — Render (desarrollo)

{
  "mcpServers": {
    "webscrape": {
      "url": "https://webscrape-mcp.onrender.com"
    }
  }
}

Opción 3 — Local

git clone https://github.com/carrasquelalex1/webscrape-mcp.git
cd webscrape-mcp
pip install -r requirements.txt
playwright install chromium
python webscrape_mcp.py

Registro Oficial

io.github.carrasquelalex1/webscrape-mcp

Dependencias

mcp, httpx, beautifulsoup4, markdownify, pydantic, ddgs, readability-lxml, PyMuPDF, playwright

Licencia

MIT