Web Scraper Server

A Model Context Protocol (MCP) server that provides web scraping capabilities using FastMCP. It allows AI agents to fetch, clean, and extract readable content from web pages, optionally including links and images.

Features

• 🌐 Fetches and parses web pages
• 🧹 Cleans and normalizes extracted text
• 🔗 Optionally extracts links
• 🖼️ Optionally extracts image URLs
• ⚡ Async HTTP requests using httpx
• 🧠 Exposed as an MCP tool (scrape_url)
• 🐳 Dockerized for easy deployment

Project Structure

. 
├── .env.example
├── .gitignore
├── Dockerfile         # Container image definition
├── Makefile           # Common development and deployment commands
├── README.md          # This file
├── docker-compose.yml # Compose file for easy local development
├── main.py            # MCP server and scraping logic
└── requirements.txt   # Dependencies file

main.py

This file contains the MCP server implementation.

Key Components

• FastMCP Server

  • Runs an MCP-compatible HTTP server
  • Exposes tools to AI agents

• scrape_url Tool

  • Fetches and extracts content from a given URL

Tool Signature

scrape_url(
    url: str,
    include_links: bool = True,
    include_images: bool = False,
    clean_text: bool = True
) -> str

Behavior

• Validates URLs before fetching

• Removes <script> and <style> tags

• Extracts readable text

• Optionally appends:

  • All links (<a href>)
  • All images (<img src>)

• Returns a formatted plain-text result

Configuration (Environment Variables)

Variable	Default	Description
HOST_ADDR	0.0.0.0	Bind address
HOST_PORT	9292	Server port
LOG_LEVEL	INFO	Logging level

Dockerfile

The Dockerfile builds a lightweight containerized version of the MCP server.

Highlights

• Based on python:3.12-slim

• Optimized for production use

• Sets helpful defaults:

  • No bytecode
  • Unbuffered output
  • Reduced pip noise

Example Build & Run

docker build -t mcp-web-scraper .
docker run -p 9292:9292 mcp-web-scraper

Makefile

The Makefile provides shortcuts for common tasks.

Common Targets

Target	Description
clean	Remove local virtual environment
dev-compose	Start development environment with Docker Compose
dev-down	Stop development Docker Compose stack
infra	Apply Terraform infrastructure

Example Usage

make dev-compose
make dev-down
make clean

Running Locally (Without Docker)

pip install -r requirements.txt
python main.py

The server will start at:

http://localhost:9292

MCP Integration

This server is designed to be consumed by MCP-compatible clients or agents, enabling them to:

• Read web pages
• Extract structured textual content
• Use scraped data as model context