Web Scraper Stack

A complete toolkit for web scraping with MCP (Model Context Protocol) support. This project provides a FastAPI REST API, an MCP server, a LangGraph agent, and a Streamlit chat interface—all working together to scrape, search, and extract web content.

📦 Project Structure

This monorepo contains four interconnected components:

File	Purpose
client.py	MCP client that proxies requests to a remote web scraper API
server.py	FastAPI + MCP server that exposes scraping tools as REST endpoints
scraper_demo.py	LangGraph agent demo using the MCP tools with Groq LLM
streamlit_GUI.py	Interactive chat interface for the MCP agent

🔧 Available Tools

All components share these 6 core web scraping tools:

1. search_google – Search Google and get structured results
2. scrape_url – Extract clean content (markdown/text/html) from any webpage
3. scrape_structured – Extract specific data using CSS selectors
4. extract_metadata – Get page metadata (title, OG tags, description)
5. extract_links – Extract all links with optional regex filtering
6. take_screenshot – Capture webpage screenshots (base64 encoded)

🏗️ Architecture

┌─────────────────┐ ┌──────────────┐ ┌─────────────────────┐ │ Streamlit GUI │────▶│ MCP Client │────▶│ Remote Scraper API │ │ (Chat Interface)│ │ (client.py) │ │ (agent-scraper-mcp) │ └─────────────────┘ └──────────────┘ └─────────────────────┘ │ │ │ ▼ ▼ ▼ ┌─────────────────┐ ┌──────────────┐ ┌─────────────────────┐ │ LangGraph Demo │────▶│ FastAPI │────▶│ Local MCP Server │ │ (scraper_demo) │ │ + MCP Server │ │ (server.py --mcp) │ └─────────────────┘ └──────────────┘ └─────────────────────┘

🚀 Quick Start

Prerequisites

• Python 3.11+
• uv package manager (install guide)
• Groq API key (for LLM features)

Installation

# Clone the repository
git clone https://github.com/yourusername/web-scraper-mcp-suite.git
cd web-scraper-mcp-suite

# Install dependencies with uv
uv sync

# Create .env file with your API keys
echo "GROQ_API_KEY=your_groq_api_key_here" > .env

Running the Components

1. Start the FastAPI + MCP Server (Recommended)

# Run as REST API server (default)
uv run server.py

# Or run as pure MCP server
uv run server.py --mcp

The REST API will be available at http://localhost:8000 with Swagger docs at /docs.

2. Launch the Streamlit Chat Interface

# In a new terminal
uv run streamlit run streamlit_GUI.py

3. Run the LangGraph Demo

# Test the agent with a search query
uv run scraper_demo.py

---

📡 API Endpoints (from server.py)

FastAPI provides these REST endpoints (when running python server.py):

Method	Endpoint	Description
POST	/search	Search Google
POST	/scrape	Extract webpage content
POST	/scrape-structured	CSS selector extraction
POST	/metadata	Get page metadata
POST	/links	Extract all links
POST	/screenshot	Capture screenshot

---

🔌 MCP Integration

Both client.py and server.py implement MCP servers that can be used with any MCP client (Claude Desktop, Cursor, etc.).

Connect to the MCP server:

{
  "mcpServers": {
    "web-scraper": {
      "command": "uv",
      "args": ["run", "server.py", "--mcp"]
    }
  }
}

---

🧪 Testing the Tools

Using the REST API

curl -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -d '{"query": "latest AI news", "num_results": 5}'

---

Using the MCP Client

from langchain_mcp_adapters.client import MultiServerMCPClient

client = MultiServerMCPClient({
    "web_scraper": {
        "transport": "streamable_http",
        "url": "http://localhost:8000/mcp"
    }
})

tools = await client.get_tools()
result = await tools[0].ainvoke({"query": "Python tutorials"})

---

🎯 Use Cases

• RAG Applications – Scrape documentation and websites for retrieval-augmented generation
• AI Agents – Give your LLM the ability to search and scrape the web
• Data Collection – Automate extraction of structured data from multiple pages
• Monitoring – Take periodic screenshots of important webpages

---

⚠️ Important Notes

1. Remote API Dependency: The current client.py relies on https://agent-scraper-mcp.onrender.com (may be deprecated). For production, either:
   • Run your own instance of server.py locally
   • Update REMOTE_SCRAPER_SERVER in client.py to point to your own endpoint
2. The 60+ Tools List: You may have seen a longer list of tools elsewhere. The actual implemented tools are only the 6 scraping tools listed above. Additional tools would need to be implemented in client.py.
3. Environment Variables: Ensure GROQ_API_KEY is set for LLM features in scraper_demo.py and streamlit_GUI.py.

---

🤝 Contributing

Feel free to submit issues and pull requests. Some ideas for extension:

• Add caching layer for repeated requests
• Implement parallel scraping with asyncio.gather()
• Add more tools (text summarization, translation, etc.)
• Add authentication to the FastAPI server

---

🙏 Acknowledgments

• FastMCP – MCP server framework
• LangChain MCP Adapters – Tool integration
• Groq – Fast LLM inference
• Streamlit – Chat interface framework