Docs Scraper

A complete Python template for building Model Context Protocol (MCP) servers with documentation scraping, vector search, and flexible deployment options.

Features

• 🔍 Dual Search Modes: Semantic search (ML-powered) or keyword search (lightweight)
• 🤖 Multi-Provider Embeddings: HuggingFace (local), OpenAI, or Azure OpenAI
• 📚 Web Scraping: Configurable documentation crawler with depth control
• 💾 Vector Storage: ChromaDB with persistent storage
• ⏰ Auto-Updates: Scheduled documentation refresh
• 🔌 Dual Transport: stdio (local MCP) or HTTP/SSE (remote access)
• 🐳 Docker Ready: Containerized deployment with profiles
• ☸️ Kubernetes Ready: Production deployment with autoscaling

Quick Start

Option 1: stdio Mode (Local MCP - Default)

For: VS Code, Claude Desktop integration on your local machine

1. Configure environment:

   cp .env.example .env
   # Edit .env and set your DOCS_URLS
   # Keep TRANSPORT=stdio (or leave it out, stdio is default)
   

2. Start container:

   docker-compose up -d
   

   This starts mcp-server-template container (no ports exposed).

3. Configure MCP client (VS Code or Claude Desktop):

   {
     "mcpServers": {
       "docs-scraper": {
         "command": "docker",
         "args": ["exec", "-i", "mcp-server-template", "python", "-m", "mcp_server_template.server"]
       }
     }
   }
   

Option 2: HTTP Mode (Remote Access)

For: Testing API, remote access, or preparing for Kubernetes deployment

1. Configure environment for HTTP:

   cp .env.example .env
   # Edit .env and set:
   # TRANSPORT=http
   # DOCS_URLS=https://your-docs-site.com/
   

2. Start container with HTTP profile:

   docker-compose --profile http up -d
   

   This starts mcp-server-template-http container on port 3003.

3. Test the server:

   # Check health
   curl http://localhost:3003/health
   
   # MCP clients connect to:
   # http://localhost:3003/sse
   

Switching Between Modes

Stop current mode:

docker-compose down

Start stdio mode:

docker-compose up -d

Start HTTP mode:

docker-compose --profile http up -d

View logs:

# stdio mode
docker logs mcp-server-template -f

# HTTP mode
docker logs mcp-server-template-http -f

Configuration

Key environment variables (see .env.example for full list):

• TRANSPORT: Transport mode - stdio (local MCP) or http (remote/Kubernetes)
• DOCS_URLS: Documentation URLs to scrape (comma-separated for multiple sites)
  • Single: DOCS_URLS=https://docs.example.com/
  • Multiple: DOCS_URLS=https://docs.example.com/,https://api.example.com/docs/,https://guides.example.com/
• SWAGGER_URLS: Swagger/OpenAPI JSON URLs (comma-separated, optional)
• USE_EMBEDDINGS: Enable semantic search (true) or keyword search (false)
• EMBEDDING_PROVIDER: huggingface, openai, or azure
• CRAWL_MAX_DEPTH: Maximum crawl depth (0-3, recommended 2)
• AUTO_UPDATE_ENABLED: Enable scheduled documentation updates (true/false)
• HTTP_PORT: Port for HTTP mode (default: 3000, mapped to 3003 on host)

Transport Modes

stdio Mode (Local MCP)

Best for: Local development, VS Code, Claude Desktop

This server uses stdio transport for direct MCP client communication via stdin/stdout pipes.

VS Code (settings.json):

{
  "mcp.servers": {
    "docs-scraper": {
      "command": "docker",
      "args": ["exec", "-i", "mcp-server-template", "python", "-m", "mcp_server_template.server"]
    }
  }
}

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "docs-scraper": {
      "command": "docker",
      "args": ["exec", "-i", "mcp-server-template", "python", "-m", "mcp_server_template.server"]
    }
  }
}

HTTP Mode (Remote Access)

Best for: Remote access, multi-user, Kubernetes deployments

Set TRANSPORT=http in .env and use the HTTP profile:

docker-compose --profile http up -d

Endpoints:

• GET /health - Health check
• GET /sse - MCP Server-Sent Events endpoint

MCP Client Configuration (for remote HTTP clients):

{
  "mcpServers": {
    "docs-scraper": {
      "url": "http://localhost:3003/sse",
      "transport": "sse"
    }
  }
}

Kubernetes Deployment

See k8s-deployment.yaml for production Kubernetes deployment with:

• Horizontal Pod Autoscaling
• Persistent Volume Claims for ChromaDB
• Ingress configuration
• Health checks and readiness probes

Customization

This is a template - fork and customize for your documentation sources:

1. Update DOCS_URLS in .env
2. Customize scraping in src/mcp_server_template/documentation_scraper.py
3. Add custom tools in src/mcp_server_template/server.py
4. Adjust chunking and search parameters as needed

Requirements

• Python 3.12+
• Docker & Docker Compose (recommended)
• 500MB+ memory for semantic search (50MB for keyword-only)