Scraper Maintenance

A comprehensive Model Context Protocol (MCP) server for automating web scraper maintenance through intelligent browser inspection, selector generation, and code updates.

📁 Project Structure

mcp/
├── src/                    # TypeScript source files
│   ├── server.ts          # Main MCP server implementation
│   ├── browser-manager.ts # Browser automation and management
│   ├── selector-generator.ts # Selector generation and scoring
│   └── types.ts           # Type definitions
├── dist/                   # Compiled JavaScript files
│   ├── server.js          # Main MCP server (executable)
│   ├── browser-manager.js # Browser automation
│   ├── selector-generator.js # Selector intelligence
│   └── types.js           # Type definitions
├── config/                 # Configuration files
│   ├── test-config.json   # Test configuration
│   ├── claude-desktop-config.json # Claude Desktop setup
│   └── *.json             # Various scraper configurations
├── examples/               # Usage examples and documentation
├── docs/                   # Documentation files
├── scripts/                # Build and utility scripts
├── package.json           # Project configuration
└── tsconfig.json          # TypeScript configuration

🚀 Quick Start

1. Install Dependencies

cd mcp
npm install

2. Build the Project

npm run build

3. Run the Server

npm start

🛠️ Available MCP Tools

Configuration Management

• load_scraper_config - Load scraper configuration files
• update_config - Update configurations with new selector mappings

Browser Operations

• initialize_browser - Launch browser (headless/visible mode)
• navigate_to_page - Navigate to target URLs
• take_screenshot - Capture debugging screenshots
• close_browser - Cleanup browser resources

Element Inspection

• inspect_field_manually - Interactive visual element selection
• auto_detect_field - AI-powered automatic element detection
• validate_selectors - Test selector reliability and performance
• generate_selectors - Create multiple selector variations with scoring
• test_extraction - Test data extraction using current selectors

Maintenance & Code Generation

• run_maintenance_check - Comprehensive scraper health analysis
• generate_extractor_code - Multi-language code generation

📖 Usage

For Claude Desktop

Add to your Claude Desktop configuration:

{
  "mcpServers": {
    "scraper-maintenance": {
      "command": "node",
      "args": ["/path/to/mcp/dist/server.js"],
      "env": {
        "NODE_ENV": "production"
      }
    }
  }
}

For Cursor

Add to your Cursor MCP configuration:

{
  "mcpServers": {
    "scraper-maintenance": {
      "command": "node",
      "args": ["/path/to/mcp/dist/server.js"],
      "env": {
        "NODE_ENV": "production"
      }
    }
  }
}

🔧 Development

Build

npm run build

Development Mode

npm run dev

Test

npm test

📚 Documentation

• Setup Guide
• Implementation Guide
• Workflow Examples
• Advanced Usage