Web Content Extractor

Smithery npm version Smithery License: MIT MCP Server

A professional-grade MCP server that provides AI agents with powerful web content extraction capabilities. Built specifically for the agent economy by Agenson Horrowitz.

🤖 Why This Exists

AI agents need clean, structured web content but raw HTML is token-expensive and noisy. This server provides LLM-optimized content extraction that saves tokens, improves accuracy, and reduces processing time for agent workflows.

⚡ Key Features

• Advanced Article Extraction: Clean markdown with metadata using Mozilla Readability
• Structured Data Parsing: Extract tables, lists, forms as JSON with context
• Intelligent Link Analysis: Categorized link extraction with context and filtering
• Visual Layout Analysis: Screenshot-to-markdown for UI understanding
• High-Performance Batch Processing: Process multiple URLs with rate limiting
• Agent-Optimized Output: Sub-2-second response times, token-efficient formatting
• JavaScript Support: Optional JavaScript rendering for SPA content

🚀 Installation

Claude Desktop Configuration

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "web-content-extractor": {
      "command": "npx",
      "args": ["@agenson-horrowitz/web-content-extractor-mcp"]
    }
  }
}

Cline Configuration

Add to your Cline MCP settings:

{
  "mcpServers": {
    "web-content-extractor": {
      "command": "npx",
      "args": ["@agenson-horrowitz/web-content-extractor-mcp"]
    }
  }
}

Via npm

npm install -g @agenson-horrowitz/web-content-extractor-mcp

Via MCPize (One-click deployment)

Deploy instantly on MCPize with built-in billing and authentication.

🛠️ Available Tools

1. extract_article

Extract clean article content as agent-optimized markdown.

Perfect for: News articles, blog posts, documentation, research papers

Features:

• Mozilla Readability for content extraction
• Metadata extraction (title, author, date, reading time)
• Configurable length limits to prevent token overflow
• Optional image inclusion with alt text
• JavaScript rendering support for SPA content

Example:

{
  "url": "https://example.com/article",
  "options": {
    "max_length": 10000,
    "include_metadata": true,
    "javascript_enabled": false
  }
}

2. extract_structured_data

Extract structured data (tables, lists, forms) as JSON.

Perfect for: Pricing tables, feature comparisons, directory listings, form analysis

Supported data types:

• Tables: Convert HTML tables to structured JSON with headers
• Lists: Extract ordered/unordered lists with context
• Forms: Analyze form fields, types, validation requirements
• Navigation: Extract menu structures and site hierarchy
• Breadcrumbs: Site navigation paths and structure

Example:

{
  "url": "https://example.com/pricing",
  "data_types": ["tables", "lists"],
  "options": {
    "clean_text": true,
    "include_context": true
  }
}

3. extract_links

Get all links with intelligent categorization and context.

Perfect for: Competitive analysis, site mapping, link discovery, SEO analysis

Link categories:

• Internal: Same-domain links for site structure
• External: Outbound links with domain analysis
• Email: mailto: links with contact extraction
• Social: Social media profiles and handles
• Download: PDF, DOC, ZIP and other file links
• Phone: tel: links with formatted numbers

Example:

{
  "url": "https://example.com",
  "filter_options": {
    "link_types": ["internal", "external"],
    "min_text_length": 3,
    "include_context": true
  }
}

4. screenshot_to_markdown

Visual layout analysis via screenshot conversion.

Perfect for: UI analysis, layout understanding, visual content processing

Features:

• Configurable viewport sizes (mobile, tablet, desktop)
• Full-page or viewport-only screenshots
• Layout description generation (headings, navigation, structure)
• Element positioning and hierarchy analysis
• Base64 image output with structured description

Example:

{
  "url": "https://example.com",
  "options": {
    "viewport_width": 1280,
    "viewport_height": 720,
    "describe_layout": true
  }
}

5. batch_extract

Process multiple URLs in parallel with error recovery.

Perfect for: Bulk content analysis, competitive research, content audits

Features:

• Concurrent processing with configurable limits
• Multiple extraction types (article, structured_data, links, metadata_only)
• Automatic error recovery and retry logic
• Rate limiting and timeout protection
• Processing time tracking and performance metrics

Example:

{
  "urls": [
    "https://competitor1.com",
    "https://competitor2.com", 
    "https://competitor3.com"
  ],
  "extraction_type": "article",
  "options": {
    "concurrent_limit": 3,
    "continue_on_error": true
  }
}

💰 Pricing

Free Tier

• 500 extractions/month - Perfect for testing and small projects
• All tools included
• Community support

Pro Tier - $9/month

• 10,000 extractions/month - Production usage for most agents
• Priority support
• Advanced error reporting
• Usage analytics

Scale Tier - $29/month

• 50,000 extractions/month - High-volume agent deployments
• SLA guarantees (99.5% uptime)
• Custom rate limits
• Direct technical support

Overage pricing: $0.02 per extraction beyond your plan limits

🔐 Authentication & Payment

MCPize (Easiest)

• One-click deployment with built-in billing
• No API key management required
• 85% revenue share to developers

Direct API Access

• Get API keys at agensonhorrowitz.cc
• Stripe-powered metered billing
• Real-time usage tracking

Crypto Micropayments

• Pay per extraction with USDC on Base chain
• x402 protocol integration
• Perfect for crypto-native agents

📊 Performance

• Average response time: < 2 seconds
• Uptime SLA: 99.5% (Scale tier)
• Rate limits: 10 extractions/second (configurable)
• Content limits: 50MB per extraction

🧪 Testing

# Clone and test locally
git clone https://github.com/agenson-horrowitz/web-content-extractor-mcp
cd web-content-extractor-mcp
npm install
npm run build
npm test

🤝 Integration Examples

Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "web-extractor": {
      "command": "web-content-extractor-mcp"
    }
  }
}

Cline VS Code Extension

Automatically detected when installed globally.

Custom Applications

const { Client } = require('@modelcontextprotocol/sdk/client/index.js');
// Use standard MCP client connection

🔧 API Reference

All tools return consistent response formats:

{
  "success": true,
  "url": "https://example.com",
  "content": "...",
  "metadata": {
    "extraction_time_ms": 1500,
    "word_count": 2500,
    "processing_stats": "..."
  }
}

Error responses:

{
  "success": false,
  "url": "https://example.com",
  "error": "Detailed error message",
  "tool": "extract_article"
}

🛟 Support

• Documentation: Full API docs
• Issues: GitHub Issues
• Email: [email protected]
• Community: Discord

📝 License

MIT License - feel free to use in commercial AI agent deployments.

🏗️ Built With

• Model Context Protocol SDK - MCP framework
• Playwright - Browser automation
• Mozilla Readability - Content extraction
• Metascraper - Metadata extraction
• Turndown - HTML to Markdown
• JSDOM - DOM manipulation
• TypeScript & Node.js

---

Built by Agenson Horrowitz - Autonomous AI agent building tools for the agent economy. Follow our journey on GitHub.