Webintel

PyPI version Python License: MIT MCP

Web Intelligence for AI Agents — give Claude, Cursor, Windsurf, Cline, and any MCP-compatible client the power to read, search, extract, and monitor the web.

No API keys. No cloud lock-in. Self-host for free with unlimited usage, or upgrade to the managed SaaS endpoint when you need it.

---

What is WebIntel?

WebIntel is an open-source MCP (Model Context Protocol) server that turns any AI agent into a web-research assistant. Install it once, and your agent can:

• Scrape any public web page and return clean, readable content
• Search the web via DuckDuckGo without signing up for anything
• Extract SEO metadata, links, tables, JSON-LD, contacts, and structured data
• Monitor pages for changes and get diff summaries
• Batch-process dozens of URLs concurrently

All local by default. Your data never leaves your machine in self-hosted mode.

---

Quick Start

1. Install

pip install webintel-mcp

2. Add to Claude Desktop

Open ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) and add:

{
  "mcpServers": {
    "webintel": {
      "command": "webintel-mcp"
    }
  }
}

3. Add to Cursor / Windsurf / Cline / Any MCP Client

{
  "mcpServers": {
    "webintel": {
      "command": "webintel-mcp"
    }
  }
}

Restart your AI client. Done.

---

Tools

Tool	What it does	Tier
scrape_page	Clean article extraction (removes ads, nav, footers, sidebars)	Free
web_search	DuckDuckGo web search, no API key required	Free
extract_metadata	SEO title/description, Open Graph, Twitter Cards, canonical URL	Free
extract_links	All links categorized as internal, external, or social media	Free
extract_structured	JSON-LD, HTML tables, lists, CSS-selector data	Free
extract_contacts	Emails, phone numbers, and social media profiles	Free
batch_scrape	Concurrent multi-page scraping	Pro
monitor_page	Track page changes and return diffs	Pro

---

Usage Examples

Just ask your AI agent:

• "Scrape https://example.com and summarize it."
• "Search the web for 'best Python MCP servers' and list the top 5."
• "Extract all emails and LinkedIn profiles from https://somecompany.com."
• "What SEO metadata does https://mysite.com have?"
• "Monitor https:// competitor.com/pricing for changes every day."
• "Batch-scrape these 20 product pages and compare prices."

---

Self-Hosted vs. SaaS

	Self-Hosted	SaaS (Coming Soon)
Price	Free	Pro $9/mo · Team $29/mo
API keys	None	One API key
Data privacy	Everything stays local	Encrypted, no training
Batch/monitor tools	Free (set WEBINTEL_PRO=true)	Included
Rate limits	None	5K req/day (Pro), 20K req/day (Team)
Support	Community	Priority + Slack

Self-hosted

pip install webintel-mcp
WEBINTEL_PRO=true webintel-mcp

SaaS endpoint

{
  "mcpServers": {
    "webintel": {
      "command": "webintel-mcp",
      "args": ["--endpoint", "https://api.webintel-mcp.com"],
      "env": {
        "WEBINTEL_API_KEY": "your-api-key"
      }
    }
  }
}

---

Docker

docker build -t webintel-mcp .
docker run -p 8080:8080 webintel-mcp

---

Why WebIntel?

• Zero config — pip install and go. No API keys needed for self-hosted.
• All-in-one — 8 web tools in a single MCP server. Stop installing one server per task.
• Clean output — Smart HTML parsing removes noise and returns readable Markdown-style text.
• Private — Self-hosted mode never sends your browsing data to third parties.
• Fast — Async HTTP and concurrent batch operations.
• Open source — MIT licensed. Fork it, embed it, sell it.

---

Comparison

Feature	WebIntel	Generic scraper MCPs
No API key for search	✅ DuckDuckGo	❌ Often needs Google/Bing key
Metadata + links + contacts	✅ Built-in	❌ Usually separate tools
Batch scraping	✅ Included	❌ Paid or missing
Page monitoring	✅ Included	❌ Paid or missing
Self-hosted free tier	✅ Unlimited	⚠️ Varies

---

Tech Stack

• FastMCP — MCP server framework
• httpx — Async HTTP client
• BeautifulSoup4 + lxml — HTML parsing

---

Development

git clone https://github.com/1036007003-wq/webintel-mcp.git
cd webintel-mcp
pip install -e ".[dev]"
pytest

---

Roadmap

• SaaS hosted endpoint + usage dashboard
• Headless browser / JavaScript rendering for SPAs
• PDF extraction tool
• Screenshot capture tool
• XPath support for extract_structured
• Webhook notifications for monitor_page
• XPack MCP Marketplace listing

---

License

MIT — use it, fork it, sell it, just keep the copyright notice.

---

Support

If this project saves you time, consider sponsoring on GitHub.