Internet

Python 3.10+ MCP License: MIT Multi-Platform LM Studio

Give your local LLMs the power of the internet — with real data extraction, not just links! 🚀

An open-source Model Context Protocol (MCP) server providing 25 internet tools for local Large Language Models. The v3.0 upgrade adds intelligent search with auto-extraction, multi-source aggregation, result caching, and structured output — making your AI return actual data instead of raw links.

Seamlessly deployable across Windows, macOS, and Linux. No API keys required!

---

🆕 What's New in v3.0

Feature	Before (v2.0)	After (v3.0)
Search Results	Links + snippets only	Full data extraction from pages
Tool Calls Needed	5-10+ per query	1-2 per query
Output Format	Plain text	Markdown tables, bullet points
Caching	None	SQLite with TTL (30min-24h)
Rate Limiting	None	Per-domain throttling
Multi-Source	Manual per source	Auto-aggregation from top results
Content Extraction	Generic text only	Domain-aware (Wikipedia, IMDb, LinkedIn)

New Tools

• smart_search — 🌟 Searches + scrapes + extracts + formats in one call
• deep_search — Like smart_search but examines more sources
• clear_cache — Manage the result cache

---

✨ Key Features

• 🧠 Smart Search: One tool call returns extracted data — movie lists, profiles, technical docs — not just links.
• 🔄 Auto-Extraction: Automatically scrapes top results and extracts structured data (JSON-LD, tables, meta tags).
• 📊 Structured Output: Results come as markdown tables, bullet points, and organized sections.
• ⚡ Cached Results: Repeated queries are served from SQLite cache in milliseconds.
• 🛡️ Rate Limiting: Per-domain request throttling prevents IP bans.
• 🔀 UA Rotation: Pool of 7 browser user-agents for stealth.
• 🌐 Domain-Aware: Special extractors for Wikipedia, IMDb, LinkedIn, and more.
• 🔒 Zero Config: No API keys, no external subscriptions, no setup hassle.

---

🛠️ Available Tools (25 Total)

Category	Tools
🌟 Smart Search	smart_search, deep_search
🔍 Basic Search	search_web, quick_lookup, search_site
📄 Reading	read_webpage, read_pdf
📰 News/Social	get_news, search_reddit, search_twitter
🎬 YouTube	search_youtube, get_video_info
💻 GitHub	search_github, get_repo_info
🌤️ Info / Utils	get_weather, get_current_time, translate_text, calculate
🔗 URLs/IP	shorten_url, get_my_ip, geolocate_ip
📱 Generation	generate_qr, generate_wifi_qr, send_email
🗄️ Cache	clear_cache

---

🚀 Quick Start

Prerequisites

• Python 3.10+
• uv (recommended) or pip

Installation

# 1. Clone the repository
git clone https://github.com/Sonesh-2202/mcp-internet.git
cd mcp-internet

# 2. Install dependencies
uv sync

LM Studio Configuration

Add to your LM Studio mcp.json:

{
  "mcpServers": {
    "mcp-internet": {
      "command": "uv",
      "args": ["run", "mcp-internet"],
      "cwd": "/absolute/path/to/mcp-internet"
    }
  }
}

(Replace /absolute/path/to/mcp-internet with your actual project path)

📖 Detailed OS Guides

For complete, step-by-step instructions for Windows, macOS, and Linux, see the 👉 Multi-Platform Usage Guide.

---

💡 How It Works: smart_search vs search_web

Before (v2.0): Multiple tool calls, manual scraping

User: "What are the upcoming Bollywood movies in 2026?"

Tool call 1: search_web("upcoming Bollywood movies 2026")
→ Returns 10 links with snippets

Tool call 2: read_webpage("https://www.imdb.com/...")
→ Returns raw text

Tool call 3: read_webpage("https://www.wikipedia.org/...")
→ Returns raw text

AI must manually piece together the answer from raw text.

After (v3.0): One tool call, structured data

User: "What are the upcoming Bollywood movies in 2026?"

Tool call 1: smart_search("upcoming Bollywood movies 2026")
→ Returns:
  - Extracted movie list with titles, dates, genres
  - Markdown tables with ratings
  - Data from multiple sources combined
  - All in one response, ready to present

---

🏗️ Architecture

mcp-internet/
├── pyproject.toml              # Dependencies & Config
├── README.md                   # This file
├── MULTI_PLATFORM_GUIDE.md     # OS-specific setup guide
└── src/mcp_internet/
    ├── server.py               # Entry point — 25 MCP tools
    ├── tools/
    │   ├── smart_search.py     # 🌟 Intelligent search + extraction
    │   ├── search.py           # Basic DuckDuckGo search
    │   ├── webpage.py          # Page reader with structured extraction
    │   ├── news.py             # News headlines
    │   ├── youtube.py          # YouTube search & video info
    │   ├── github.py           # GitHub repos & users
    │   ├── reddit.py           # Reddit search
    │   ├── twitter.py          # Twitter/X via Nitter
    │   └── ...                 # weather, time, translate, math, etc.
    └── utils/
        ├── http_client.py      # Async HTTP with UA rotation & rate limiting
        ├── cache.py            # SQLite TTL cache
        └── extractors.py       # Domain-specific data extractors

Data Flow

User Query → smart_search
    ├── 1. Check cache (return if fresh)
    ├── 2. Optimize query & classify (person/movie/news/tech/general)
    ├── 3. Search DuckDuckGo (HTML scraping + ddgs fallback)
    ├── 4. Prioritize results by domain authority
    ├── 5. Scrape top 3 pages in parallel (asyncio.gather)
    ├── 6. Apply domain-specific extractors
    │       ├── Wikipedia → infobox, summary, key facts
    │       ├── IMDb → titles, ratings, cast, genres
    │       ├── LinkedIn → name, role, skills, experience
    │       └── Generic → JSON-LD, OpenGraph, tables
    ├── 7. Aggregate & format as structured markdown
    ├── 8. Cache result
    └── 9. Return comprehensive response

---

🔧 Supported Data Sources

Source	What's Extracted	Extractor Type
Wikipedia	Infobox, summary, key facts	Domain-specific
IMDb	Movie/show details, ratings, cast, genres	Domain-specific (JSON-LD)
LinkedIn	Name, role, skills, experience, education	Domain-specific (Googlebot UA)
GitHub	Repos, stars, forks, languages, topics	API-based
Reddit	Posts, scores, comments, subreddit info	JSON API
YouTube	Video titles, channels, views, duration	HTML parsing
Any website	JSON-LD, OpenGraph, HTML tables, main text	Generic fallback

---

⚡ Performance

Metric	Value
Cache hit	< 50ms
Simple search	1-3 seconds
Smart search (3 sources)	3-8 seconds
Deep search (5+ sources)	5-15 seconds
Rate limiting	2 req/sec per domain
UA rotation	7 browser user-agents

---

🧪 Testing

# Run the integration test suite
uv run python test_v3.py

Recommended LM Studio Test Queries:

1. "What are the upcoming Bollywood movies in 2026?" — Should return movie lists
2. "Search the web for Sundar Pichai" — Should return profile data
3. "Latest albums by Taylor Swift" — Should return music data
4. "Python asyncio tutorial" — Should return technical content

---

📄 License & Contributing

This project is licensed under the MIT License - See LICENSE for details.

Contributions, bug reports, and feature requests are highly welcome!

---

Made with ❤️ for the Local Developer & LLM community