ScrapingDog

The complete Model Context Protocol server for ScrapingDog — all 77 API endpoints as AI-ready tools.

Give Claude (or any MCP client) the power to scrape any webpage, search Google, pull Amazon product data, read LinkedIn profiles, fetch YouTube transcripts, track Google Trends, and much more — through one API key.

MIT License Node >= 18 MCP

Built by extracting the entire official documentation (~90 pages at scrapingdog.com/documentation) into machine-readable specs, then verifying endpoints against the live API — including fixing endpoints the official docs get wrong (see Deviations).

---

⚡ Quick start

1. Get an API key — free 200-credit trial, no credit card: api.scrapingdog.com/register

2. Install (pick your client):

Claude Code (recommended: user scope = available in every project)

claude mcp add --scope user scrapingdog -e SCRAPINGDOG_API_KEY=YOUR_KEY -- npx -y github:alessandrobenigni/ScrapingDog-MCP

Claude Desktop

Add to claude_desktop_config.json (Settings → Developer → Edit Config):

{
  "mcpServers": {
    "scrapingdog": {
      "command": "npx",
      "args": ["-y", "github:alessandrobenigni/ScrapingDog-MCP"],
      "env": { "SCRAPINGDOG_API_KEY": "YOUR_KEY" }
    }
  }
}

Cursor / Windsurf / any MCP client

Same JSON shape as above in the client's MCP config (.cursor/mcp.json, ~/.codeium/windsurf/mcp_config.json, …).

From a local clone (fastest startup, easiest to hack on)

git clone https://github.com/alessandrobenigni/ScrapingDog-MCP.git
cd ScrapingDog-MCP
npm install
claude mcp add --scope user scrapingdog -e SCRAPINGDOG_API_KEY=YOUR_KEY -- node "$(pwd)/src/index.js"

3. Use it — just ask your AI naturally:

"Search Google for 'best espresso machine 2026' and summarize the top results" "Scrape this product page and extract the price" "Get the transcript of this YouTube video" "What's trending on Google Trends for 'AI agents' in the US this year?" "Pull the LinkedIn company profile for Anthropic" "How many ScrapingDog credits do I have left?" (→ account tool)

That's it. No proxy management, no CAPTCHAs, no blocked requests.

---

🧰 The 77 tools

Every tool maps 1:1 to a ScrapingDog endpoint, with every documented parameter typed and described. Credit costs are per successful request and are shown in each tool's description.

🌐 Web Scraping

Tool	What it does	Credits
scrape	Scrape any URL → raw HTML. JS rendering (dynamic), premium residential proxies (premium), geotargeting (country), sticky sessions (session_number), Cloudflare/CAPTCHA bypass (stealth_mode), custom header forwarding	1–25
scrape_post	Forward a POST body (forms, POST-only APIs) through the proxy layer	1+
screenshot	Screenshot any page (viewport or full-page, png/jpg/webp) → returned as an actual image	5

🔍 Google Search & AI

Tool	What it does	Credits
google_search	Full SERP as JSON: organic results, ads, AI Overviews, knowledge graph, local pack, top stories, related questions/searches, inline images/videos/shopping, discussions, immersive products, pagination	5
google_ai_mode	Google's AI Mode answers	10
google_ai_overview	Standalone AI Overview extraction	5
google_autocomplete	Query suggestions	—
google_lens	Visual search by image URL	—

🗺️ Google Maps, Trends, News

Tool	What it does
google_maps_search / google_maps_places / google_maps_reviews / google_maps_photos / google_maps_posts	Complete Maps data: place search, place details, paginated reviews, photos, business posts
google_trends / google_trends_autocomplete / google_trends_trending_now	Interest over time, entity lookup, real-time trending topics
google_news_search / google_news_v2	News SERP + the newer topic-token News API

🎓 Google Scholar & Patents

google_scholar · google_scholar_profiles · google_scholar_author · google_scholar_author_citation · google_scholar_cite · google_patents · google_patent_details

Tip: google_patents returns patent_id as patent/US11734097B1/en; pass just the publication number (US11734097B1) to google_patent_details.

🛍️ Google Shopping & verticals

google_shopping · google_immersive_product · google_images · google_videos · google_shorts · google_finance · google_jobs · google_local · google_hotels · google_flights · google_ads_transparency

Multi-step flows are documented in the tool descriptions — e.g. Flights chains departure_token → booking_token; Shopping filters chain via shoprs.

🔎 Other search engines

bing_search · bing_shopping · duckduckgo_search · baidu_search · universal_search (20 credits, engine-agnostic)

📦 Amazon

Tool	Credits
amazon_product (by ASIN)	1 (US) / 5 (other countries)
amazon_search	1 (US) / 5 (other) / +25 premium
amazon_reviews · amazon_offers · amazon_autocomplete	5 / 5 / —

🛒 Walmart, eBay, Flipkart, Myntra

walmart_product · walmart_search · walmart_reviews · walmart_autocomplete · ebay_search · ebay_product · flipkart_search · flipkart_product · myntra_search · myntra_product

Walmart/eBay/Flipkart/Myntra tools take the full target-site URL (locale lives in the domain, e.g. ebay.co.uk).

💼 LinkedIn, Jobs & Local

Tool	What it does	Credits
linkedin_person_profile	Full person profile by public ID	50–100
linkedin_company_profile	Company/school profile	10
linkedin_post	Post data	5
linkedin_jobs_search / linkedin_job_overview	Job search + job details	5
yelp_scraper / indeed_scraper / zillow_scraper	Local businesses, job listings, real estate	4 / 1 / 2

📱 Social & Video

x_profile · x_post · tiktok_profile · tiktok_post · tiktok_ads · youtube_search · youtube_video · youtube_channel · youtube_comments · youtube_transcripts (only 1 credit!)

🤖 AI & Utilities

Tool	What it does	Credits
chatgpt_scraper	Send a prompt to ChatGPT, get the structured conversation back	30
account	Remaining credits, concurrency, plan info — costs nothing, check it anytime	0
webhook	Fire-and-forget async scraping: returns a sid immediately, result POSTs to your dashboard-configured webhook	—

---

🏗️ How it works

specs/*.json  ──►  src/index.js  ──►  77 MCP tools over stdio
(source of truth)   (generic engine)

• specs/*.json — nine category files holding every endpoint: exact URL, method, credit cost, every documented parameter (name/type/required/default/description), quirks, and response structure. Extracted from the full official docs on 2026-07-10.
• src/index.js — a single ~200-line engine. At startup it loads the specs and registers one tool per endpoint. To fix or extend an API, edit a JSON file — zero code changes.
• docs/*.md — distilled human-readable API reference per category (param tables, response shapes, gotchas).
• test/smoke.js — end-to-end test: boots the server over stdio as a real MCP client and makes live API calls.

Built-in robustness

Concern	Handling
Your API key	Injected server-side from SCRAPINGDOG_API_KEY — never exposed as a tool parameter, never visible to the model
Rate limits / transient errors	Automatic retry (3 attempts, backoff) on 429 / 5xx
Slow scrapes	90s client timeout (ScrapingDog's own limit is 60s)
Giant HTML pages	Capped at 400k chars with a clear truncation notice
Screenshots	Returned as real MCP image content (base64 png/jpg/webp), not text
API errors	Mapped to actionable hints (401 → bad key, 403 → plan doesn't include this API, 410 → target returned nothing, …)

---

🧪 Verify your install

SCRAPINGDOG_API_KEY=YOUR_KEY npm test          # macOS/Linux
$env:SCRAPINGDOG_API_KEY="YOUR_KEY"; npm test  # Windows PowerShell

Boots the server, lists all 77 tools, and makes 4 live calls (account, google_search, scrape, youtube_transcripts). Uses ~7 credits.

---

⚠️ Deviations from the official docs

Found and fixed by testing against the live API (2026-07-10):

1. YouTube endpoints — the docs list https://api.scrapingdog.com/youtube for all five YouTube APIs. That URL returns the ScrapingDog web app, not data. The real endpoints (used by this server) are /youtube/search, /youtube/video, /youtube/channel, /youtube/comments, /youtube/transcripts.
2. youtube_video parameter — docs say video_id; the live API rejects it and requires v.
3. Datacenter proxy mode (http://scrapingdog:[email protected]:8081) is a passthrough HTTP proxy, not a REST endpoint — documented in docs/engines-utilities.md but intentionally not exposed as a tool.
4. Where the docs omit a credit cost or response sample (a handful of endpoints), the tool description says so explicitly rather than guessing.

---

💳 Credits, limits & error codes

• Free plan: 200 credits/month. Every tool description states its cost; account is free and shows your balance.
• Cost-saving tip: scrape defaults to JS rendering (5 credits). Pass dynamic: false for static pages → 1 credit.
• Requests time out server-side after 60 seconds; concurrency is limited per plan tier (excess → 429, which this server retries automatically).

Code	Meaning
200	Success
202	Accepted, queued
400	Bad request — check parameters
401	Invalid or missing API key
403	Your plan doesn't include this API
404	Resource not found
410	Target URL returned no content
429	Rate limit — auto-retried

---

🔧 Troubleshooting

Symptom	Fix
Server exits immediately: SCRAPINGDOG_API_KEY ... not set	Pass the key via -e (Claude Code) or the env block (JSON configs)
401 Unauthorized on every call	Key is wrong/expired — check your dashboard
403 Forbidden on one specific tool	That API isn't in your plan tier
npx install is slow on first launch	It's cloning + installing once; subsequent launches use the cache. Or use the local-clone install
Tool returns truncated HTML	Expected on huge pages (400k-char cap) — scrape a more specific URL or use a structured tool instead of raw scrape
Windows: server won't start from a path with spaces	Quote the path in your config: "args": ["C:\\path with spaces\\src\\index.js"]

---

🗺️ Repo map

├── src/index.js        # the server (spec-driven engine)
├── specs/              # machine-readable endpoint specs — the source of truth
│   ├── web-scraping.json
│   ├── google-search.json
│   ├── google-ai-maps-trends.json
│   ├── google-news-scholar-patents.json
│   ├── google-shopping-other.json
│   ├── engines-utilities.json
│   ├── amazon-walmart.json
│   ├── ecommerce.json
│   ├── profiles-jobs-local.json
│   └── social-youtube.json
├── docs/               # human-readable API reference per category
├── test/smoke.js       # live end-to-end test
└── .env.example

🤝 Contributing

ScrapingDog ships new APIs regularly. Adding one is trivial:

1. Add an endpoint object to the matching specs/*.json (or a new spec file) — tool_name, endpoint, params, credits, description, response_summary.
2. Restart the server. The tool exists.
3. Run npm test, open a PR.

📄 License

MIT © Alessandro Benigni

Not affiliated with ScrapingDog — an independent MCP integration. ScrapingDog is a product of its respective owners. Use responsibly and in accordance with the terms of service of the sites you scrape.