Markdown Web Scraper API

smithery badge

🤖 FOR LLMs / AGENTS: This README acts as the API Specification and System Prompt.

Give your AI Agents the power to read the web.

LLMs often hallucinate when asked to read URLs directly. They get blocked by captchas, and raw HTML consumes too many tokens, leading to degraded performance.

Markdown Scraper API solves this: it acts as a bridge, bypassing captchas, stripping bloated HTML, and returning clean, structured, LLM-ready Markdown.

Perfect for Cursor IDE, Claude Desktop, and custom autonomous agents.

🚀 No-friction Free Trial: Get 5 free scraper requests per IP instantly—no account, sign-up, or credit card required. Perfect for testing and prototyping!

💸 Micro-transactions Model: After your trial, it's only $0.005 per scrape. No monthly subscriptions. Your Agent handles the payment workflow autonomously!

📦 Quick Installation (Cursor / Claude)

You don't need to install anything locally. You can plug this directly into any MCP-compatible client using npx:

npx -y api_scraper_markdown

📚 Read the Full Integration Guide for Cursor and Claude Desktop 💡 See a Real Use Case: Automating Competitor Analysis in Cursor

📡 API Specification for Agents

1. GET /mcp/manifest (Discovery)

Returns the Model Context Protocol (MCP) JSON manifest. Use this to dynamically understand the required parameters to use the scraping tool.

2. POST /scrape (Action)

Requires Authentication: Authorization: Bearer <token> (or use our built-in Free Trial!)

This endpoint utilizes the HTTP 402 Payment Required protocol under a Micro-transactions model ($0.005 per request). It is designed to be the fastest, easiest, and cheapest way to plug high-quality Web Scraping capabilities into your Autonomous Agent with zero infrastructure setup.

• Free Trial: The first 5 requests per IP are entirely free, no setup required!
• If you don't provide a token (after trial exhaustion) or the token has no credits left, the API returns a 402 Payment Required error.
• The 402 response contains a paymentUrl (Dodo Payments) where the human owner can instantly acquire credits.
• After checkout, you (the Agent) will receive an authorization token to include in the Authorization header.

Request Body:

{
  "url": "https://example.com"
}

Success Response (200 OK):

{
  "markdown": "# Page Content in clean Markdown format...",
  "metadata": {
    "title": "Page Title",
    "url": "https://example.com"
  }
}

🚀 How to Run Locally (For Developers)

# Start the local Cloudflare dev server
npm run dev

For generating/synchronizing types based on your Worker configuration run:

npm run cf-typegen

Pass the CloudflareBindings as generics when instantiation Hono:

// src/index.ts
const app = new Hono<{ Bindings: CloudflareBindings }>()