Website Reader

A small production-ready Model Context Protocol (MCP) server built with Python and FastAPI. It exposes Website Reader tools over Streamable HTTP so an AI chat backend can fetch public webpages and receive cleaned, readable text.

What it does

• Runs as a FastAPI app locally with uvicorn over HTTPS
• Deploys to Vercel as a Python serverless app (HTTPS provided by Vercel)
• Exposes MCP at /mcp (Streamable HTTP transport)
• Protects the MCP endpoint with a static API key
• Provides a small content extraction pipeline: raw fetch, Markdown, article extraction, metadata and summary preparation
• Caches successful URL fetches (opt-in via CACHE_ENABLED; in-memory or Redis backend) to avoid duplicate downloads across tools
• Rate-limits outbound URL fetches per client (opt-in via RATE_LIMIT_ENABLED; API key + IP composite key; in-memory or Redis backend)
• Validates DNS-resolved IPs across multiple resolvers and pins the connection to the vetted IP (on by default via SSRF_DNS_VALIDATION_ENABLED) to block SSRF and DNS rebinding

Tools

The server exposes five MCP tools so the AI Chat backend can pick the right extraction layer for the task:

Tool	Best for	Output
fetch_url	Raw/simple fetch for debugging or fallback when you also need HTTP status, final URL and content type	Cleaned page text plus basic title/description from HTML
fetch_markdown	RAG ingestion and LLM context	Clean, LLM-friendly Markdown with headings, paragraphs, links, lists and code blocks; boilerplate removed
extract_article	Summaries, blog posts, news, docs and long-form pages	Main article text and Markdown, plus author, published date, description, site name and language
extract_metadata	Link previews and routing	Title, description, author, published date, site name, language, image and canonical URL (Open Graph, Twitter card, JSON-LD, meta tags)
summarize_article	Preparing an article summary without coupling the server to an LLM	Article text plus a ready-to-use summary_prompt the chat backend passes to its own model

Markdown and article extraction are powered by trafilatura, with a lightweight BeautifulSoup fallback when trafilatura cannot find usable content. All tools return structured error messages instead of crashing on invalid URLs, timeouts, unsupported content types, empty pages or extraction failures.

summarize_article and LLMs

summarize_article deliberately does not call OpenAI or any other model from inside the MCP server. It returns the extracted text together with a summary_prompt string. The AI Chat backend can send summary_prompt to its existing model pipeline to produce the actual summary. This keeps the MCP server provider-agnostic.

Local setup

Requirements: Python 3.11+ and OpenSSL (for local dev certs)

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

Edit .env and set a real value for MCP_API_KEY.

Environment variables

Copy .env.example to .env:

MCP_API_KEY=change-me
APP_ENV=local
REQUEST_TIMEOUT_SECONDS=12
MAX_RESPONSE_CHARS=12000
MAX_HTML_BYTES=2000000
ALLOWED_SCHEMES=https,http

# MCP client host allowlist (see .env.example)
MCP_ALLOWED_HOSTS=
MCP_ALLOWED_ORIGINS=

# URL fetch cache (disabled by default)
CACHE_ENABLED=false
CACHE_BACKEND=memory
CACHE_TTL_SECONDS=300
CACHE_MAX_ENTRIES=100
REDIS_URL=

# Fetch rate limiting (disabled by default)
RATE_LIMIT_ENABLED=false
RATE_LIMIT_BACKEND=memory
RATE_LIMIT_PER_MINUTE=15
RATE_LIMIT_PER_HOUR=120

# DNS-based SSRF protection (enabled by default)
SSRF_DNS_VALIDATION_ENABLED=true
SSRF_DNS_TIMEOUT_SECONDS=2
SSRF_DNS_RESOLVERS=system,1.1.1.1,1.0.0.1,8.8.8.8,8.8.4.4

HOST=0.0.0.0
PORT=8001
DEV_HTTPS=true
SSL_CERTFILE=certs/localhost.pem
SSL_KEYFILE=certs/localhost-key.pem

The real .env file is gitignored and should not be committed.

URL fetch cache

Caching is off by default. Set CACHE_ENABLED=true to cache successful HTTP responses at the fetch layer (shared by all tools).

Variable	Default	Description
CACHE_ENABLED	false	Enable or disable URL fetch caching
CACHE_BACKEND	memory	memory for local/single-instance; redis for shared cache on Vercel
CACHE_TTL_SECONDS	300	How long cached responses are kept
CACHE_MAX_ENTRIES	100	Max in-memory entries (ignored for Redis)
REDIS_URL	(empty)	Required when CACHE_BACKEND=redis (e.g. Upstash)

Only successful fetches are cached; errors and validation failures are not. Tests run with caching disabled.

Fetch rate limiting

Rate limiting is off by default. Set RATE_LIMIT_ENABLED=true to cap how many outbound URL fetches each client can perform in a time window. Limits apply at the fetch layer (shared by all tools) and only count cache misses — cached responses do not consume quota.

Each client is identified by a composite key of hashed API key + client IP (the raw key is never stored in Redis). On Vercel, the client IP is taken from x-vercel-forwarded-for or x-forwarded-for when present.

Variable	Default	Description
RATE_LIMIT_ENABLED	false	Enable or disable fetch rate limiting
RATE_LIMIT_BACKEND	memory	memory for local/single-instance; redis for shared limits on Vercel
RATE_LIMIT_PER_MINUTE	15	Max URL fetches per client per minute (burst protection)
RATE_LIMIT_PER_HOUR	120	Max URL fetches per client per hour (sustained cap)
REDIS_URL	(empty)	Required when RATE_LIMIT_BACKEND=redis (e.g. Upstash; can be shared with the URL cache)

When a client exceeds a limit, tools return a structured error such as Rate limit exceeded. Retry after N seconds. instead of fetching the URL.

DNS-based SSRF protection

Beyond the synchronous checks on URL schemes, blocked hostnames and IP literals, the server performs DNS-level validation before every fetch. This is on by default and defends against Server-Side Request Forgery (SSRF), including DNS-rebinding attacks.

How it works:

1. Multi-resolver resolution. The hostname is resolved concurrently across several independent resolvers — the system resolver plus public resolvers (Cloudflare 1.1.1.1/1.0.0.1 and Google 8.8.8.8/8.8.4.4). The union of all A/AAAA answers is validated, so if any resolver returns a private/internal address the request is rejected.
2. Hardened IP classification. Resolved IPs are rejected if they fall into loopback, private (RFC1918), link-local (incl. the 169.254.169.254 cloud metadata address), carrier-grade NAT, multicast, reserved, benchmarking or test ranges. IPv6-embedded IPv4 (IPv4-mapped, NAT64, 6to4) is unwrapped and checked too, so a private target cannot be smuggled through an IPv6 encoding.
3. Connection pinning (anti-rebinding). The vetted IP is pinned for the actual connection, so the name cannot resolve to a safe IP during validation and to an internal IP a moment later. The original Host header and TLS SNI/certificate verification are preserved against the real hostname.
4. Redirects are validated too. Every redirect hop flows through the same validation and pinning.

Very low per-query timeouts are used and resolvers are queried concurrently, so total added latency is bounded by a single timeout window regardless of how many resolvers are configured.

Variable	Default	Description
SSRF_DNS_VALIDATION_ENABLED	true	Enable DNS resolution + IP validation + connection pinning
SSRF_DNS_TIMEOUT_SECONDS	2	Low per-query timeout; resolvers are queried concurrently
SSRF_DNS_RESOLVERS	system,1.1.1.1,1.0.0.1,8.8.8.8,8.8.4.4	Comma-separated resolvers. system uses /etc/resolv.conf; others are public resolver IPs

When a host resolves to a blocked IP (or cannot be resolved), tools return a structured error instead of fetching the URL.

Create local HTTPS certs

Local development uses self-signed TLS certs. Generate them once:

chmod +x scripts/dev.sh scripts/generate_dev_certs.sh
./scripts/generate_dev_certs.sh

This creates:

certs/localhost.pem
certs/localhost-key.pem

These files are gitignored and are for local dev only.

You do not need to run this manually if you use ./scripts/dev.sh — it auto-generates missing certs on first start.

Optional: trusted local certs with mkcert

If you prefer browser- and client-trusted local certs instead of self-signed ones:

brew install mkcert
mkcert -install
mkdir -p certs
mkcert -cert-file certs/localhost.pem -key-file certs/localhost-key.pem localhost 127.0.0.1

Then use ./scripts/dev.sh as usual.

Run locally

./scripts/dev.sh

This starts uvicorn with reload on:

https://localhost:8001

Useful overrides:

# HTTP instead of HTTPS
DEV_HTTPS=false ./scripts/dev.sh

# Bind only to localhost
HOST=127.0.0.1 ./scripts/dev.sh

Health check

Self-signed certs require -k with curl:

curl -k https://localhost:8001/health

Example response:

{
  "status": "ok",
  "service": "website-reader-mcp"
}

MCP endpoint

The MCP Streamable HTTP endpoint is:

https://localhost:8001/mcp

Authentication is required. Use either header:

Authorization: Bearer <MCP_API_KEY>

or:

X-API-Key: <MCP_API_KEY>

Quick MCP test with curl

Initialize a session (stateless mode):

curl -k -sS -X POST "https://localhost:8001/mcp/" \
  -H "Authorization: Bearer change-me" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "initialize",
    "params": {
      "protocolVersion": "2024-11-05",
      "capabilities": {},
      "clientInfo": {"name": "curl-test", "version": "0.1"}
    }
  }'

List tools:

curl -k -sS -X POST "https://localhost:8001/mcp/" \
  -H "Authorization: Bearer change-me" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0",
    "id": 2,
    "method": "tools/list",
    "params": {}
  }'

Call fetch_url:

curl -k -sS -X POST "https://localhost:8001/mcp/" \
  -H "Authorization: Bearer change-me" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0",
    "id": 3,
    "method": "tools/call",
    "params": {
      "name": "fetch_url",
      "arguments": {"url": "https://example.com"}
    }
  }'

Replace change-me with your configured MCP_API_KEY.

Call fetch_markdown

curl -k -sS -X POST "https://localhost:8001/mcp/" \
  -H "Authorization: Bearer change-me" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0",
    "id": 4,
    "method": "tools/call",
    "params": {
      "name": "fetch_markdown",
      "arguments": {
        "url": "https://example.com/blog/my-article",
        "max_chars": 12000
      }
    }
  }'

Example structured output:

{
  "url": "https://example.com/blog/my-article",
  "final_url": "https://example.com/blog/my-article",
  "title": "My Article",
  "markdown": "# My Article\n\nClean readable content...\n\n- point one\n- point two",
  "content_length": 842,
  "truncated": false,
  "extraction_method": "trafilatura",
  "error": null
}

Call extract_article

curl -k -sS -X POST "https://localhost:8001/mcp/" \
  -H "Authorization: Bearer change-me" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0",
    "id": 5,
    "method": "tools/call",
    "params": {
      "name": "extract_article",
      "arguments": {
        "url": "https://example.com/blog/my-article",
        "max_chars": 12000,
        "include_metadata": true,
        "include_markdown": true
      }
    }
  }'

Example structured output:

{
  "url": "https://example.com/blog/my-article",
  "final_url": "https://example.com/blog/my-article",
  "title": "My Article",
  "author": "Jane Doe",
  "published_date": "2024-05-01T10:00:00Z",
  "description": "Short article description",
  "site_name": "Example",
  "language": "en",
  "text": "Clean readable article text...",
  "markdown": "# My Article\n\nClean readable article text...",
  "content_length": 8452,
  "truncated": false,
  "extraction_method": "trafilatura",
  "error": null
}

If extraction fails, the tool returns a structured error instead of crashing:

{
  "url": "https://example.com/article",
  "final_url": "https://example.com/article",
  "error": "Could not extract readable article content from this page.",
  "text": null,
  "markdown": null,
  "extraction_method": "trafilatura"
}

Call extract_metadata

curl -k -sS -X POST "https://localhost:8001/mcp/" \
  -H "Authorization: Bearer change-me" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0",
    "id": 6,
    "method": "tools/call",
    "params": {
      "name": "extract_metadata",
      "arguments": {"url": "https://example.com/blog/my-article"}
    }
  }'

Example structured output:

{
  "url": "https://example.com/blog/my-article",
  "final_url": "https://example.com/blog/my-article",
  "title": "My Article",
  "description": "Short article description",
  "author": "Jane Doe",
  "published_date": "2024-05-01T10:00:00Z",
  "site_name": "Example",
  "language": "en",
  "image": "https://example.com/images/cover.png",
  "canonical_url": "https://example.com/blog/my-article",
  "error": null
}

Call summarize_article

curl -k -sS -X POST "https://localhost:8001/mcp/" \
  -H "Authorization: Bearer change-me" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0",
    "id": 7,
    "method": "tools/call",
    "params": {
      "name": "summarize_article",
      "arguments": {
        "url": "https://example.com/blog/my-article",
        "max_chars": 12000,
        "max_words": 150
      }
    }
  }'

Example structured output:

{
  "url": "https://example.com/blog/my-article",
  "final_url": "https://example.com/blog/my-article",
  "title": "My Article",
  "author": "Jane Doe",
  "published_date": "2024-05-01T10:00:00Z",
  "description": "Short article description",
  "text": "Clean readable article text...",
  "content_length": 8452,
  "truncated": false,
  "summary_prompt": "Summarize My Article in at most 150 words. Focus on the key points...\n\nArticle content:\nClean readable article text...",
  "extraction_method": "trafilatura",
  "error": null
}

The chat backend passes summary_prompt to its own LLM to generate the final summary.

You can also connect with the MCP Inspector using Streamable HTTP transport, the HTTPS URL above, and the same API key. You may need to accept the self-signed certificate in your client.

Tests

pytest

Vercel deployment

1. Push this repository to GitHub.
2. Import the project in Vercel.
3. Set environment variables in the Vercel dashboard (at minimum MCP_API_KEY; for production, also consider RATE_LIMIT_ENABLED=true, RATE_LIMIT_BACKEND=redis, and REDIS_URL).
4. Deploy.

The included vercel.json routes all requests to app/main.py, which exports the ASGI app object required by @vercel/python. Vercel terminates HTTPS for you in production; the local cert files are not used there.

After deployment, your MCP endpoint will be:

https://<your-project>.vercel.app/mcp

Use the same API key headers as in local development.

Security notes and limitations

• The MCP endpoint is protected by a single static API key. Rotate the key if it is exposed.
• Local HTTPS uses self-signed certificates. Do not reuse them outside local development.
• SSRF protection blocks localhost, common internal hostnames, and private/link-local/multicast IP literals before fetching.
• DNS-based SSRF protection (SSRF_DNS_VALIDATION_ENABLED, on by default): hostnames are resolved across multiple resolvers and rejected if any resolved IP is private/internal (including IPv6-embedded IPv4). The connection is then pinned to the vetted IP to block DNS rebinding, while preserving Host and TLS SNI. See DNS-based SSRF protection.
• MCP host allowlist (MCP_ALLOWED_HOSTS): optional restriction on which Host header values may access /mcp (in addition to the API key). Leave empty to disable. Set * to allow any host, list exact hosts (example.com), subdomain wildcards (*.example.com), or port wildcards for local dev (localhost:*). Optional MCP_ALLOWED_ORIGINS restricts browser Origin headers when set.
• Only http and https URLs are allowed.
• Responses are capped by MAX_HTML_BYTES while downloading and MAX_RESPONSE_CHARS (or max_chars) for returned text.
• Fetch rate limiting (RATE_LIMIT_ENABLED): optional per-client caps on outbound URL fetches (API key hash + IP). Use Redis on Vercel so limits are shared across serverless instances. Cached responses do not count toward the limit.
• No JavaScript rendering: pages that require a browser will not be fully readable.
• No crawling: each tool fetches a single URL per call.

Project structure

app/
  main.py              FastAPI app, health routes, MCP mount
  config.py            Environment settings
  auth.py              API key and MCP host allowlist middleware
  schemas.py           Response models
  cache/               URL fetch cache (noop, memory, Redis backends)
  ratelimit/           Fetch rate limiting (noop, memory, Redis backends)
  ssrf/                DNS SSRF protection (IP rules, multi-resolver, pinning transport)
  tools/
    website_reader.py  MCP tool registration (all five tools)
  services/
    fetcher.py             HTTP fetch + URL validation (SSRF checks)
    host_allowlist.py      MCP_ALLOWED_HOSTS parsing and Host-header matching
    extractor.py           HTML to readable text (BeautifulSoup, used by fetch_url)
    markdown_extractor.py  HTML to Markdown (trafilatura + BeautifulSoup fallback)
    metadata_extractor.py  Metadata (Open Graph, Twitter, JSON-LD, meta tags)
    article_extractor.py   Article extraction and summary prompt preparation
scripts/
  generate_dev_certs.sh  Create local self-signed TLS certs
  dev.sh                 Run uvicorn with HTTPS locally
tests/
  test_fetcher.py
  test_ssrf.py
  test_cache.py
  test_ratelimit.py
  test_extractor.py
  test_extract_article.py
  test_markdown.py
  test_metadata.py
  test_summarize.py

Next steps

Possible follow-ups:

• add logging and request IDs
• add an MCP client inside the existing AI Chat backend
• add tools for search_web and read_url

License

Licensed under the MIT License. See LICENSE.