LLMSTXT Server

A Model Context Protocol (MCP) server that provides access to LLMS.TXT documentation files. This server allows AI agents to fetch and process documentation from various sources.

Features

• Multiple Documentation Sources: Access documentation from React, Next.js, Node.js and more
• HTTP Fetching: Fetch documentation from any HTTPS URL with domain restrictions
• HTML to Markdown: Automatically converts HTML content to clean Markdown
• Environment Configuration: Customize behavior via environment variables
• MCP Protocol: Full Model Context Protocol compliance

Installation

npm install -g @pinkpixel/llmstxt-mcp

Usage

The server starts automatically and provides two tools:

Available Tools

1. list_doc_sources - Lists all configured documentation sources
2. fetch_docs - Fetches documentation from a URL and converts to Markdown

Default Configuration

By default, the server is configured with:

• React documentation: https://react.dev/llms.txt
• Next.js documentation: https://nextjs.org/llms.txt
• Node.js documentation: https://nodejs.org/llms.txt
• LLMSTXT Directory (Cloud): https://directory.llmstxt.cloud/ - Curated directory of companies using llms.txt
• LLMSTXT Site Directory: https://llmstxt.site/ - Comprehensive list with token counts and stats

Discovery Directories

The two directory sources provide access to thousands of websites that have adopted the llms.txt standard:

• directory.llmstxt.cloud - Curated directory with companies like Anthropic, Supabase, Modal, NVIDIA, and many others across AI, developer tools, finance, and products categories
• llmstxt.site - Comprehensive searchable directory with detailed statistics and token counts for each llms.txt file

These directories are invaluable for discovering documentation from the growing ecosystem of companies adopting the llms.txt standard.

Environment Variables

Customize behavior with these environment variables:

• LLMSTXT_CONFIG: Path to custom configuration file (not implemented yet)
• LLMSTXT_FOLLOW_REDIRECTS: Whether to follow HTTP redirects (true/false)
• LLMSTXT_TIMEOUT: Request timeout in seconds (default: 10)
• LLMSTXT_ALLOWED_DOMAINS: Comma-separated list of allowed domains (default: * for all)

Example:

export LLMSTXT_FOLLOW_REDIRECTS=true
export LLMSTXT_TIMEOUT=30
export LLMSTXT_ALLOWED_DOMAINS="react.dev,nextjs.org,nodejs.org"

MCP Client Configuration

You can install globally with npm i -g @pinkpixel/llmstxt-mcp and then it can be ran with "llmstxt-mcp"

Add to your mcp_config.json:

{
  "mcpServers": {
    "llmstxt": {
      "command": "llmstxt-mcp",
      "env": {
        "LLMSTXT_FOLLOW_REDIRECTS": "true",
        "LLMSTXT_TIMEOUT": "30"
      }
    }
  }
}

OR use with npx

{
  "mcpServers": {
    "llmstxt": {
      "command": "npx",
      "args": ["-y", "@pinkpixel/llmstxt-mcp", "llmstxt-mcp"],
      "env": {
        "LLMSTXT_FOLLOW_REDIRECTS": "true",
        "LLMSTXT_TIMEOUT": "30"
      }
    }
  }
}

Testing

Test with MCP Inspector

# Build first
npm run build

# Test with inspector
npm run inspector

# Or directly
npx @modelcontextprotocol/inspector ./build/index.js

Manual Testing

# List available tools
npx @modelcontextprotocol/inspector --cli ./build/index.js --method tools/list

# List doc sources
npx @modelcontextprotocol/inspector --cli ./build/index.js --method tools/call --tool-name list_doc_sources

# Fetch documentation
npx @modelcontextprotocol/inspector --cli ./build/index.js --method tools/call --tool-name fetch_docs --tool-arg url="https://example.com"

Security

• Domain Restrictions: Configure allowed domains via environment variables
• HTTPS Recommended: Always use HTTPS URLs when possible
• No Local Files: Current version only supports HTTP/HTTPS URLs

Development

# Clone the repository
git clone https://github.com/pinkpixel-dev/llmstxt-mcp.git
cd llmstxt-mcp

# Install dependencies
npm install

# Build the server
npm run build

# Test with inspector
npm run inspector

Error Handling

The server provides detailed error messages for:

• Invalid or unreachable URLs
• Domain restriction violations
• Timeout errors
• HTTP errors (404, 500, etc.)

License

MIT License - see LICENSE file for details.

---

Made with ❤️ by Pink Pixel