Crawl Parity

Part of the GEO Stack research programme — where we discovered that Googlebot and AI crawlers make independent authority assessments on new domains.

An MCP (Model Context Protocol) server for analyzing Googlebot vs AI crawler parity from Nginx access logs and Google Search Console (GSC) data. Determines how consistently AI crawlers access your content relative to Googlebot.

Installation

npm install

Requires Node.js 18+.

Usage

Start the Server

npm start

Or directly:

node src/index.js

Tool: analyze_logs

Parse an Nginx combined log format file and classify requests by crawler type.

Input:

• log_path (string, required): Path to Nginx access log file

Output:

{
  "googlebot_requests": 1234,
  "ai_crawler_requests": 456,
  "parity_ratio": 37.0,
  "parity_level": "low_parity",
  "paths_googlebot": 89,
  "paths_ai_crawler": 42,
  "unique_paths": 120
}

Tool: gsc_crossref

Cross-reference log analysis results with GSC search analytics data.

Input:

• logs_analysis (object, required): Output from analyze_logs or similar structure
• gsc_data (array, required): GSC analytics records with page, impressions, clicks, ctr, position

Output:

{
  "both": 45,
  "logs_only": 20,
  "gsc_only": 35,
  "total_analyzed": 100
}

Tool: parity_report

Generate a comprehensive crawl parity report combining logs and GSC data.

Input:

• log_path (string, required): Path to Nginx access log file
• gsc_data (array, required): GSC analytics records

Output:

{
  "timestamp": "2026-03-26T14:30:00.000Z",
  "logs_analysis": { ... },
  "gsc_crossref": { ... },
  "summary": {
    "googlebot_activity": "detected",
    "ai_crawler_activity": "detected",
    "parity_status": "low_parity",
    "parity_percentage": 37.0,
    "recommendation": "Low parity - AI crawlers are significantly underrepresented"
  }
}

Parity Levels

Level	Ratio	Meaning
high_parity	≥80%	AI crawlers access your content nearly as often as Googlebot
medium_parity	40-79%	Moderate parity; consider optimizing AI crawler discoverability
low_parity	<40%	AI crawlers significantly underrepresented; review robots.txt and crawlability
insufficient_data	N/A	Googlebot requests = 0; cannot calculate parity

Bot Signatures

Googlebot Detection

• Googlebot (with space, slash, or end-of-string)
• Googlebot-Image
• AdsBot-Google

AI Crawler Detection

• GPTBot (OpenAI)
• ClaudeBot (Anthropic)
• PerplexityBot (Perplexity)
• YouBot (You.com)
• Bytespider (ByteDance)
• Google-Extended (Google AI)
• CCBot (Common Crawl)
• PetalBot (Alibaba)
• Applebot-Extended (Apple)

Nginx Log Format

Expects combined log format:

ip - - [timestamp] "METHOD /path HTTP/1.1" status bytes "referer" "user-agent"

License

MIT — See LICENSE file

Author

Artur Ferreira / The GEO Lab

---

A project by The GEO Lab — Generative Engine Optimisation research