Scraper Health

Autonomous Self-Healing Scraper Fleet Manager for Bright Data Scraper Studio

The first agent that closes Bright Data's self-healing loop end-to-end without a human in the middle — turning the resume_automation_job endpoint the official demo couldn't find into a one-command "deploy and forget" scraper fleet.

Built for WeMakeDevs Into the Scrape-Verse hackathon (Aug 17–23, 2026).

🎯 The Problem

Web scrapers break silently. When a target site renames a CSS class, moves a field, or redesigns a layout, traditional scrapers return empty/garbage rows with no alarm. The failure is discovered only when a human notices the data is wrong — by which point downstream systems have been consuming bad data for hours or days.

✅ The Solution

Scraper-Health-MCP is an autonomous AI agent that wraps Bright Data's Scraper Studio self-healing API into a fully automated loop:

1. Monitor — Runs scrapers on schedule and health-checks every result
2. Detect — Catches breakage via schema validation, null-field detection, and row-count anomaly detection (Sentinel-style statistical baselines)
3. Heal — Triggers Bright Data's AI self-healing (refactor_template)
4. Auto-Approve — Programmatically accepts the AI's proposed extraction diff via resume_automation_job (the step Bright Data's own demo couldn't do)
5. Verify — Re-scrapes and confirms health is restored
6. Escalate — If healing fails, regenerates the scraper from scratch

🔑 The Technical Unlock

Bright Data's official self-healing demo repo (anil-bd/scraper-studio-self-healing-demo) explicitly states the public API "does not document an endpoint to approve programmatically" and exits with code 3 (awaiting approval) when the heal hits pending_answer, requiring manual UI approval.

The current docs DO expose POST /resume_automation_job with {message: true, auto_save: true} to accept the diff automatically. This means a fully autonomous, hands-off self-healing loop is now buildable — and almost no competitor will have found this.

🤖 MCP Server

The agent exposes 5 MCP tools callable from Claude Code, Cursor, or any MCP-compatible AI agent:

Tool	Description
create_scraper	Create a new scraper from URL + description via Bright Data AI Flow
run_collector	Trigger a collector and return results
health_check	Run collector + check output health (schema, nulls, row-count anomaly)
self_heal	Full autonomous loop: detect → heal → auto-approve → re-scrape → verify
verify	Verify a healed collector has restored health

🏗️ Architecture

Register collector (ID + schema + required fields)
        │
        ▼
  Schedule / trigger run  ──►  POST /dca/trigger
        │
        ▼
  Health-check result  ──►  GET /dca/dataset  (schema drift? nulls? row-count anomaly?)
        │
   ├── healthy ─► log success, schedule next run
   └── broken  ─► TRIGGER SELF-HEAL
                    │
                    ▼
          POST /refactor_template  (targeted heal prompt from the failed field)
                    │
                    ▼
          Poll /refactor_template/progress
                    │
                    ├── status:"pending_answer" ─► POST /resume_automation_job
                    │                              {message:true, auto_save:true}   ← the unlock
                    └── done/failed
                    │
                    ▼
          Re-scrape + verify against baseline schema
                    │
                    ├── passes ─► log heal event, resume schedule
                    └── fails  ─► escalate: regenerate scraper from scratch (Workflow 1)
                                  + alert/notification

🛠️ Tech Stack

Layer	Tech
Scraper infra	Bright Data Scraper Studio (CLI + AI Flow API)
Backend	Python 3.12 + FastAPI
Agent interface	MCP server (5 tools)
Anomaly detection	In-package stats (mean/std row-count + schema validator)
Dashboard	Jinja2 templates + dark-themed HTML/CSS/JS
Deploy	Vercel serverless (free tier)
Repo	github.com/0xConsole/scraper-health-mcp

🚀 Setup

# Clone
git clone https://github.com/0xConsole/scraper-health-mcp.git
cd scraper-health-mcp

# Install
pip install -r requirements.txt

# Run (mock mode — no API key needed)
python -m uvicorn app.main:app --reload

# Run (live mode — with Bright Data API key)
export BRIGHTDATA_API_KEY="your-key-here"
python -m uvicorn app.main:app --reload

Open http://localhost:8000 to see the dashboard.

📡 API Endpoints

Method	Path	Description
GET	/	Dashboard
GET	/api/health	Health check
GET	/api/status	Full orchestrator status
GET	/api/tools	List MCP tools
GET	/api/mcp/manifest	MCP server manifest
POST	/api/create_scraper	Create scraper (MCP tool 1)
POST	/api/run_collector	Run collector (MCP tool 2)
POST	/api/health_check	Health check (MCP tool 3)
POST	/api/self_heal	Self-heal loop (MCP tool 4)
POST	/api/verify	Verify heal (MCP tool 5)
POST	/api/demo	Full demo (break + heal cycle)
POST	/api/trigger_breakage	Simulate breakage + trigger heal

🎪 Demo

1. Click "Run Full Demo" on the dashboard
2. The agent health-checks all registered collectors
3. Simulates a site change (breakage) on the first collector
4. Detects the breakage via health check (null fields, row count anomaly)
5. Triggers AI self-healing (refactor_template)
6. Polls until the AI proposes a diff (pending_answer)
7. Auto-approves the diff via resume_automation_job ← the unlock
8. Re-scrapes and verifies health is restored
9. Logs the entire heal event in the timeline

📊 What's Real vs Mocked

Component	Status
MCP server (5 tools)	✅ Real — fully functional
Health checker (schema, nulls, anomaly)	✅ Real — Sentinel-style stats
Heal orchestrator (full loop)	✅ Real — all states wired
Auto-approve (resume_automation_job)	✅ Real — calls actual endpoint when API key set
Bright Data API calls	🔧 Mock mode (no key) / Real (with key)
Demo collectors	✅ 3 seeded (HN, e-commerce, docs)
Dashboard + heal timeline	✅ Real — live data from orchestrator

📝 License

Apache 2.0

🔗 Links

• Live demo: https://scraper-health-mcp.vercel.app
• GitHub: https://github.com/0xConsole/scraper-health-mcp
• Hackathon: WeMakeDevs Into the Scrape-Verse (Aug 17–23, 2026)
• Bright Data Scraper Studio: https://brightdata.com/scraper-studio

---

Built by Sentinel Dev · Team: Xayaan Ibrahim (Individual)