The Agentic SEO Engine

A headless, API-first MCP server that transforms unstructured creative portfolios into validated, machine-readable llms.txt manifests for the agentic web.

Live deployment: https://agentic-seo-engine-796932810904.us-central1.run.app/sse

Built for the Kaggle AI Agents: Intensive Vibe Coding Capstone — Agents for Business track.

---

The Problem

Traditional SEO is becoming obsolete in the agentic web. Autonomous agents — systems that crawl the web to procure talent, data, and services on behalf of enterprises — rely on clean, machine-readable specifications rather than visual layouts. Visual-heavy portfolios, experiential landing pages, and creative directories are fundamentally invisible to these programmatic crawlers, leaving human creators undiscoverable by the very systems increasingly responsible for sourcing them.

The Solution

The Agentic SEO Engine bridges this gap. Given any portfolio URL, it:

1. Ingests the page deterministically, stripping layout noise and presentation styling
2. Synthesizes the creator's implicit disciplines, tone, and structure using Gemini inside a strict instruction harness
3. Validates the output against a rigid Pydantic V2 schema, guaranteeing standards-compliant, structured JSON every time

The result is a compiled llms.txt manifest — a foundational SEO artifact for the agentic internet, ensuring human creators remain discoverable by enterprise procurement agents.

Why Agents

A static scraper can extract text, but it cannot reason about implicit creative intent — the difference between a portfolio's literal content and its actual thematic identity. This task is fundamentally interpretive, which is why an LLM-driven synthesis step, not a fixed extraction template, sits at the core of the pipeline.

Architecture

[ Unstructured URL ] ➔ [ Deterministic Ingestion ] ➔ [ Gemini Inference ] ➔ [ Pydantic V2 Validation ] ➔ [ Verified llms.txt JSON ]

• Ingestion layer: httpx + BeautifulSoup strip scripts, styles, nav, and footer noise, leaving core structural text.
• Inference layer: The cleaned text is passed to gemini-2.5-flash-lite inside a disciplined system instruction that rejects conversational filler and assumption.
• Validation layer: Pydantic V2 models (CreatorProfile, ProjectOrAsset, MetadataPair) enforce strict typing on the model's output, with a recursive schema-cleaning step that strips permissive additionalProperties markers before the request is even sent.
• Interface layer: Exposed as a single tool, generate_llms_manifest, via a headless Model Context Protocol (MCP) server — no UI, pure machine-to-machine utility.

Security

• No API keys or credentials are hardcoded anywhere in the codebase.
• The Gemini API key is injected at the infrastructure level as a Cloud Run environment variable, never bundled into the container image.
• All model inputs/outputs pass through strict Pydantic V2 schema validation, preventing malformed or injected data from reaching the final manifest.
• Service access is scoped via GCP IAM policy bindings controlling which identities can build, push, and invoke the deployed container.

Tech Stack

• MCP: Official MCP Python SDK (mcp.server.fastmcp.FastMCP)
• LLM: Google Gemini (google-genai SDK, gemini-2.5-flash-lite)
• Validation: Pydantic V2
• Scraping: httpx, BeautifulSoup4
• Deployment: Google Cloud Run (Docker, python:3.11-slim)

Setup & Local Development

Prerequisites

• Python 3.11+
• A Google Gemini API key (Google AI Studio)

Installation

git clone https://github.com/<your-username>/agentic-seo-engine.git
cd agentic-seo-engine
pip install -r requirements.txt

Configuration

Create a .env file in the project root (this file is git-ignored and never committed): GEMINI_API_KEY=your_actual_key_here

Running Locally

python server.py

With no PORT environment variable set, the server runs in stdio transport — ideal for local testing with the MCP Inspector or Claude Desktop.

Deploying to Google Cloud Run

gcloud run deploy agentic-seo-engine \
  --source . \
  --region us-central1 \
  --allow-unauthenticated \
  --port 8080

gcloud run services update agentic-seo-engine \
  --region us-central1 \
  --update-env-vars GEMINI_API_KEY=your_actual_key_here

When PORT is present in the environment (as Cloud Run sets automatically), the server switches to sse transport and binds to 0.0.0.0 on the assigned port.

Usage

Connect any MCP-compatible client to the server's SSE endpoint and invoke the generate_llms_manifest tool with a target portfolio URL:

from mcp import ClientSession
from mcp.client.sse import sse_client

async with sse_client("https://agentic-seo-engine-796932810904.us-central1.run.app/sse") as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()
        result = await session.call_tool(
            "generate_llms_manifest",
            {"portfolio_url": "https://example-creator-portfolio.com"}
        )
        print(result)

Example Output

{
  "profile": {
    "name": "Vibe Code",
    "role_title": "AI App Builder",
    "summary": "Builds apps and websites with AI, focusing on rapid development and user-friendly interfaces...",
    "disciplines": ["AI Application Development", "Web Development", "Product Prototyping"]
  },
  "featured_works": [
    {
      "title": "Maison",
      "description": "Editorial home goods storefront.",
      "tags": ["ecommerce", "storefront", "template"]
    }
  ]
}

Project Status

This project was built as a rapid capstone prototype. It is deployed live and functioning, and serves as the foundation for continued development in a subsequent, more extensive agentic build.

Author

Ralph Torres