Content Pipeline

PyPI version Downloads License: MIT Python

A content analysis and digest pipeline for YouTube videos and X (Twitter) feeds, exposed as MCP tools. Extract transcripts, fetch posts from curated accounts, and generate key takeaways, TLDRs, social hooks, and comic-book infographics — all callable by any MCP-compatible AI client like Claude Desktop.

flowchart LR
    A[YouTube URL<br/>or X feed] --> B[Extract content<br/>Supadata / X API]
    B --> C[Claude analysis<br/>takeaways, TLDR, hook]
    C --> D[Gemini image<br/>comic infographic]
    D --> E[Sync to GitHub<br/>markdown + image]

Why?

Keeping up with YouTube channels and X accounts means scattered tabs, manual note-taking, and lost insights. This MCP server turns content consumption into structured, chainable tools. Analyse a Bloomberg video, digest your X feed, generate infographics, and sync everything to GitHub — all from a single conversation with Claude.

Role in ecosystem

• Uses: mcp-llm-eval for evaluation and CI quality gates
• Produces: benchmark JSON written to llm-benchmarks under text-generation/content-pipeline-*.json
• Visible at: LLMShot's Text Generation domain, Content Pipeline sub-benchmark

The eval dataset (eval/dataset.json) lives with this repo because the questions are specific to YouTube and X feed analysis — the dataset belongs with the use case, not the engine.

Quick Start

uvx mcp-content-pipeline

Or install explicitly:

uv tool install mcp-content-pipeline
mcp-content-pipeline

Claude Desktop Configuration

Add to your Claude Desktop MCP config (~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "content-pipeline": {
      "command": "/usr/local/bin/uvx",
      "args": ["mcp-content-pipeline"],
      "env": {
        "MCP_CP_ANTHROPIC_API_KEY": "sk-ant-...",
        "MCP_CP_SUPADATA_API_KEY": "sd_...",
        "MCP_CP_GITHUB_TOKEN": "ghp_...",
        "MCP_CP_GITHUB_REPO": "your-username/your-repo",
        "MCP_CP_GEMINI_API_KEY": "your-gemini-api-key",
        "MCP_CP_X_BEARER_TOKEN": "your-x-bearer-token",
        "MCP_CP_X_ACCOUNTS": "karpathy,bcherny,atmoio,steipete",
        "MCP_CP_X_TOPICS": "AI,tech,engineering"
      }
    }
  }
}

Usage

Once configured in Claude Desktop, use the tools in a single conversation.

Tip: Including "content-pipeline" for YouTube or "X feed" for Twitter helps Claude Desktop route to the right tool.

YouTube Analysis

"Use content-pipeline to analyse this video: https://www.youtube.com/watch?v=..." "Generate an image for this analysis" "Sync the analysis and image to GitHub"

Or all in one prompt:

"Use content-pipeline to analyse this video, generate the image, and sync to GitHub: https://www.youtube.com/watch?v=..."

X Feed Digest

"Analyse the X feed" "Analyse the X feed for karpathy, bcherny, atmoio, and steipete about AI today" "Analyse the X feed from the last 7 days"

Or with the full pipeline:

"Analyse the X feed, generate the image, and sync to GitHub"

Tools

Tool	Description	Requires
analyse_video	Analyse a single YouTube video — transcript, takeaways, TLDR, social hook	ANTHROPIC_API_KEY, SUPADATA_API_KEY
batch_analyse	Analyse multiple videos from a URL list or config file	ANTHROPIC_API_KEY, SUPADATA_API_KEY
list_channel_videos	Fetch recent videos from a YouTube channel	YOUTUBE_API_KEY
sync_to_github	Push analyses as markdown files to a GitHub repo	GITHUB_TOKEN, GITHUB_REPO
analyse_x_feed	Analyse recent posts from curated X accounts — daily digest	X_BEARER_TOKEN
generate_image	Generate comic-book infographic from analysis result	GEMINI_API_KEY

Environment Variables

All prefixed with MCP_CP_:

Variable	Required	Description
MCP_CP_ANTHROPIC_API_KEY	Yes	Anthropic API key for Claude analysis
MCP_CP_SUPADATA_API_KEY	Yes for YouTube	Supadata API key for YouTube transcript extraction
MCP_CP_YOUTUBE_API_KEY	No	YouTube Data API v3 key (only for list_channel_videos)
MCP_CP_GITHUB_TOKEN	For sync	GitHub personal access token
MCP_CP_GITHUB_REPO	For sync	Target repo in owner/repo format
MCP_CP_GITHUB_BRANCH	No	Branch to push to (default: main)
MCP_CP_GITHUB_OUTPUT_DIR	No	Output directory for YouTube analyses (default: content/youtube)
MCP_CP_GITHUB_X_OUTPUT_DIR	No	Output directory for X digests (default: content/x-digest)
MCP_CP_IMAGE_OUTPUT_DIR	No	Directory for generated images (default: ~/Downloads)
MCP_CP_CLAUDE_MODEL	No	Claude model to use (default: claude-sonnet-4-6)
MCP_CP_MAX_TRANSCRIPT_TOKENS	No	Max transcript length in tokens (default: 100000)
MCP_CP_GEMINI_API_KEY	For image	Google AI Studio API key for image generation
MCP_CP_GEMINI_MODEL	No	Gemini model for images (default: gemini-3.1-flash-image-preview)
MCP_CP_X_BEARER_TOKEN	For X digest	X API v2 bearer token
MCP_CP_X_ACCOUNTS	For X digest	Comma-separated X usernames
MCP_CP_X_TOPICS	No	Comma-separated topics (default: AI,tech)

Cost Projections

Estimated monthly costs for two usage patterns:

Service	Daily (every day)	Weekly X + daily YouTube
YouTube analysis (Claude API)	~$3–5/mo (1 video/day)	~$3–5/mo (1 video/day)
X feed digest (Claude API)	~$2–3/mo	~$0.50/mo
Image generation (Gemini API)	~$2/mo ($0.067/image)	~$2/mo ($0.067/image)
X API reads	~$4/mo ($0.13/day)	~$0.60/mo ($0.15/week)
Supadata transcript API	~$0 (free tier: 100/mo)	~$0 (free tier: 100/mo)
Total (excl. Claude API)	~$6–9/mo	~$3–5/mo

Claude API costs depend on your Anthropic billing plan and are not included in the totals above. If you already use Claude Pro ($20/mo), there is no additional Claude cost. The X API spending cap can be configured in the developer console.

What this replaces

Subscription	Monthly cost	What the pipeline covers instead
Google One AI Premium	~$20/mo	Image generation via Gemini API (~$2/mo)
X Premium	~$8/mo	X feed reading via API (~$0.60–4/mo)
YouTube Premium	~$14/mo	Transcript extraction via Supadata (free tier)
Total saved	~$42/mo	Pipeline cost: ~$3–9/mo (plus your existing Claude plan)

Eval Gates

Prompt and model changes are automatically evaluated in CI using mcp-llm-eval. The eval dataset covers both YouTube analysis and X feed digest prompts, benchmarking 8 models (Claude Opus 4.7, Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.5, GPT-4o-mini, Gemini 3 Flash Preview, Gemini 2.5 Flash, Gemini 2.5 Flash-Lite) on the same test cases. PRs that touch system prompts or model config trigger an evaluation run that scores faithfulness and relevance against a reference dataset. The PR is blocked if quality regresses below configured thresholds.

See .eval-gate.yml for threshold configuration and eval/dataset.json for the test dataset.

Running benchmarks locally

The benchmark requires API keys for all providers. Create a .env file in the project root:

ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
GOOGLE_API_KEY=AIza...

Then run:

make benchmark        # Run eval against all 8 models
make benchmark-copy   # Copy results to llm-benchmarks repo

Results are written to eval/results/ (gitignored). The benchmark output feeds into LLMShot via the llm-benchmarks repo at text-generation/content-pipeline-summary.json and text-generation/content-pipeline-benchmark.json.

This project uses mcp-llm-eval for benchmarking and CI quality gates. Production uses Claude Sonnet (claude-sonnet-4-6). The benchmark tracks all 8 models (3 Anthropic, 2 OpenAI, 3 Google) so we can re-evaluate provider choice as capabilities and pricing evolve.

Development

git clone https://github.com/your-username/mcp-content-pipeline.git
cd mcp-content-pipeline
uv sync
uv run pytest -v --cov=src/mcp_content_pipeline
uv run ruff check src/ tests/

Security

• All credentials are configured via local environment variables — never committed to the repo
• The tool is open source but your API keys, YouTube key, and GitHub token stay on your machine
• Never create a .env file in the repo — use shell exports or Claude Desktop config instead

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feat/my-feature)
3. Commit using Conventional Commits (feat: add new feature)
4. Push and open a Pull Request

License

MIT