Prometheus Mcp Agent

🚀 Overview

This is a production-ready, enterprise-grade Prometheus MCP (Model Context Protocol) server designed for DevOps and SRE teams. It provides intelligent observability, automated root cause analysis, and natural language interaction with your Prometheus metrics.

✨ Key Features

🔍 Intelligent Observability

• 20+ Advanced Tools for comprehensive monitoring
• Natural Language Interface - Ask questions in plain English
• Multi-Prometheus Support - Query multiple Prometheus instances
• Real-time Analysis - Get instant insights

🧠 AI-Powered Analysis

• Root Cause Analysis - Automated RCA with hypothesis generation
• Anomaly Detection - Statistical anomaly detection
• Alert Correlation - Find relationships between alerts
• Incident Investigation - Comprehensive multi-metric analysis

📚 Learning & Memory

• Context Memory - Learns from past incidents
• Pattern Recognition - Identifies common alert patterns
• Solution Tracking - Remembers what worked
• Query Optimization - Learns from successful queries

☁️ GCP Integration

• Cloud Logging - Query GKE pod logs
• Cloud Monitoring - Access GCP metrics
• Cloud Trace - Distributed tracing
• Correlation - Correlate logs, metrics, and traces

📋 Quick Start

Prerequisites

1. GCP VM with:

   • Debian Linux
   • NVIDIA T4 GPU
   • CUDA support
   • 16GB+ RAM
   • Internal network access to Prometheus

2. Prometheus Server:

   • Accessible at https://prometheus.intalto.ruckuswireless.com
   • Or your Prometheus endpoint

Deployment

# 1. Clone/upload project to VM
cd ~/ts-ai-agent-workshop

# 2. Run deployment script
chmod +x scripts/deploy_gcp.sh
./scripts/deploy_gcp.sh

# 3. Configure Prometheus connection
nano config/prometheus_config.yaml

# 4. Start the agent
source venv/bin/activate
python pkg/mcp/agent.py

Basic Usage

# Start interactive agent
python pkg/mcp/agent.py

# Example queries:
🔹 Your query: What is the cluster health?
🔹 Your query: Show me critical alerts
🔹 Your query: investigate High CPU usage in production
🔹 Your query: Root cause analysis for HighErrorRate alert

🏗️ Architecture

┌─────────────────────────────────────────┐
│         User Query (Natural Language)   │
└─────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────┐
│         MCP Agent (Ollama LLM)          │
│  - Natural Language Understanding       │
│  - Tool Selection & Orchestration        │
│  - Context Memory Integration           │
└─────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────┐
│         MCP Server (FastMCP)            │
│  - 20+ Monitoring Tools                 │
│  - Query Caching                        │
│  - Multi-instance Support                │
└─────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────┐
│         Prometheus API                  │
│  - Metric Queries                       │
│  - Alert Queries                         │
│  - Range Queries                        │
└─────────────────────────────────────────┘

🛠️ Available Tools

Basic Monitoring

• current_metric_for_pods - Get metric values for pods
• top_n_pods_by_metric - Find top pods by metric
• node_disk_usage - Node disk usage analysis
• describe_cluster_health - Overall cluster health

Advanced Query

• query_promql - Execute PromQL queries
• query_range - Time-series range queries
• get_metric_metadata - Get metric information
• correlate_metrics - Correlate multiple metrics

Alert Analysis

• analyze_active_alerts - Analyze firing alerts
• alert_correlation - Correlate related alerts
• alert_timeline - Build alert timeline

Incident Investigation

• investigate_incident - Multi-metric investigation
• root_cause_analysis - Automated RCA
• analyze_error_patterns - Error pattern analysis

Anomaly Detection

• detect_anomalies - Statistical anomaly detection
• compare_time_periods - Compare metrics across periods

📖 Documentation

• Architecture Guide - System architecture and design
• Deployment Guide - Step-by-step deployment instructions
• User Guide - How to use the agent
• Implementation Summary - What was built and why

🎯 Use Cases

1. Morning Health Check

🔹 Your query: What is the cluster health?
→ Returns: Health score, pod status, node status, alerts summary

2. Incident Investigation

🔹 Your query: investigate High error rate in payment service
→ Returns: Multi-metric analysis, RCA, similar past incidents, recommendations

3. Alert Analysis

🔹 Your query: Show me all critical alerts and analyze their root causes
→ Returns: Alert list with severity, correlation analysis, RCA for each

4. Capacity Planning

🔹 Your query: Which pods are approaching resource limits?
→ Returns: Resource usage analysis, trends, recommendations

🔧 Configuration

Prometheus Config (config/prometheus_config.yaml)

prometheus_instances:
  - name: prometheus_1
    base_url: "https://prometheus.intalto.ruckuswireless.com"
    headers: {}
    disable_ssl: false

Ollama Config (config/ollama_config.yaml)

ollama_url: "http://localhost:11434"
ollama_model: "qwen2.5-coder:7b"

MCP Server Config (config/mcp_server_config.yaml)

mcp_server_url: "http://localhost:8001/mcp"

🚦 Service Management

# Check service status
sudo systemctl status prometheus-mcp-server
sudo systemctl status ollama

# View logs
sudo journalctl -u prometheus-mcp-server -f
sudo journalctl -u ollama -f

# Restart services
sudo systemctl restart prometheus-mcp-server
sudo systemctl restart ollama

🔒 Security

• Internal Network Only - No external HTTP/HTTPS access
• Service Accounts - GCP service account authentication
• TLS Support - Secure connections to Prometheus
• Credential Management - Secure credential handling

📊 Performance

• Query Caching - 5-minute TTL for frequent queries
• Connection Reuse - Prometheus clients are reused
• Async Operations - Non-blocking operations
• GPU Acceleration - Leverages T4 GPU for LLM inference

🧪 Testing

# Test Ollama
curl http://localhost:11434/api/tags

# Test MCP Server
curl http://localhost:8001/health

# Test Prometheus connectivity
python3 -c "
from prometheus_api_client import PrometheusConnect
client = PrometheusConnect(url='https://prometheus.intalto.ruckuswireless.com', disable_ssl=False)
print(client.all_metrics()[:5])
"

🐛 Troubleshooting

Agent Not Responding

1. Check services: sudo systemctl status prometheus-mcp-server ollama
2. Check logs: sudo journalctl -u prometheus-mcp-server -f
3. Test connectivity: curl http://localhost:8001/health

GPU Issues

# Check GPU
nvidia-smi

# Check Ollama GPU support
ollama info

Prometheus Connection Issues

# Test connectivity
curl -k https://prometheus.intalto.ruckuswireless.com/api/v1/query?query=up

# Check SSL
openssl s_client -connect prometheus.intalto.ruckuswireless.com:443

🔮 Future Enhancements

• Predictive alerting
• Automated remediation
• Multi-modal analysis (logs + metrics + traces)
• Incident management integration (PagerDuty, etc.)
• Real-time streaming analysis
• Custom tool framework
• Dashboard generation
• Automated report generation

📝 Key Improvements Over Original

Original Implementation

• 4 basic tools
• Simple query execution
• No learning/memory
• No incident investigation
• Basic error handling

Enhanced Implementation

• 20+ advanced tools
• Intelligent query orchestration
• Context memory and learning
• Comprehensive incident investigation
• Advanced alert analysis
• Robust error handling
• GCP integration
• Statistical analysis
• Anomaly detection

🤝 Contributing

This is a production system. For enhancements:

1. Review architecture documentation
2. Test thoroughly
3. Update documentation
4. Follow existing code patterns

📄 License

See LICENSE file for details.

🙏 Acknowledgments

Built for DevOps/SRE teams to make observability and incident response faster and more intelligent.

---

Ready to deploy? Start with the Deployment Guide

Need help? Check the User Guide

Want to understand the architecture? Read the Architecture Guide