Ai Usage Metrics

A Model Context Protocol (MCP) server for tracking AI usage metrics and structured logs across your applications. Monitor model calls, analyze usage patterns, track costs, and debug AI interactions with a clean, extensible architecture.

Table of Contents

• Overview
• Features
• Quick Start
• One-Click Install
  • Claude Desktop
  • Claude Code CLI
  • Cursor
  • Windsurf
  • VS Code + Continue
  • Cline
  • Zed
• Manual Installation
• Usage
• API Reference
• Data Model
• Architecture
• Extending the Server
• Development
• Testing
• License

Overview

This MCP server provides a centralized way to track and analyze AI model usage across your applications. Whether you're building chatbots, RAG systems, or autonomous agents, this server helps you:

• Track every model call with full context (inputs, outputs, metadata)
• Analyze usage patterns across projects, environments, and users
• Monitor costs via token counting and aggregation
• Debug interactions by replaying sessions and conversations
• Ensure safety by logging safety check results

The server implements the Model Context Protocol specification, making it compatible with Claude, Cursor, Windsurf, and any MCP-enabled AI assistant or agent framework.

Features

🚀 Quick Start

No installation required! Just add the server to your AI platform config:

{
  "mcpServers": {
    "ai-usage-metrics": {
      "command": "npx",
      "args": ["-y", "ai-usage-metrics-mcp"]
    }
  }
}

That's it! The package will be automatically downloaded and run when your AI platform starts.

📦 One-Click Install

Choose your AI platform and copy the configuration.

Claude Desktop

{
  "mcpServers": {
    "ai-usage-metrics": {
      "command": "npx",
      "args": ["-y", "ai-usage-metrics-mcp"]
    }
  }
}

mkdir -p ~/Library/Application\ Support/Claude

cat > ~/Library/Application\ Support/Claude/claude_desktop_config.json << 'EOF'
{
  "mcpServers": {
    "ai-usage-metrics": {
      "command": "npx",
      "args": ["-y", "ai-usage-metrics-mcp"]
    }
  }
}
EOF

{
  "mcpServers": {
    "ai-usage-metrics": {
      "command": "npx",
      "args": ["-y", "ai-usage-metrics-mcp"]
    }
  }
}

New-Item -ItemType Directory -Force -Path "$env:APPDATA\Claude"

@'
{
  "mcpServers": {
    "ai-usage-metrics": {
      "command": "npx",
      "args": ["-y", "ai-usage-metrics-mcp"]
    }
  }
}
'@ | Out-File -FilePath "$env:APPDATA\Claude\claude_desktop_config.json" -Encoding UTF8

{
  "mcpServers": {
    "ai-usage-metrics": {
      "command": "npx",
      "args": ["-y", "ai-usage-metrics-mcp"]
    }
  }
}

mkdir -p ~/.config/Claude
cat > ~/.config/Claude/claude_desktop_config.json << 'EOF'
{
  "mcpServers": {
    "ai-usage-metrics": {
      "command": "npx",
      "args": ["-y", "ai-usage-metrics-mcp"]
    }
  }
}
EOF

Claude Code CLI

Recommended: Use the Claude Code command:

claude mcp add ai-usage-metrics -- npx -y ai-usage-metrics-mcp

Or manually edit ~/.claude/settings.json:

{
  "mcpServers": {
    "ai-usage-metrics": {
      "command": "npx",
      "args": ["-y", "ai-usage-metrics-mcp"]
    }
  }
}

Cursor

Config file: ~/.cursor/mcp.json

{
  "mcpServers": {
    "ai-usage-metrics": {
      "command": "npx",
      "args": ["-y", "ai-usage-metrics-mcp"]
    }
  }
}

One-liner setup (macOS/Linux):

mkdir -p ~/.cursor
cat > ~/.cursor/mcp.json << 'EOF'
{
  "mcpServers": {
    "ai-usage-metrics": {
      "command": "npx",
      "args": ["-y", "ai-usage-metrics-mcp"]
    }
  }
}
EOF

Windsurf

Config file: ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "ai-usage-metrics": {
      "command": "npx",
      "args": ["-y", "ai-usage-metrics-mcp"]
    }
  }
}

One-liner setup (macOS/Linux):

mkdir -p ~/.codeium/windsurf
cat > ~/.codeium/windsurf/mcp_config.json << 'EOF'
{
  "mcpServers": {
    "ai-usage-metrics": {
      "command": "npx",
      "args": ["-y", "ai-usage-metrics-mcp"]
    }
  }
}
EOF

VS Code + Continue

Config file: ~/.continue/config.json

{
  "experimental": {
    "modelContextProtocolServers": [
      {
        "transport": {
          "type": "stdio",
          "command": "npx",
          "args": ["-y", "ai-usage-metrics-mcp"]
        }
      }
    ]
  }
}

Cline

Config file: VS Code Settings (settings.json)

{
  "cline.mcpServers": {
    "ai-usage-metrics": {
      "command": "npx",
      "args": ["-y", "ai-usage-metrics-mcp"],
      "disabled": false
    }
  }
}

Zed

Config file: ~/.config/zed/settings.json

{
  "context_servers": {
    "ai-usage-metrics": {
      "command": {
        "path": "npx",
        "args": ["-y", "ai-usage-metrics-mcp"]
      }
    }
  }
}

Other MCP-Compatible Platforms

For any MCP-compatible platform, use these standard connection details:

Generic MCP Configuration:

{
  "name": "ai-usage-metrics",
  "transport": "stdio",
  "command": "npx",
  "args": ["-y", "ai-usage-metrics-mcp"]
}

Manual Installation

For most users, the npx method above is recommended. Manual installation is useful for development or if you prefer a global install.

Global Install via npm

npm install -g ai-usage-metrics-mcp

Then use ai-usage-metrics-mcp as the command in your MCP config instead of npx.

From Source (for development)

# Clone the repository
git clone https://github.com/charlie-818/ai-history-mcp.git
cd ai-history-mcp

# Install dependencies
npm install

# Build the project
npm run build

# Run the server
npm start

Package Manager Scripts

Verifying Installation

After installation, verify the server works:

# Test the server starts correctly
echo '{"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {"capabilities": {}}}' | node dist/index.js

You should see a JSON response with server capabilities.

Usage

Logging Model Calls

After each AI model invocation in your application, log the call:

// Using MCP client
await mcpClient.callTool("log_model_call", {
  project: "my-chatbot",
  environment: "prod",
  sessionId: "session-abc-123",
  modelName: "claude-3-opus",
  inputMessages: [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: "What is the capital of France?" }
  ],
  outputMessages: [
    { role: "assistant", content: "The capital of France is Paris." }
  ],
  tokensIn: 45,
  tokensOut: 12,
  latencyMs: 234
});

Response:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "stored": true
}

Searching Calls

Find specific calls with flexible filtering:

// Search by project and date range
const calls = await mcpClient.callTool("search_model_calls", {
  project: "my-chatbot",
  environment: "prod",
  from: "2024-01-01T00:00:00Z",
  to: "2024-01-31T23:59:59Z",
  limit: 100
});

// Search by user
const userCalls = await mcpClient.callTool("search_model_calls", {
  userId: "user-12345",
  modelName: "gpt-4"
});

Session Management

Track conversations by grouping calls into sessions:

// List all sessions for a project
const sessions = await mcpClient.callTool("list_sessions", {
  project: "my-chatbot",
  environment: "prod"
});

// Get all calls in a specific session
const sessionCalls = await mcpClient.callTool("get_session_calls", {
  sessionId: "session-abc-123"
});

Session Summary Response:

{
  "sessionId": "session-abc-123",
  "project": "my-chatbot",
  "environment": "prod",
  "firstCallAt": "2024-01-15T10:30:00Z",
  "lastCallAt": "2024-01-15T10:45:00Z",
  "callCount": 8,
  "totalTokensIn": 1250,
  "totalTokensOut": 890,
  "avgLatencyMs": 245
}

Aggregate Metrics

Get high-level usage statistics:

// Get metrics for a project
const metrics = await mcpClient.callTool("get_aggregate_metrics", {
  project: "my-chatbot",
  environment: "prod",
  from: "2024-01-01T00:00:00Z",
  to: "2024-01-31T23:59:59Z"
});

Response:

{
  "callCount": 15420,
  "totalTokensIn": 2450000,
  "totalTokensOut": 1890000,
  "avgLatencyMs": 312
}

📚 API Reference

Tools

log_model_call

Log a model call for tracking.

Returns: { id: string, stored: boolean }

search_model_calls

Search logged calls with filters.

Returns: Array of ModelCallLog objects (message content truncated for safety)

list_sessions

List session summaries.

Returns: Array of SessionSummary objects

get_session_calls

Get all calls for a session.

Returns: Array of ModelCallLog objects (chronological order)

get_aggregate_metrics

Get aggregate metrics across calls.

Returns: { callCount, totalTokensIn, totalTokensOut, avgLatencyMs? }

Resources

Resources provide read-only access via URI patterns:

Example resource access:

// Get a specific call
const call = await mcpClient.readResource("ai-usage://calls/550e8400-e29b-41d4-a716-446655440000");

// Get session details
const session = await mcpClient.readResource("ai-usage://sessions/session-abc-123");

// Get filtered metrics
const metrics = await mcpClient.readResource(
  "ai-usage://metrics/aggregate?project=my-chatbot&environment=prod"
);

Data Model

ModelCallLog

interface ModelCallLog {
  id: string;                    // Unique identifier (UUID)
  timestamp: string;             // ISO timestamp
  project: string;               // Project identifier
  environment: string;           // "dev" | "staging" | "prod" | custom
  userId?: string;               // Optional user identifier
  sessionId?: string;            // Optional session identifier
  modelName: string;             // Model name
  modelVersion?: string;         // Model version
  promptType?: string;           // "chat" | "rag" | "tool" | "agent" | custom
  inputMessages: Message[];      // Input messages
  outputMessages: OutputMessage[]; // Output messages
  retrievedContext?: RetrievedContext[]; // RAG context
  latencyMs?: number;            // Latency in milliseconds
  tokensIn?: number;             // Input tokens
  tokensOut?: number;            // Output tokens
  safety?: SafetyResult;         // Safety check result
  metrics?: Record<string, number | string>; // Custom metrics
  traceId?: string;              // Distributed tracing ID
  requestId?: string;            // Provider request ID
}

SessionSummary

interface SessionSummary {
  sessionId: string;
  project: string;
  environment: string;
  firstCallAt: string;           // ISO timestamp
  lastCallAt: string;            // ISO timestamp
  callCount: number;
  totalTokensIn: number;
  totalTokensOut: number;
  avgLatencyMs?: number;
}

Message Types

interface Message {
  role: "system" | "user" | "assistant" | "tool";
  content: string;
}

interface OutputMessage {
  role: "assistant" | "tool";
  content: string;
}

interface RetrievedContext {
  source: string;
  docId: string;
  hash?: string;
}

interface SafetyResult {
  status: "passed" | "flagged" | "blocked";
  details?: string;
}

Architecture

┌─────────────────────────────────────────────────────────────┐
│                      MCP Server                             │
│  ┌─────────────────────────────────────────────────────┐   │
│  │                    src/index.ts                      │   │
│  │              Server wiring & handlers                │   │
│  └─────────────────────────────────────────────────────┘   │
│                            │                                │
│         ┌──────────────────┼──────────────────┐            │
│         ▼                  ▼                  ▼            │
│  ┌─────────────┐   ┌─────────────┐   ┌─────────────┐      │
│  │   Tools     │   │  Resources  │   │   Schema    │      │
│  │ src/tools/* │   │src/resources│   │ src/schema  │      │
│  └─────────────┘   └─────────────┘   └─────────────┘      │
│         │                  │                  │            │
│         └──────────────────┼──────────────────┘            │
│                            ▼                                │
│              ┌─────────────────────────┐                   │
│              │    MetricsStore         │                   │
│              │    Interface            │                   │
│              │    src/store.ts         │                   │
│              └─────────────────────────┘                   │
│                            │                                │
│              ┌─────────────┴─────────────┐                 │
│              ▼                           ▼                 │
│    ┌──────────────────┐      ┌──────────────────┐         │
│    │ InMemoryStore    │      │ PostgresStore    │         │
│    │ (included)       │      │ (extend)         │         │
│    └──────────────────┘      └──────────────────┘         │
└─────────────────────────────────────────────────────────────┘

Project Structure

ai-usage-metrics-mcp/
├── src/
│   ├── index.ts           # MCP server entry point
│   ├── schema.ts          # TypeScript type definitions
│   ├── store.ts           # Storage interface & in-memory implementation
│   ├── tools/
│   │   ├── index.ts       # Tool exports
│   │   ├── log-model-call.ts
│   │   ├── search-model-calls.ts
│   │   ├── list-sessions.ts
│   │   ├── get-session-calls.ts
│   │   └── get-aggregate-metrics.ts
│   └── resources/
│       └── index.ts       # Resource handlers
├── tests/
│   ├── fixtures.ts        # Test data factories
│   ├── store.test.ts
│   ├── resources.test.ts
│   ├── integration.test.ts
│   └── tools/
│       └── *.test.ts
├── package.json
├── tsconfig.json
└── vitest.config.ts

Extending the Server

Adding PostgreSQL Support

The MetricsStore interface makes it straightforward to add database support:

// src/postgres-store.ts
import { Pool } from 'pg';
import { MetricsStore, ModelCallLog, SessionSummary, ... } from './schema.js';

export class PostgresMetricsStore implements MetricsStore {
  private pool: Pool;

  constructor(connectionString: string) {
    this.pool = new Pool({ connectionString });
  }

  async logCall(input: LogCallInput): Promise<ModelCallLog> {
    const id = crypto.randomUUID();
    const timestamp = new Date().toISOString();

    await this.pool.query(
      `INSERT INTO model_calls (id, timestamp, project, ...) VALUES ($1, $2, $3, ...)`,
      [id, timestamp, input.project, ...]
    );

    return { id, timestamp, ...input };
  }

  async getCall(id: string): Promise<ModelCallLog | null> {
    const result = await this.pool.query(
      'SELECT * FROM model_calls WHERE id = $1',
      [id]
    );
    return result.rows[0] || null;
  }

  // Implement remaining methods...
}

Database Schema (PostgreSQL)

CREATE TABLE model_calls (
  id UUID PRIMARY KEY,
  timestamp TIMESTAMPTZ NOT NULL,
  project VARCHAR(255) NOT NULL,
  environment VARCHAR(50) NOT NULL,
  user_id VARCHAR(255),
  session_id VARCHAR(255),
  model_name VARCHAR(255) NOT NULL,
  model_version VARCHAR(50),
  prompt_type VARCHAR(50),
  input_messages JSONB NOT NULL,
  output_messages JSONB NOT NULL,
  retrieved_context JSONB,
  latency_ms INTEGER,
  tokens_in INTEGER,
  tokens_out INTEGER,
  safety JSONB,
  metrics JSONB,
  trace_id VARCHAR(255),
  request_id VARCHAR(255),
  created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Indexes for common queries
CREATE INDEX idx_model_calls_project ON model_calls(project);
CREATE INDEX idx_model_calls_environment ON model_calls(environment);
CREATE INDEX idx_model_calls_session_id ON model_calls(session_id);
CREATE INDEX idx_model_calls_user_id ON model_calls(user_id);
CREATE INDEX idx_model_calls_timestamp ON model_calls(timestamp);
CREATE INDEX idx_model_calls_model_name ON model_calls(model_name);

Adding Custom Metrics

Log custom metrics with any model call:

await mcpClient.callTool("log_model_call", {
  project: "my-app",
  modelName: "gpt-4",
  inputMessages: [...],
  outputMessages: [...],
  metrics: {
    // Custom numeric metrics
    confidence_score: 0.95,
    relevance_score: 0.87,
    response_quality: 4.5,

    // Custom string metrics
    intent_category: "information_query",
    sentiment: "neutral",
    language: "en"
  }
});

Development

Prerequisites

• Node.js 18+
• npm or pnpm

Setup

# Install dependencies
npm install

# Build
npm run build

# Run in development mode (watch)
npm run dev

Code Style

The project uses TypeScript with strict mode enabled. Key conventions:

• All types defined in src/schema.ts
• Tool implementations in separate files under src/tools/
• Input validation using Zod schemas
• Async/await for all asynchronous operations

Testing

The project includes a comprehensive test suite with 220+ tests:

# Run all tests
npm test

# Run tests in watch mode
npm run test:watch

# Run with coverage report
npm run test:coverage

Test Coverage

Test Structure

• Unit Tests: Individual components (store, tools, resources)
• Integration Tests: End-to-end workflows
• Edge Cases: Error handling, boundary conditions, concurrent operations

Troubleshooting

Common Issues

Server not connecting:

• Verify the path in your MCP client configuration is absolute
• Check that the server is built (npm run build)
• Ensure Node.js 18+ is installed

Data not persisting:

• The default in-memory store loses data on restart
• Implement a database-backed store for persistence

High memory usage:

• The in-memory store grows unbounded
• Implement pagination or data expiration for production use

Debug Mode

Enable debug logging by setting the DEBUG environment variable:

DEBUG=mcp:* node dist/index.js

Platform-Specific Issues

License

MIT License - see LICENSE file for details.

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request

Support

• Issues: Report bugs and request features via GitHub Issues
• Documentation: See the MCP specification
• Discussions: Join the conversation on GitHub Discussions