AgentLens Server

---

Why AgentLens?

Everyone is shipping agents now. Almost nobody knows what their agents actually do in production: which tools they call, what the LLM sees, how much it costs, and — crucially — whether yesterday's prompt change made the agent worse.

AgentLens brings the discipline of classical observability (traces, spans, cost accounting) and classical QA (regression suites, semantic scoring) to the agent world, in a single lightweight package with no infrastructure requirements.

Pain point	AgentLens answer
"Which tool call blew up my agent's latency?"	Auto-nested async span traces via @agent, @tool, @llm decorators
"Is this prompt actually better than v2.3.1?"	Prompt regression suites with keyword + semantic checks, CI-gated
"How do I know the agent did a good job?"	LLM-as-judge evaluations with configurable rubrics
"What did that run cost me?"	Per-span token counting (tiktoken) and cost estimation
"I want my coding agent to look at run history"	Built-in MCP server, pluggable into Claude Code / Cursor

Quick start

pip install agentlens

# 1. Start the server + dashboard
lens serve            # → http://localhost:3368

# 2. Run the traced demo agent
lens demo

# 3. Run evaluations and regression suites
lens regression init my_suite.yml
lens regression run my_suite.yml

Instrument your agent (30 seconds)

from agentlens.sdk import agent, tool, llm
from agentlens.evals import evaluate
from agentlens.regression import load_suite, run_suite

@llm(model="gpt-4o-mini")
async def answer(question: str) -> str:
    # your LLM call here — tokens and cost are tracked automatically
    ...

@tool("search_index")
async def search_index(query: str) -> list[str]:
    ...

@agent("rag_agent")
async def rag_agent(question: str) -> str:
    ctx = await search_index(question)
    return await answer(f"Context: {ctx}\nQuestion: {question}")

# Span tree, parent-child nesting and cost — all automatic.
result = await rag_agent("How do I reset my password?")

Features

🧭 Tracing SDK

Drop-in decorators (@agent, @tool, @llm, @retriever) that build a full span tree with automatic parent-child nesting through contextvars. Works for both async and synchronous functions. Each span records input/output, model, token counts, estimated cost (using real per-model pricing for OpenAI and Anthropic models, with an extensible registry for your own models) and status. Spans are flushed to the AgentLens API through a pluggable callback, so you can persist them, export them to OTLP, or mock them in tests.

🧑‍⚖️ LLM-as-judge evaluations

Score agent outputs against built-in criteria (CORRECTNESS, RELEVANCE, COHERENCE, SAFETY, HALLUCINATION, COMPLETENESS) or define your own rubric. The judge is any OpenAI-compatible endpoint — OpenAI, Anthropic gateways, Ollama, LiteLLM — configured with two environment variables. Every case x criterion pair produces a numeric score, a human-readable reason, and a pass/fail verdict, aggregated into an evaluation report.

🛡️ Prompt regression suites

YAML-defined test suites, the way prompt engineers have been wishing for:

name: support-agent
agent: support_agent
model: gpt-4o-mini
version: "1.2.0"

cases:
  - name: refund question
    input: "When will I receive my refund?"
    expected_contains: ["refund", "days"]
    expected_not_contains: ["deny"]
    min_score: 0.8
    tags: ["billing"]

lens regression run suite.yml executes every case, applies keyword checks plus optional semantic scoring, and exits non-zero on failure — perfect for CI. Compare versions side by side in the dashboard.

💰 Cost tracking

Token counting via tiktoken (exact for OpenAI encoders, character-ratio fallback for unknown models), per-model pricing registry, and aggregated cost summaries across runs. Know exactly what your agent fleet spends.

🤖 Native MCP server

AgentLens ships as a proper MCP server (mcp server), exposing four tools:

Tool	Purpose
lens_search_runs	Find recent agent runs, filter by project
lens_run_summary	Full summary + span tree of a run
lens_eval_latest	Latest LLM-as-judge evaluation results
lens_regression_status	Latest regression report

Wire it into your favorite coding agent and let it debug your production agents for you:

// ~/.config/claude/claude_desktop_config.json
{
  "mcpServers": {
    "agentlens": {
      "command": "lens",
      "args": ["mcp-server"]
    }
  }
}

📊 REST API + dashboard

A FastAPI application (/v1/spans, /v1/runs, /v1/evals, /v1/regressions, /v1/summary) with OpenAPI docs at /docs, plus a dark-themed embedded dashboard with run tables, interactive span trees, evaluation results, and regression history. SQLite zero-config by default; flip to Postgres with LENS_STORAGE_BACKEND=postgres.

Architecture

┌────────────┐   decorators   ┌──────────────┐   flush   ┌──────────────────┐
│  Your code │ ─────────────► │ Tracing SDK  │ ───────►  │  AgentLens API   │
└────────────┘                └──────────────┘           │  (FastAPI)       │
┌────────────┐   run      ┌──────────────────┐           │                  │
│  YAML      │ ─────────► │ Regression runner│ ──┐       │  ┌─────────────┐ │
│  suites    │            │ (+ keyword/sem.) │   │       │  │ Dashboard / │ │
└────────────┘            └──────────────────┘   │       │  │ MCP server  │ │
┌────────────┐   evaluate ┌──────────────────┐   │       │  └─────────────┘ │
│  Test cases│ ─────────► │ LLM-as-judge     │ ──┼──────►└────────┬─────────┘
└────────────┘            └──────────────────┘   │                │
                                                 │         ┌──────▼──────┐
                                                 └────────►│ SQLite /    │
                                                           │ Postgres    │
                                                           └─────────────┘

CLI reference

Command	Description
lens serve	Start API + dashboard (default port 3368)
lens mcp-server	Run the MCP server over stdio
lens regression run <file>	Run a regression suite (exits non-zero on failure)
lens regression init <file>	Scaffold a regression suite
lens demo	Run a traced demo agent, print the span tree

Configuration happens through LENS_ environment variables or a .env file (LENS_PORT, LENS_STORAGE_BACKEND, LENS_DATABASE_URL, LENS_JUDGE_MODEL, LENS_JUDGE_BASE_URL, LENS_JUDGE_API_KEY, ...). See agentlens/core/settings.py for the full list.

Docker

docker compose up --build
# → http://localhost:3368

Development

git clone https://github.com/Nexus-universe-space/agentlens.git
cd agentlens
pip install -e ".[dev]"

ruff check agentlens tests examples     # lint
ruff format agentlens tests examples    # format
pytest tests/unit tests/integration     # 41 tests
mypy agentlens                          # strict-ish typing

CI runs lint, tests (Python 3.10–3.13 with coverage), and typing on every push and PR.

Project layout

agentlens/
├── agentlens/
│   ├── api/        FastAPI REST API (spans, runs, evals, regressions, summary)
│   ├── cli/        Click CLI (serve, mcp-server, regression, demo)
│   ├── core/       Settings, Pydantic models, token/cost engine
│   ├── evals/      LLM-as-judge evaluation engine
│   ├── mcp/        MCP server with typed tools
│   ├── regression/ YAML suite loader + regression runner
│   ├── sdk/        Tracing decorators (agent / tool / llm / retriever)
│   ├── storage/    Async SQLAlchemy store (SQLite + Postgres)
│   └── web/        Embedded React dashboard (zero build step)
├── tests/          41 tests — unit + integration (SQLite, API, MCP)
├── examples/       Traced RAG agent + sample regression suite
├── Dockerfile
├── docker-compose.yml
└── .github/workflows/ci.yml

Roadmap

• OTLP trace exporter (Honeycomb, Grafana Tempo, Jaeger)
• Multi-agent session grouping and conversation views
• Real-time WebSocket updates in the dashboard
• Evaluation presets per domain (coding, RAG, customer support)
• Human-in-the-loop annotation of eval cases
• Prompt version diffing with blame attribution

Contributing

Contributions are very welcome! Please read CONTRIBUTING.md for the workflow, and review our Code of Conduct.

License

MIT — see LICENSE.