Agent Eval

A dependency-light evaluation platform for RAG/wiki-quality AI agents. It scores agent outputs for faithfulness, retrieval relevance, hallucination risk, latency, and cost; produces CI-friendly quality gates; emits regression/canary reports; and exposes the workflow through a lightweight MCP-style stdio tool server.

What Is Included

• JSONL evaluation case format for RAG/wiki workflows.
• Deterministic checks for:
  • faithfulness to retrieved context and reference answer,
  • retrieval relevance against question and expected keywords,
  • hallucination risk from unsupported answer content,
  • latency and cost thresholds.
• 100+ synthetic case generator.
• CI/CD-style suite-level and case-level quality gates.
• Markdown and JSON evaluation reports.
• Regression report comparing baseline and candidate runs.
• Canary promotion policy with traffic ramp decisions.
• OpenTelemetry-compatible JSONL traces/metrics.
• MCP-style stdio server exposing evaluation tools.

Quick Start

git clone https://github.com/ad-github1/ENTERPRISE-AI-AGENT-EVALUATION-PLATFORM.git
cd ENTERPRISE-AI-AGENT-EVALUATION-PLATFORM
PYTHONPATH=src python3 -m agent_eval_platform generate-cases --count 120 --out examples/wiki_eval_cases.jsonl
PYTHONPATH=src python3 -m agent_eval_platform evaluate \
  --cases examples/wiki_eval_cases.jsonl \
  --gate examples/quality_gate.json \
  --variant candidate \
  --json-out reports/eval_result.json \
  --markdown-out reports/eval_report.md \
  --traces-out reports/traces.jsonl
PYTHONPATH=src python3 -m unittest discover -s tests

Testing

Run the test suite:

PYTHONPATH=src python3 -m unittest discover -s tests

Current local result:

Ran 5 tests in 0.015s

OK

After installation:

pip install -e .
agent-eval evaluate --cases examples/wiki_eval_cases.jsonl --gate examples/quality_gate.json
agent-eval-mcp

Case Format

Each JSONL row contains one evaluated agent run:

{
  "case_id": "case-0001",
  "question": "What contribution is Ada Lovelace known for in mathematics?",
  "reference_answer": "Ada Lovelace is known for Analytical Engine notes.",
  "expected_keywords": ["Ada Lovelace", "Analytical Engine"],
  "retrieved_docs": [
    {"doc_id": "wiki-1", "title": "Ada Lovelace", "text": "...", "score": 0.94}
  ],
  "agent_answer": "Ada Lovelace is known for Analytical Engine notes.",
  "latency_ms": 240.5,
  "cost_usd": 0.0031,
  "tags": ["wiki", "rag"]
}

CI Quality Gate

The evaluator exits non-zero when --fail-on-gate is used and thresholds fail:

PYTHONPATH=src python3 -m agent_eval_platform evaluate \
  --cases examples/wiki_eval_cases.jsonl \
  --gate examples/quality_gate.json \
  --fail-on-gate

See .github/workflows/agent-eval.yml for a GitHub Actions example.

MCP-Style Tool Server

Run:

PYTHONPATH=src python3 -m agent_eval_platform.mcp_server

Supported JSON-RPC methods:

• initialize
• tools/list
• tools/call with:
  • run_evaluation_suite
  • compare_regression
  • decide_canary

This is intentionally stdio and dependency-free. It follows the MCP tool shape closely enough for local agent integration demos without requiring the MCP Python SDK.

Canary Workflow

PYTHONPATH=src python3 -m agent_eval_platform canary \
  --result reports/eval_result.json \
  --config examples/canary_config.json \
  --json-out reports/canary_decision.json

The decision is hold, increase_traffic, or promote based on suite quality and minimum case coverage.

Evaluation Results

Evaluation Run

PYTHONPATH=src python3 -m agent_eval_platform evaluate \
  --cases examples/wiki_eval_cases.jsonl \
  --gate examples/quality_gate.json \
  --variant candidate \
  --json-out reports/eval_result.json \
  --markdown-out reports/eval_report.md \
  --traces-out reports/traces.jsonl

Aggregate Metrics

Metric	Value
Evaluation cases	120
Pass rate	82.5%
Average faithfulness	0.825
Average retrieval relevance	0.838
Average hallucination risk	0.153
p50 latency	392.58 ms
p95 latency	663.40 ms
p99 latency	872.87 ms
Average cost	$0.00393
Total cost	$0.47158

Canary Decision

PYTHONPATH=src python3 -m agent_eval_platform canary \
  --result reports/eval_result.json \
  --config examples/canary_config.json \
  --json-out reports/canary_decision.json

Result:

{
  "action": "hold",
  "next_traffic_percent": 10.0,
  "reasons": [
    "pass_rate 0.825 < 0.900"
  ]
}

The canary policy correctly blocked promotion because the candidate run did not meet the configured 90% pass-rate threshold. This demonstrates how the platform can prevent low-quality agent versions from being promoted automatically.

Observability

The evaluation emits OpenTelemetry-style JSONL traces to:

reports/traces.jsonl

Each case generates spans for faithfulness, retrieval relevance, hallucination risk, and final case-level pass/fail status, enabling debugging of failed agent responses.