Project Sentry

An intelligent MCP (Model-Context-Protocol) server for QA automation that bridges AI agents with backend microservices for seamless end-to-end testing.

Overview

Project Sentry transforms your Postman collections into a powerful MCP server that enables AI agents to perform complex, stateful API testing using natural language. It handles authentication, state management, and provides a clean interface for test execution and reporting.

Features

• Autonomous Authentication: Automatic username/password authentication with Basic auth and retry logic
• Stateful Testing: Session-based state store for multi-step test scenarios
• Postman Integration: Convert existing Postman collections into MCP tools
• AI-Friendly: Designed to reduce cognitive load on AI agents
• Transparent Error Handling: Structured error responses with full downstream API details (v1.3)
• Environment Configuration: Programmatic access to valid test data and configuration (v1.3)
• Comprehensive Reporting: Generate structured test reports in Markdown format

Quick Start

1. Installation

# Clone the repository
git clone <repository-url>
cd postman-replacement

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install fastmcp fastapi uvicorn pydantic pydantic-settings requests typer python-dotenv

2. Configuration

Copy the example environment file and configure your settings:

cp .env.example .env

Edit .env with your actual values:

# Authentication Configuration
AUTH_BASE_URL=https://steven01-auth.cp.manh.cloud
AUTH_BASIC_TOKEN=b21uaWNvbXBvbmVudC4xLjAuMDpiNHM4cmdUeWc1NVhZTnVu
[email protected]
PASSWORD=Password@1

# Target API Configuration
API_BASE_URL=https://omni-steven01.cp.manh.cloud
API_VERSION=v1

# Server Configuration
HOST=localhost
PORT=8000
LOG_LEVEL=INFO

3. Ingest Postman Collection

Convert your Postman collection into MCP tools:

sentry-ingest Orders.postman_collection.json

This will generate Python tool files in sentry/generated_tools/.

4. Start the Server

source venv/bin/activate
python -m sentry

The server will start on http://localhost:8000 and be ready for MCP client connections.

5. Connect MCP Clients

Cursor IDE Configuration

Add this to your Cursor settings (.cursor-settings/settings.json):

{
  "mcp": {
    "servers": {
      "project-sentry": {
        "command": "python",
        "args": ["-m", "sentry"],
        "cwd": "/Users/nmorrisroe/Desktop/postman-replacement",
        "env": {
          "PATH": "/Users/nmorrisroe/Desktop/postman-replacement/venv/bin:${PATH}"
        }
      }
    }
  }
}

Windsurf IDE Configuration

Add this to your Windsurf MCP settings (.windsurf/mcp_servers.json):

{
  "mcpServers": {
    "project-sentry": {
      "command": "python",
      "args": ["-m", "sentry"],
      "cwd": "/Users/nmorrisroe/Desktop/postman-replacement",
      "env": {
        "PATH": "/Users/nmorrisroe/Desktop/postman-replacement/venv/bin:${PATH}"
      }
    }
  }
}

Generic MCP Client Configuration

For other MCP-compatible clients, use these connection details:

{
  "name": "project-sentry",
  "description": "QA automation server for API testing",
  "transport": {
    "type": "stdio",
    "command": "python",
    "args": ["-m", "sentry"],
    "cwd": "/Users/nmorrisroe/Desktop/postman-replacement",
    "env": {
      "PATH": "/Users/nmorrisroe/Desktop/postman-replacement/venv/bin:${PATH}"
    }
  }
}

Usage

Available Tools

Static Tools

• state_setValue(session_id, key, value) - Store data in session state
• state_getValue(session_id, key) - Retrieve data from session state
• state_clear(session_id) - Clear all session data
• generate_test_report(log, test_name, session_id) - Generate test reports
• search_api_documentation(query) - Search available API tools
• context_get_environment_config() - Get environment configuration (v1.3)
• context_get_valid_skus(category) - Get valid test SKUs (v1.3)
• context_get_default_customer() - Get default customer data (v1.3)

Generated Tools (37 Total)

Tools automatically generated from your Orders.postman_collection.json:

Order Management:

• create_order_1_item - Create single item order
• create_order_2_item - Create multi-item order
• confirm_order - Confirm order details
• allocate_order - Allocate inventory
• ship_order - Ship completed order
• cancel_order_line - Cancel specific order lines

Payment Processing:

• create_paymentheader1_valid_cc - Create valid credit card payment
• create_paymentheader1_fraud_cc - Test fraud detection
• trigger_payment_processing - Process payments
• fetch_paymentheader - Retrieve payment details

Fulfillment & Config:

• fetch_allocation_config - Get allocation settings
• release_order - Release order for fulfillment
• short_order - Handle short shipments
• fetch_release_config - Get release configuration

And 27 more tools covering your complete API surface.

Unified Tool Registry and Discovery

Project Sentry now centralizes endpoint definitions in sentry/presets/endpoints.json and dynamically registers both:

• Preset tools: one per entry (backward compatible names)
• Canonical tools: one per unique (method, path) named like invoke_{method}_{path}

Key components:

• sentry/core/tool_registry.py: Loads rich metadata (description, tags, optional schemas/examples) and exposes helpers:
  • list_presets() and list_canonicals()
  • describe(name) for presets or canonical tools
• Static discovery tools:
  • list_endpoints(kind="all|presets|canonicals")
  • describe_endpoint(name)
  • search_api_documentation(query)

Authoring presets:

• Minimal fields supported: method, path, default_payload, description, tags
• Optional rich context fields (all optional): params_schema, body_schema, headers, examples, usage, notes

Runtime:

• sentry/__main__.py loads presets, registers tools, and records registrations in the registry for discoverability.
• Use describe_endpoint to see how to use a tool correctly, including presets that share the same canonical endpoint.

What's New in v2.0

🚀 Dynamic Configuration Building

Project Sentry now builds config.json from real tenant API data instead of using static test data:

New CLI Command:

# Build config from tenant APIs
python -m sentry build-config

# Preview what would be built (dry run)
python -m sentry build-config --dry-run

# Save to custom location
python -m sentry build-config --output /path/to/config.json

Automatic Config Refresh:

• Server automatically rebuilds config on startup
• Fails fast if tenant APIs are unreachable
• Ensures configuration is always current with your tenant

What Gets Fetched from APIs:

• Enterprise codes from Organization API
• Supported currencies from Organization configuration
• Real SKUs from Item Search API
• Order types and document types from Order Config API

What Remains Static:

• Test credit card numbers for payment testing
• Default customer information (customizable)
• Environment metadata

🔥 Transparent Error Propagation (v1.3)

API errors are no longer hidden! When a tool makes an API call that results in a non-2xx status code, you now receive:

{
  "success": false,
  "error": {
    "status_code": 400,
    "body": {
      "errors": [{
        "field": "currency",
        "message": "Invalid currency code 'EUR'. Must be 'USD' for this enterprise."
      }]
    },
    "headers": {...}
  }
}

🎯 Environment Configuration (v1.3)

Static tools provide programmatic access to configuration data:

• context_get_environment_config() - Get all environment configuration
• context_get_valid_skus() - Get valid test product SKUs
• context_get_default_customer() - Get default customer information

Example Test Flow

Here's how an AI agent would use Project Sentry for end-to-end testing:

1. Start with session: "test-session-123"
2. Call: create_order_1_item() -> Store OrderId in state
3. Call: state_setValue(session_id="test-session-123", key="order_id", value="ORD-456")
4. Call: confirm_order() using stored OrderId
5. Call: allocate_order() -> Update order status
6. Call: ship_order() -> Complete fulfillment
7. Call: generate_test_report() -> Create markdown report

The AI agent can chain these API calls naturally using stored state, without needing to manage authentication or remember IDs between calls.

Architecture

Core Components

• AuthManager: Thread-safe singleton for username/password authentication with Basic auth
• StateStore: Session-aware in-memory state management
• RequestWrapper: HTTP client with automatic retry on 401 errors
• Tool Registry: Dynamic discovery and registration of API tools

Project Structure

project-sentry/
├── .env.example              # Environment configuration template
├── pyproject.toml            # Project dependencies and metadata
├── README.md                 # This file
└── sentry/
    ├── __main__.py           # Server entry point
    ├── core/                 # Core framework modules
    │   ├── auth.py           # Authentication management
    │   ├── config.py         # Configuration loading
    │   ├── state.py          # State management
    │   └── request_wrapper.py # HTTP client with retry logic
    ├── generated_tools/      # Auto-generated API tools
    ├── static_tools/         # Hand-crafted utility tools
    │   ├── state_tools.py    # State management tools
    │   ├── reporting.py      # Test reporting tools
    │   └── search_tools.py   # API discovery tools
    └── ingest.py             # Postman collection ingestion CLI

Development

Adding Custom Tools

Create new tools in sentry/static_tools/ following this pattern:

from pydantic import BaseModel, Field

class MyToolParams(BaseModel):
    param1: str = Field(..., description="Description of parameter")

def my_custom_tool(params: MyToolParams) -> dict:
    """Tool description for AI agents."""
    # Implementation here
    return {"success": True, "result": "data"}

Testing

# Install development dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Format code
black sentry/
isort sentry/

Configuration Files

Environment Variables (.env)

Variable	Description	Example
AUTH_BASE_URL	Authentication server base URL	https://auth.example.com
CLIENT_ID	OAuth2 client identifier	your-client-id
CLIENT_SECRET	OAuth2 client secret	your-secret
TENANT_ID	Tenant identifier for multi-tenant auth	tenant-123
API_BASE_URL	Target API server base URL	https://api.example.com
API_VERSION	API version to use	v1
HOST	Server bind address	localhost
PORT	Server port	8000
LOG_LEVEL	Logging verbosity	INFO

Environment Configuration (config.json)

The config.json file is now automatically generated from your tenant's live API data:

{
  "environment": {
    "name": "Live Tenant Environment",
    "description": "Configuration built from real tenant API data",
    "version": "2.0",
    "built_at": "2025-09-09T20:43:10.285Z"
  },
  "enterprise_codes": {
    "default": "YOUR-TENANT-CODE",
    "alternatives": ["ALT-CODE-1", "ALT-CODE-2"]
  },
  "currencies": {
    "default": "USD",
    "supported": ["USD", "EUR", "GBP", "CAD"]
  },
  "skus": {
    "general": [
      {
        "id": "REAL-SKU-001",
        "name": "Real Product from Your Catalog",
        "category": "general",
        "price": 99.99,
        "description": "Actual product from your tenant"
      }
    ]
  }
}

Configuration Sources:

• Enterprise codes: Fetched from /cfg/api/configDirector/export
• Currencies: Retrieved from /api/organization/organization/search
• SKUs: Pulled from /api/search/item with real product data
• Order types: Loaded from /api/order/orderConfig

This configuration is accessible via the context_get_environment_config() tool and is automatically refreshed on server startup.

Troubleshooting

Authentication Issues

• Verify your credentials in .env
• Check that the auth server is accessible
• Review server logs for detailed error messages

Tool Generation Issues

• Ensure your Postman collection is valid JSON
• Check that request bodies contain valid JSON examples
• Review generated files in sentry/generated_tools/

Server Startup Issues

• Verify all dependencies are installed
• Check that the configured port is available
• Review the startup logs for specific errors

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

License

[Add your license information here]