AI Test Automation Server

Enterprise-Grade, AI-Powered Multi-Agent Test Automation Framework
Built with Playwright · Appium · TypeScript · OpenAI · MCP · LangSmith · Redis · Gatling

CI: API Tests Playwright TypeScript Appium OpenAI LangSmith License: MIT

---

🚀 Overview

An enterprise-grade, AI-driven multi-agent test automation platform that:

• 📋 Reads Jira stories and parses Acceptance Criteria automatically
• 📝 Generates BRD documents using AI-powered analysis
• ⚡ Auto-generates Playwright TypeScript test scripts (Web, Mobile, GraphQL & REST API)
• 🔧 Self-heals broken test locators using AI confidence scoring
• 💰 Saves 100% LLM token costs via SHA-256 distributed Redis caching
• 📊 Streams AI telemetry to LangSmith for real-time observability
• 🌐 Executes cross-platform tests on Local, BrowserStack, AWS Device Farm

---

🏗️ High-Level Architecture

┌─────────────────────────────────────────────────────────────────┐
│                  AI MULTI-AGENT PIPELINE                         │
│                                                                   │
│  Jira Story ──► PlannerAgent ──► GeneratorAgent ──► HealerAgent  │
│     (AC)          (BRD .md)      (.spec.ts files)  (Auto-Heal)   │
│                                                                   │
│  Distributed Cache: In-Memory ──► Disk ──► Redis                 │
│  Observability: LangSmith APM (Token Telemetry + Evals)          │
│  Orchestration: MCP Server (Claude/Cursor IDE Integration)        │
└─────────────────────────────────────────────────────────────────┘

---

🛠️ Tech Stack

Layer	Technology
Web Automation	Playwright 1.49+ (TypeScript)
Mobile Automation	Appium 2.0 + WebdriverIO 9 (TypeScript)
API / GraphQL Testing	Playwright APIRequestContext (Native)
Load Testing	Gatling TypeScript Framework
AI Model Integration	OpenAI GPT-4o / Google Gemini
AI Observability	LangSmith (Token Telemetry + LLM-as-a-Judge Evals)
LLM Cache Engine	SHA-256 In-Memory + Disk + Redis (Distributed)
Protocol Standard	Anthropic MCP (Model Context Protocol)
CI/CD Pipeline	GitLab CI (Shift-Left AI Orchestration)
Cloud Grids	BrowserStack Automate + AWS Device Farm
Reporting	Monocart HTML Reporter (Videos + Traces)

---

📁 Project Structure

ai-test-automation-platform/
├── agents/
│   ├── pipeline.ts              # 🎯 Central Orchestration Layer (5-Step Pipeline)
│   ├── jira/
│   │   ├── JiraClient.ts        # Jira REST API integration
│   │   └── StoryParser.ts       # ADF → Structured JSON parser
│   ├── planner/
│   │   └── PlannerAgent.ts      # AI-powered BRD generator (GPT-4o/Gemini)
│   ├── generator/
│   │   └── GeneratorAgent.ts    # Playwright TypeScript test generator
│   ├── healer/
│   │   ├── HealerAgent.ts       # JSON results parser & failure classifier
│   │   └── LocatorHealer.ts     # Self-healing engine (data-testid/role/text)
│   └── utils/
│       └── LlmCache.ts          # SHA-256 Two-Tier LLM Cache (Disk + Redis)
├── mcp-server/
│   └── server.ts                # MCP Protocol Server (Claude/Cursor integration)
├── tests/
│   ├── ui/                      # Playwright Web E2E test suites
│   ├── mobile/                  # Appium WebdriverIO mobile specs
│   └── api/
│       ├── graphql.spec.ts      # Native GraphQL query & mutation tests
│       └── location.spec.ts     # REST API test suite
├── load-tests/                  # Gatling TypeScript performance suite
├── screens/                     # Mobile Page Object Models (POM)
├── config/                      # Environment & capability configurations
└── .gitlab-ci.yml               # Shift-Left AI CI/CD pipeline

---

⚡ Quick Start

Prerequisites

• Node.js 20+
• npm 10+
• Appium Server (for mobile tests)

Installation

git clone https://github.com/abhirise1981/ai-test-automation-platform.git
cd ai-test-automation-platform
npm install
npx playwright install

Environment Setup

cp .env.example .env
# Fill in your API keys in .env

Running Tests

# Run full AI pipeline for a Jira story
npx ts-node agents/pipeline.ts --issue PROJ-123

# Run on BrowserStack Cloud
npx ts-node agents/pipeline.ts --issue PROJ-123 --target browserstack

# Run Playwright Web tests only
npm test

# Run Mobile tests (Android)
npm run test:mobile:android

# Run GraphQL API tests
npx playwright test tests/api/graphql.spec.ts

# Start MCP Server (for Claude/Cursor integration)
npm run mcp:start

# Run Gatling load tests
npx gatling run

---

🤖 AI Agents

1. PlannerAgent

• Reads Jira User Story + Acceptance Criteria
• Calls OpenAI GPT-4o to identify Happy Path, Negative, and Edge Case scenarios
• Outputs structured BRD Markdown document

2. GeneratorAgent

• Reads BRD and generates Playwright TypeScript test scripts
• Supports Web (Playwright), Mobile (Appium), and API (GraphQL/REST)

3. HealerAgent + LocatorHealer

• Parses test-results/results.json for failed tests
• Classifies failures: LOCATOR_BROKEN | TIMING_FLAKY | ASSERTION_DRIFT | APP_CRASH
• Auto-patches broken locators with 85%+ confidence threshold

---

💰 LLM Cost Optimization

Our LlmCache.ts implements a 3-Tier Caching Architecture:

Request → SHA-256 Hash → In-Memory (0ms) → Disk (2ms) → Redis (5ms) → LLM API

• Cache HIT (In-Memory): 0ms, 100% token savings
• Cache HIT (Disk): 2ms, 100% token savings
• Cache HIT (Redis): 5ms cross-runner shared cache for CI/CD
• Cache MISS: Full LLM call, result cached for all future runs

---

📊 LangSmith Observability

Set LANGCHAIN_TRACING_V2=true in .env to enable:

• Real-time agent execution traces
• Token count & cost monitoring per run
• LLM-as-a-Judge evaluation scores
• Prompt-level debugging in LangSmith Playground

---

📱 Mobile Testing (Appium 2.0)

• Android: UiAutomator2 driver with W3C Pointer Actions
• iOS: XCUITest driver with accessibility-id locator strategy
• Zero Java: 100% TypeScript with WebdriverIO 9
• Cloud: BrowserStack App Automate & AWS Device Farm support

---

🔗 MCP Server Integration

This framework exposes all AI agents as Model Context Protocol (MCP) tools compatible with:

• Claude Desktop
• Cursor IDE
• VS Code (Copilot)
• Any MCP-compatible AI assistant

npm run mcp:start
# → Registers: fetch_jira_story, generate_brd, generate_tests, run_tests, heal_tests

---

👨‍💻 Author

Abhishek Kumar
Senior AI Test Automation Architect
GitHub

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.