LS-SIEM-LLP/qa-probe

Find out exactly why your pages are blank - in under 5 minutes.

qa-probe is a Node.js command-line tool that reads your React frontend, maps every API call to its backend route, probes those endpoints live with real auth, and tells you the root cause when something is wrong - not just a status code.

Overall score: 74/100

Root causes detected:
  empty_db:              5 route(s)   seed the database
  feature_flag_disabled: 3 route(s)   enable HAS_BILLING, HAS_REPORTS
  contract_mismatch:     1 route(s)   frontend calls /api/users/, backend serves /users

Route          Score   Status     Root cause
-----------    -----   --------   ----------------------
/dashboard     100     healthy
/users          80     healthy    empty_db
/billing         0     broken     feature_flag_disabled
/reports        50     degraded   empty_db
/settings       85     healthy

It works best on React apps backed by FastAPI, Express, Next.js, GraphQL, or tRPC with an OpenAPI spec available. Built by the LightShield SIEM team and battle-tested against a production SIEM.

---

The problem it solves

Your dashboard loads. Nothing crashes. But every table is empty, every chart shows zero, and you have no idea why. The cause is almost always one of:

• The frontend calls /api/users/ but the backend route is /api/users (trailing slash)
• A feature flag (HAS_BILLING=false) silently disabled an entire router
• The database is connected but the table has no rows
• A backend refactor renamed a field and the frontend still reads the old name
• An endpoint is missing its auth guard, or leaks data without a token

qa-probe builds a source-aware map from frontend calls to live backend responses, then explains each failure in one line with a fix hint. It complements Playwright (user journeys) and Schemathesis (contract fuzzing) - it is the layer that answers "why is this page blank?" first.

---

Quick start

1. Install

npx qa-probe run        # use without installing
# or
npm install -g qa-probe

2. Create a config

qa-probe.config.js in your project root:

module.exports = {
  baseUrl: 'http://localhost:8000',      // your backend
  frontendSrc: './frontend/src',          // your React source
  routerFile: './frontend/src/App.tsx',   // your router file

  auth: {
    type: 'bearer',
    loginUrl: '/auth/login',
    credentials: {
      username: process.env.QA_USER,       // never hardcode credentials
      password: process.env.QA_PASS,
    },
    tokenPath: 'access_token',
  },
};

3. Run it

QA_USER=tester QA_PASS=secret npx qa-probe run

qa-probe will parse your frontend, fetch /openapi.json, probe every endpoint with real auth, and write a full report to .qaprobe/ (report.md, report.json, ai-context.md, report.html). A summary table prints to the terminal.

---

Understanding the report

Route status

Status	Meaning
healthy	endpoints respond, data present, schema valid
degraded	works but has an issue (empty data, slow response)
broken	unreachable, 404, 5xx, or a security/logic failure

Common root causes

Root cause	What happened	What to do
feature_flag_disabled	router not registered (a HAS_* flag is false)	enable the flag and restart
contract_mismatch	a similar route exists (slash/casing)	align the frontend path
missing_route	404, not in the OpenAPI spec	fix the typo or include_router()
empty_db	200 OK but empty array	seed the database
precondition_required	428 - terms/onboarding/MFA gate not satisfied	clear the gate for the probe account
auth_scope_mismatch	401/403 - wrong role/scope	use a user with the required permissions
schema_mismatch	a field was renamed/removed vs the spec	align frontend, backend, and spec
auth_bypass	endpoint returns 200 without auth	add the missing auth guard
assertion_failed	response violated a declared invariant	fix the response or the assertion
server_error	5xx	check backend logs

Every result is verifiable. Each diagnosis carries its evidence (the request, a snapshot of the response, timing) and a confidence level - high, medium, or none. A confidence: none / unknown result is explicitly not a confirmed pass.

Score: each route is 0-100; the overall score is the average. Use --fail-under 80 to make CI fail when the score drops.

---

Use cases

• "Why is this page blank?" - the everyday case. Get a root cause and a fix hint in one run instead of digging through network tabs and logs.
• CI gate - qa-probe run --fail-under 80 blocks a deploy when pages regress.
• Demo / staging sign-off - confirm an environment actually works before a customer call, not just that it returns 200s.
• Catch contract drift - a backend refactor renames a field or moves a route; qa-probe flags the exact endpoint and frontend call affected.
• Security smoke - find endpoints missing an auth guard or leaking PII (opt-in, read-only, see Security checks).
• Logic checks - assert response invariants (valid enum values, non-negative counts, required fields) without writing a test suite.
• AI-assisted debugging - let Claude, Cursor, or Codex query the QA state and explain failures in plain English (see below).

---

Use it with an AI assistant (Claude Code, Cursor, Codex)

qa-probe ships a Model Context Protocol (MCP) server. Point your AI assistant at it and the assistant can read your QA data and explain failures directly - no manual report-reading.

Setup

Run qa-probe once to populate the cache:

npx qa-probe run

Then add the MCP server. For Claude Code / Cursor, add to .mcp.json at your repo root:

{
  "mcpServers": {
    "qa-probe": {
      "command": "npx",
      "args": ["qa-probe", "mcp"]
    }
  }
}

Restart the assistant. Now you can ask in plain English:

• "Why is /reports showing no data?"
• "Which pages break if the users endpoint goes down?"
• "Show me all broken routes."
• "Mark the empty /alerts result as expected." (records feedback, reapplied next run)

Available MCP tools

Tool	Ask it
qa_probe_get_report	"Show me all broken routes and their scores"
qa_probe_explain_failure	"Why is /reports showing no data?"
qa_probe_probe_endpoint	"Is GET /users returning data right now?"
qa_probe_get_graph	"Which backend routes does /dashboard call?"
qa_probe_get_blast_radius	"What pages break if the users endpoint goes down?"
qa_probe_suggest_fix	"What should I do about feature_flag_disabled?"
qa_probe_run_analysis	"Run a full QA check and give me the summary"
qa_probe_label	"Mark this finding as expected / a real bug"

MCP output is sanitized - SQL errors, stack traces, and table names are redacted before they reach the assistant. Raw data stays on disk. Results carry their evidence and confidence, and explain_failure returns a plain-English trust note when a finding is unverified, so the assistant does not report guesses as passes.

You can also simply open the qa-probe folder (or your project with .qaprobe/) in an AI coding tool and ask it to read ai-context.md - a compact, LLM-oriented summary written on every run.

---

CLI commands

qa-probe run       [--config <path>] [--fail-under <N>]   analyze + probe + report
qa-probe analyze   [--config <path>]                       build the dependency graph
qa-probe probe     [--config <path>]                       probe all endpoints
qa-probe report    [--config <path>] [--fail-under <N>]   score, classify, write reports
qa-probe label <endpoint> <verdict> [-r <reason>]          record feedback (reapplied later)
qa-probe fix       [--config <path>] [--apply] [--pr]      generate remediation diffs
qa-probe mcp       [--config <path>]                       start the MCP server (stdio)

--fail-under <N> exits non-zero when the overall score is below N (a CI gate).

---

Capabilities

• Source-aware mapping - Babel parses your frontend; detects axios/fetch calls and common hook patterns; extracts routes from React Router (JSX and createBrowserRouter) and TanStack Router.
• Live probing with real auth - bearer, cookie, api-key, or none. Concurrency-limited, with a hard per-request deadline so a streaming endpoint can never hang the run.
• Root-cause classification - around 25 causes, each with a fix hint.
• ID chaining - probes collections first, harvests a real id, then probes detail routes with it instead of a guessed value (kills unseeded-DB noise). On by default.
• Adaptive baselines - learns each endpoint's normal latency and row counts from recent runs and flags deviation from itself. No model; fully inspectable.
• Feedback - qa-probe label (or the MCP tool) records a verdict that is reapplied on future runs. Scoped labels auto-revoke if behavior changes, so they cannot hide a regression.
• Security checks (opt-in) - anonymous auth-bypass re-probe, PII scan, and a per-persona privilege-escalation matrix. GET-only; never issues a write.
• Response assertions - declare invariants (in, type, gte, pattern, present, minItems, ...) and qa-probe verifies them on every 2xx response.
• Write-flows (opt-in, mutating) - full create/read/update/delete chains with guaranteed cleanup. Off by default; for a disposable / test-tenant environment only.
• SSE and WebSocket checks, schema-drift detection, blast radius, regression diff, coverage, and HAR import for dynamic frontends.
• OpenAPI auto-discovery - tries common spec paths and supports a local spec file, so fewer apps fall back to headless mode.

---

Configuration

The example above is enough to start. Key options (all optional unless noted):

module.exports = {
  baseUrl: 'http://localhost:8000',      // required
  frontendApiPrefix: '/api',              // stripped when matching calls to routes
  framework: 'fastapi',                   // fastapi | express | nextjs | graphql | trpc | generic
  openApiUrl: '/openapi.json',            // auto-discovers fallbacks if missing
  openApiFile: null,                      // or load the spec from a local file
  frontendSrc: './frontend/src',
  routerFile: './frontend/src/App.tsx',

  auth: { type: 'bearer', loginUrl: '/auth/login',
          credentials: { username: process.env.QA_USER, password: process.env.QA_PASS },
          tokenPath: 'access_token' },

  probe: {
    concurrency: 5,
    timeoutMs: 10000,
    idDiscovery: true,                    // ID chaining (on by default)
    safePosts: [],                        // POST endpoints that are safe reads
    pathParamValues: { id: '1' },         // fallback values for {id} etc.
    ignoreHTTPSErrors: false,             // dev/staging only
  },

  // Opt-in extras:
  security: { enabled: true, piiAllow: ['email'] },
  assertions: { 'GET /alerts': [{ field: 'items[].severity', in: ['low','high','critical'] }] },
  writeFlows: { enabled: false, flows: [] },
  analyze: { har: { enabled: false, harFile: './traffic.har' } },

  output: { dir: '.qaprobe', formats: ['json', 'markdown', 'ai-context', 'html'] },
  seedCommand: 'npm run db:seed',         // shown in the empty_db fix hint
};

qa-probe.config.js is executed as JavaScript (like ESLint or Jest configs). Never hardcode credentials - use environment variables.

---

How it works

Three phases, each cached so you can re-run them independently.

1. Analyze - parse the frontend AST, extract routes, fetch the OpenAPI spec (or import a HAR), and build a dependency graph with blast radius.
2. Probe - authenticate once, then probe every discovered endpoint live: HTTP, SSE, WebSocket, with hard deadlines and ID chaining.
3. Report - classify root causes, apply baselines and feedback, score routes 0-100, and write JSON / Markdown / ai-context / HTML reports.

---

vs other tools

Capability	qa-probe	Schemathesis	Playwright	Postman
Frontend to backend route mapping	yes	no	no	no
Root-cause labels with fix hints	yes	no	no	no
Live probe with real auth	yes	yes	yes	yes
Property-based contract fuzzing	no	yes	no	no
User-journey E2E	no	no	yes	no
MCP server for AI assistants	yes	no	no	no
CI exit-code gate	yes	yes	yes	yes

Use Schemathesis for thorough contract fuzzing and Playwright for user journeys. qa-probe maps and explains; reach for the others to go deeper on what it surfaces.

---

Security and safe use

• Credentials are held in memory for the run and never written to any report.
• Config is executed as JavaScript - only run qa-probe with a config you trust.
• ignoreHTTPSErrors: true disables TLS verification; dev/staging only.
• Security re-probing is GET-only and never writes. Write-flows are off by default and clean up after themselves; use them only against a disposable environment.

To report a vulnerability, see SECURITY.md.

---

Known limitations

• Detects visible HTTP calls and common hook patterns. Generated SDK clients, GraphQL clients, tRPC routers, and custom service layers may need an adapter - or use HAR import to discover them from real traffic.
• Dynamic URLs built by arbitrary factory functions are not detected.
• Best results need an OpenAPI spec; headless mode loses the spec-dependent rules but still detects observed-shape drift.

---

Contributing

See CONTRIBUTING.md. Contributions require a Developer Certificate of Origin sign-off (git commit -s).

---

Branding and trademarks

"qa-probe", "LightShield", and "LS-SIEM" are trademarks of LS-SIEM LLP. The Apache License does not grant permission to use these names or logos.

---

License

Copyright (c) 2026 LS-SIEM LLP - created and maintained by the LightShield SIEM team.

Licensed under the Apache License, Version 2.0 - see LICENSE and NOTICE. You may use, modify, and redistribute qa-probe under the Apache-2.0 terms; copyright and the trademarks above remain with LS-SIEM LLP.