Testers

AI-powered QA testing CLI — spawns cheap AI agents to test web apps with headless browsers

npm License

Install

bun install -g @hasna/testers
# or
npm install -g @hasna/testers

CLI Usage

testers --help
testers run https://my-preview.example.com

Passing a URL as the first argument will, by default, crawl the site and auto-generate scenarios if none exist for the project. Disable with --no-auto-generate.

Sandbox Workflow Fanout

Saved workflows can run in E2B-backed sandboxes through @hasna/sandboxes. Sandbox workflow uploads default to rsync staging and the image-model default is gpt-image-2.

export E2B_API_KEY=...

testers workflow create "Projects CRUD" \
  --project alumia \
  --tag projects \
  --target sandbox \
  --sandbox-provider e2b \
  --sandbox-sync rsync \
  --timeout 600000

testers workflow fanout --project alumia --workers 6 --url https://preview.example.com
testers workflow fanout wf_abc,wf_def wf_xyz --workers 12 --url https://preview.example.com --json
testers workflow fanout --project alumia --tag action-specific --workers 6 --batch-size 12 --batch 1 --url https://preview.example.com
testers workflow fanout --project alumia --tag action-specific --workers 6 --batch-size 12 --all-batches --url https://preview.example.com

--workers is bounded to 1-12 concurrent sandboxes. Use --batch-size with a 1-based --batch to run large workflow corpora in deterministic waves, or --all-batches with optional --from-batch / --to-batch to run a staged range in one command. --offset is available for a manual selected-workflow cursor. Fanout preflights provider credentials, required sandbox environment references, rsync, and app source directories before launching workers. Use --dry-run to inspect the remote commands, upload plans, and preflight checks without spawning sandboxes.

Next.js Route and Action Inventory

For large apps, generate source-derived route coverage from the Next.js app directory. The importer can create route-level scenarios and one scenario per discovered link, button, form, input, or API method, then group those action scenarios into sandbox workflows for fanout.

testers inventory next /path/to/app \
  --project alumia \
  --create-scenarios \
  --create-action-scenarios \
  --create-workflows \
  --create-action-workflows \
  --action-workflow-grouping action \
  --sandbox-provider e2b \
  --sandbox-sync rsync \
  --sandbox-app-source /path/to/app \
  --sandbox-app-start-command "bun install && bun dev --hostname 0.0.0.0" \
  --sandbox-app-url http://127.0.0.1:3000 \
  --sandbox-app-wait-url http://127.0.0.1:3000/health \
  --sandbox-env-optional OPENAI_API_KEY

testers workflow fanout --project alumia --tag action-specific --workers 6 --batch-size 12 --all-batches --from-batch 1 --to-batch 3 --url https://preview.example.com --dry-run

Use --action-workflow-grouping action for one workflow per discovered action, route for route-specific workflows, or area-kind for broader workflows such as commerce buttons or admin API methods. Add the --sandbox-app-* flags when the sandbox should rsync, install, start, and test the app source instead of only testing an already-running URL.

App Sandbox URL

Use testers sandbox launch when you want a long-lived remote app URL before running tests. It uploads the working tree with safe excludes (.git, .env*, .npmrc, node_modules, build output, private key files), resolves env values from host env or @secrets: references, starts the app in dev or production mode, waits for internal readiness, verifies the public routed URL, and prints the public URL.

export E2B_API_KEY=...
export ANTHROPIC_API_KEY=...

testers sandbox launch /path/to/app \
  --provider e2b \
  --mode dev \
  --port 3000 \
  --env ANTHROPIC_API_KEY \
  --url-env APP_URL \
  --url-env NEXT_PUBLIC_APP_URL \
  --host-env NEXT_ALLOWED_DEV_ORIGINS \
  --wait-url /health \
  --ttl 2h

testers run "$(testers sandbox launch /path/to/app --provider e2b --json | jq -r .publicUrl)"

For monorepos and database-backed apps, set a working directory and setup commands explicitly:

testers sandbox launch /path/to/alumia \
  --provider e2b \
  --image node-22-heavy \
  --working-dir packages/web \
  --mode dev \
  --port 3325 \
  --provider-env-file packages/web/.env.local \
  --min-memory-mb 7500 \
  --scan-required-env \
  --generate-missing-secret-env \
  --postgres \
  --db-setup "pnpm run db:migrate:run" \
  --db-setup "pnpm run db:seed" \
  --env ANTHROPIC_API_KEY \
  --url-env APP_URL \
  --url-env NEXT_PUBLIC_APP_URL \
  --url-env NEXT_PUBLIC_URL \
  --host-env NEXT_ALLOWED_DEV_ORIGINS \
  --url-env WEBAUTHN_ORIGIN \
  --wait-url /api/v1/health \
  --ttl 2h

Use testers sandbox list, testers sandbox logs <id>, and testers sandbox stop <id> to inspect and clean up launched app sandboxes. Providers that cannot expose a public port fail fast with a provider capability error. Sandbox commands also receive TESTERS_PUBLIC_URL, TESTERS_PUBLIC_HOST, and TESTERS_INTERNAL_URL; URL envs receive the full public origin, while host envs receive the bare public hostname.

Pass --scan-required-env when a repo centralizes production env access through helpers such as requireEnv("KEY"). The launcher scans source files for those strongly required keys before upload, treats common public origin keys such as APP_URL and NEXT_PUBLIC_APP_URL as dynamic sandbox URL envs, and fails before creating a misleading URL when a non-secret value is missing. Add --generate-missing-secret-env to create random sandbox-only values for missing internal secret-like keys such as JWT/HMAC/signing/encryption secrets; external service API keys still need to be supplied with --env, --env-file, or a secret reference.

For E2B CPU/RAM sizing, create or choose an E2B template with the required resources and pass it with --image <template-alias-or-id>. The current E2B sandbox create API makes memory a template setting, not a per-launch flag. Use --min-memory-mb <mb> to verify the created sandbox before upload/setup; this catches template/account-limit mismatches early instead of returning a URL that dies under load. When --postgres is enabled, testers uses the sandbox-local DATABASE_URL and DATABASE_DIRECT_URL so migration and seed commands cannot accidentally target a host or production database from an env file. After each database setup command, testers reapplies schema/table/function grants and grants non-system NOLOGIN roles held by the migration role to the sandbox PostgreSQL user; pass --no-postgres-grant-created-roles when you need strict role-membership modeling.

For apps that cannot build inside the sandbox resource limit, build locally first, then pass --include-build-output --build none. This explicitly allows common artifact directories like .next, dist, build, and out through the upload filter while still excluding .env*, .npmrc, private keys, and node_modules. Next.js apps configured with output: "standalone" automatically start with .next/standalone/server.js; pass --start only when your app needs a custom production server command.

Common Flags

• --json --output results.json — write structured results to a file for downstream tooling.
• --limit <n> --offset <n> — page list, status, and result output without filling a terminal or agent context.
• --verbose — expand long human-readable fields such as URLs, scenario names, reasoning, errors, paths, and tags.
• --timeout <ms> — per-scenario timeout (default: 60s).
• --overall-timeout <ms> — hard timeout for the whole run (default: 10 minutes; CI safety net).
• --github-comment — post a pass/fail summary as a comment on the current GitHub PR.

Output And Gradual Disclosure

Human-readable list, status, and result commands are compact by default. They show essential fields, cap rows, truncate long text, and print a hint for the next detail command. Use detail commands such as testers show <id>, testers project show <id>, testers workflow show <id>, or testers results <run-id> --verbose when you need full context.

--json remains the machine-readable path and returns full records where commands already did so. MCP list tools follow the same compact default and accept limit, offset, and verbose: true; use a matching get_* tool when one exists for a single full record.

Secure Production Debugging

Use prod-debug when an agent or support engineer needs to inspect a production issue without handling customer passwords, raw cookies, bearer tokens, or OAuth codes:

testers prod-debug "https://alumia.com/acme/projects/project-123?agent=agent-id" --reason "connector auth error"
testers prod-debug "req_abc123" --logs --json
testers prod-debug "https://app.example.com/org/projects/p1" --profile app --json
testers prod-debug "https://app.example.com/org/projects/p1" --support-url "https://support.example.com/scoped/session" --support-grant support-grant-123

The command parses the target, redacts sensitive URL parameters, emits safe browser/API/log checks, and blocks user-scoped browser reproduction until the target app provides an audited support browser/session URL or a configured profile that can resolve one. It is app-generic: add a profile in ~/.hasna/testers/config.json for each production app rather than hardcoding app-specific behavior in the CLI.

{
  "prodDebug": {
    "apps": {
      "app": {
        "name": "App",
        "origins": ["https://app.example.com", "*.app.example.org"],
        "supportGrantRef": "$APP_SUPPORT_GRANT",
        "supportUrlTemplate": "https://support.example.com/scoped/session?grant={supportGrant}&target={targetUrlEncoded}",
        "piiOrigin": "https://api.app.example.com",
        "logCommand": "app logs --project {project} --session {session} --request {request}"
      }
    }
  }
}

Credential-bearing profile values can point at environment variables ($APP_SUPPORT_GRANT) or the local Hasna secrets vault (@secrets:division/app/support/grant). Generated plans redact token, grant, session, key, password, OAuth code, and bearer values before printing or writing output.

Exit Codes

Code	Meaning
0	All tests passed
1	One or more tests failed
2	Configuration error (missing API key, unreachable URL, overall-timeout hit, etc.)

GitHub Actions / PR Preview Testing

# .github/workflows/qa.yml
name: AI QA Tests
on:
  pull_request:

permissions:
  contents: read
  pull-requests: write

jobs:
  qa:
    runs-on: ubuntu-latest
    steps:
      - uses: oven-sh/setup-bun@v2
      - run: bun install -g @hasna/testers
      - run: testers run ${{ needs.deploy.outputs.preview_url }} --github-comment --json --output results.json
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: testers-results
          path: results.json

The --github-comment flag automatically:

• reads the PR number from GITHUB_REF (or GITHUB_PR_NUMBER for custom workflows),
• reads the repo from GITHUB_REPOSITORY,
• uses GITHUB_TOKEN to post a Markdown summary with a pass/fail table.

Generate a starter workflow automatically:

testers ci

MCP Server

testers-mcp

64 tools available.

HTTP mode

Shared Streamable HTTP transport (stateless, localhost only):

testers-mcp --http
# or: MCP_HTTP=1 testers-mcp

Default port 8840 (--port / MCP_HTTP_PORT). Endpoints: GET /health, POST /mcp.

REST API

testers-serve

Data Directory

Data is stored locally in ~/.hasna/testers/. Override the database path with HASNA_TESTERS_DB_PATH or TESTERS_DB_PATH.

License

Apache-2.0 -- see LICENSE