We combine industry-standard QA automation with modern LLM evaluation platforms. No black box — here's exactly what we reach for, when, and why. For how we evaluate AI behavior, see the evaluation method.
Every engagement mixes manual exploratory testing with the right automation. This table shows our default stack — we adapt when your product already uses something else.
| Layer | Primary tool | Also used | What we validate | Best for |
|---|---|---|---|---|
| E2E & UI | Playwright | Cypress, Selenium | User journeys, auth, billing, AI chat UI, responsive layouts | All web and AI-built apps |
| API & integrations | Postman / Newman | REST Client, curl scripts | REST/GraphQL contracts, webhooks, model API payloads | Backend-heavy SaaS, agent tool APIs |
| LLM evals & tracing | LangSmith | Braintrust, Langfuse | Prompt regressions, trace review, dataset runs, latency | Copilots, RAG, multi-step agents |
| Prompt red-teaming | Promptfoo | Custom adversarial suites | Jailbreaks, injection, unsafe outputs, model A/B | Customer-facing chat, guardrail review |
| RAG quality | Ragas | Custom retrieval checks | Faithfulness, context recall, citation accuracy | Knowledge assistants, doc search |
| Performance & load | k6 | Lighthouse, WebPageTest | API throughput, p95 latency, token/cost spikes under load | Launch readiness, scale events |
| Security scanning | OWASP ZAP | Burp Suite (manual) | XSS, auth gaps, exposed endpoints, header misconfig | Apps handling PII or payments |
| Accessibility | axe DevTools | WAVE, keyboard audits | WCAG issues, focus order, screen reader flows | Public-facing AI UIs |
| Mobile & cross-browser | BrowserStack | Real device lab, Sauce Labs | Safari, Chrome mobile, tablet breakpoints | Consumer apps, AI mobile wrappers |
| CI/CD pipelines | GitHub Actions | CircleCI, Jenkins, GitLab CI | Playwright on PR, eval runs, Newman collections | Teams shipping daily with regression needs |
| Error monitoring | Sentry | Datadog, Bugsnag | Model failures, API 500s, client-side AI UI crashes | Production-adjacent staging, launch week |
| Dependency security | Snyk | Dependabot, Semgrep | Vulnerable packages, secret leaks in repos | AI-built apps with fast dependency churn |
| Test management | TestRail | Qase, Zephyr, Xray | Case tracking, run history, retest cycles | Longer engagements, compliance audits |
| Native mobile | Appium | Maestro, Detox, XCUITest | iOS/Android flows, AI chat in native shells | Mobile-first AI products |
| Visual regression | Percy | Chromatic, Applitools | UI drift on AI-generated components | Design-heavy AI SaaS, v0-built UIs |
| Contract testing | Pact | Schema validation scripts | Consumer/provider API contracts between services | Microservices with model gateway layers |
Playwright is our default for new work. We still use Cypress or Selenium when your team already invested in them.
| Tool | Our usage | Strengths | Limitations | AI product fit |
|---|---|---|---|---|
| Playwright | Primary | Fast, multi-browser, auto-wait, trace viewer, API mocking | Steeper learning curve for non-dev testers | Excellent — handles dynamic AI UI, streaming responses, iframes |
| Cypress | Secondary | Great DX, time-travel debug, component testing | Single-tab focus; cross-origin needs workarounds | Good for embedded copilots in same-origin apps |
| Selenium | Legacy / client stack | Wide language support, existing enterprise grids | Slower, flakier without careful waits | Used when your CI already runs Selenium Grid |
| TestCafe | Alternative | No WebDriver, simple setup, built-in waits | Smaller ecosystem than Playwright | Teams wanting lighter E2E without Node complexity |
| Puppeteer | Chrome-focused | Headless Chrome, PDF/screenshot capture | Chromium-only unless paired with others | Scraping, PDF exports, Chrome-only internal tools |
| WebdriverIO | Alternative | Unified API across browsers, Appium integration | Config can get verbose | Web + mobile in one framework |
These are the trending platforms teams use to prove LLM quality — we operate them on your staging environment so you get evidence, not vibes.
| Tool | Category | What we test with it | Typical metrics | When we lead with it |
|---|---|---|---|---|
| LangSmith | Tracing & evals | Full prompt chains, tool calls, retrieval steps, regression datasets | Pass rate, latency, token use, error rate | LangChain / LangGraph stacks, agent debugging |
| Braintrust | Eval platform | Prompt/version comparisons, human + LLM-as-judge scoring | Score deltas, regression alerts, side-by-side diffs | Teams already on Braintrust for prod evals |
| Promptfoo | Red-teaming | Adversarial prompts, injection suites, multi-model benchmarks | Attack success rate, refusal quality, guardrail bypass | Pre-launch safety review, compliance-sensitive chat |
| Langfuse | Observability | Production-like trace replay, cost tracking, session review | Cost per session, trace errors, user feedback loops | Open-source observability preference |
| Ragas | RAG metrics | Retrieval quality, answer grounding, context precision | Faithfulness, context recall, answer relevancy | Document Q&A, internal knowledge bases |
| Guardrails AI | Output validation | Schema enforcement, PII detection, topic boundaries | Validation pass rate, blocked output types | Structured agent outputs, regulated content |
| DeepEval | LLM metrics | pytest-style evals, G-Eval, hallucination detection | Metric scores per test case, pass/fail thresholds | Python teams wanting evals in existing test suites |
| TruLens | Feedback & eval | RAG triad metrics, agent feedback functions | Groundedness, relevance, sentiment | RAG pipelines needing continuous feedback loops |
| Arize Phoenix | Observability | Trace visualization, embedding analysis, eval runs | Cluster drift, retrieval quality over time | Open-source eval + observability preference |
| Weights & Biases | Experiment tracking | Prompt experiment logs, eval run comparison | Run history, metric charts, team collaboration | Teams already using W&B for ML workflows |
| OpenAI Evals | Eval framework | Custom eval templates, model grading | Accuracy, pass@k, custom rubric scores | OpenAI-native stacks, custom benchmark building |
| Helicone | LLM observability | Request logging, cost analytics, latency monitoring | Cost per request, cache hit rate | Cost-sensitive AI features at scale |
Lovable, Bolt, Cursor, v0, and similar tools ship fast. Our testing focus shifts to permissions, integrations, and the gaps speed leaves behind.
| Platform / stack | What teams ship | Common failure modes | Tools we lean on | First tests we run |
|---|---|---|---|---|
| Lovable | Full-stack React apps with Supabase | RLS gaps, auth edge cases, missing error states | Playwright, Postman, Supabase policy review | Multi-user access, CRUD permissions, AI feature flows |
| Bolt / StackBlitz | Browser-based full-stack prototypes | Env secrets, API key exposure, untested deploy paths | Playwright, OWASP ZAP, manual config audit | Secret scanning, prod vs dev config, core user paths |
| Cursor / Claude Code | Custom codebases with AI-assisted dev | Regressions after prompt-driven refactors, weak tests | Playwright, existing test runner, LangSmith | Regression suite gap analysis, critical path automation |
| v0 (Vercel) | UI components and Next.js shells | Accessibility gaps, responsive breakage, form validation | Playwright, axe, Lighthouse | Component states, mobile layouts, form edge cases |
| Replit Agent | Rapid MVPs with hosted backend | Rate limits, auth shortcuts, incomplete integrations | Postman, k6, Playwright | API contracts, load on AI endpoints, signup-to-value flow |
| OpenAI / Anthropic / Gemini APIs | Direct model integrations | Hallucinations, structured output drift, cost runaway | Promptfoo, LangSmith, custom eval sets | Edge prompts, JSON schema validation, fallback behavior |
| Tool | Type | Use case in our engagements | AI-specific checks |
|---|---|---|---|
| Postman + Newman | API functional | Collection runs in CI, contract tests, webhook replay | Model response schema, streaming endpoint timeouts |
| k6 | Load & stress | Spike tests on auth, chat, and inference endpoints | Token budget under concurrency, queue behavior |
| Lighthouse | Web performance | Core Web Vitals, bundle size, render blocking | Chat widget impact on LCP, lazy-loaded AI panels |
| WebPageTest | Real-world perf | Multi-region latency, filmstrip analysis | Time-to-first-token perception in embedded AI UI |
| Artillery | Load testing | Scenario-based load, WebSocket and HTTP mix | Streaming chat endpoint saturation |
| Insomnia / Hoppscotch | API exploration | Manual API probing during exploratory testing | Quick model endpoint experiments, header checks |
| GraphQL Playground | GraphQL testing | Query/mutation validation, schema introspection | AI resolver latency, N+1 on embedding lookups |
| REST Assured | API automation | Java/Kotlin API test suites in CI | Structured output schema assertions |
| WireMock / MSW | API mocking | Stub model providers, simulate failures and latency | Offline E2E, provider outage scenarios |
We don't force a single vendor stack. If your team already uses something here, we plug into it. Not sure what you are missing? Try the free coverage gap finder.
| Tool | Type | What we test | AI-specific use |
|---|---|---|---|
| OWASP ZAP | DAST scanner | XSS, CSRF, exposed endpoints, misconfigured headers | Chat widget injection surfaces, API key in client bundles |
| Burp Suite | Manual pentest | Session hijacking, auth bypass, business logic flaws | Multi-tenant isolation, privilege escalation via AI actions |
| Promptfoo | LLM red-team | Jailbreaks, injection, harmful content generation | Core tool for pre-launch AI safety reviews |
| Snyk | SCA / secrets | Vulnerable dependencies, leaked API keys in git | Catch exposed OpenAI keys in AI-built repos |
| Semgrep | SAST | Custom rule scans on auth and data-handling code | Rules for unsafe prompt concatenation patterns |
| truffleHog | Secret scanning | Historical commits with credentials | Model provider keys in Bolt/Lovable export repos |
Need help ranking what you found? Use the free severity classifier →
| Scenario | Manual exploratory | Automated (Playwright / evals) | Our recommendation |
|---|---|---|---|
| First launch of an AI copilot | ✓ Essential | ◐ Seed eval datasets | Manual first, then automate top 10 journeys |
| Prompt or model version change | ◐ Spot-check edge cases | ✓ Regression eval suite | Automated evals on every change |
| Prompt injection / jailbreak review | ✓ Creative adversarial thinking | ✓ Promptfoo suites | Both — automation scales, manual finds novel attacks |
| Signup → pay → use AI feature | ◐ Exploratory once | ✓ Playwright E2E | Automate the golden path, manual for edge permissions |
| RAG citation accuracy | ✓ Domain expert review | ✓ Ragas + custom checks | Hybrid — metrics flag drift, humans judge nuance |
| Visual / UX polish of AI UI | ✓ Required | ✕ Limited value | Manual + accessibility tooling |
Tell us what you're shipping and which tools you already use. We'll map a scope that fits your CI, your models, and your launch date.