| Pillar | What we measure | Pass criteria (examples) | Tools | Severity if failed |
|---|---|---|---|---|
| 1. Answer quality | Correctness, completeness, tone, refusal behavior | >90% pass on golden dataset; safe refusal on out-of-scope | LangSmith, Braintrust, human review | Critical |
| 2. Retrieval & grounding | Source accuracy, staleness, tenant isolation, citations | Faithfulness >0.85; no cross-tenant leakage in test set | Ragas, custom retrieval probes | Critical |
| 3. Agent & tool safety | Action approvals, schema validation, partial failure recovery | Zero unauthorized writes; clean rollback on tool error | Playwright, LangSmith traces, manual adversarial | Critical |
| 4. Security & guardrails | Prompt injection, PII leakage, jailbreaks, data exfiltration | No system prompt exposure; PII blocked in test harness | Promptfoo, OWASP ZAP, manual red-team | Critical |
| 5. Product experience | Loading states, fallbacks, latency, cost under real usage | Graceful degradation; p95 < SLA; no silent retry loops | Playwright, k6, Lighthouse | High |
Different AI products fail differently. This matrix shows priority order and depth for each type.
| Product type | Answer quality | RAG / grounding | Agent safety | Security | UX / fallback |
|---|---|---|---|---|---|
| Support copilot | Deep | Medium | Light | Deep | Deep |
| RAG assistant | Deep | Deep | Light | Deep | Medium |
| Agent workflow | Medium | Medium | Deep | Deep | Deep |
| AI SaaS feature | Deep | Medium | Medium | Deep | Deep |
| Internal AI tool | Medium | Deep | Medium | Deep | Medium |
| AI-built app (Lovable, Bolt, etc.) | Medium | Light | Medium | Deep | Deep |
We build 20–50 golden prompts from your docs, support tickets, and real user questions — plus adversarial variants.
Run through LangSmith, Braintrust, or Promptfoo with LLM-as-judge and rule-based assertions on structured outputs.
Human testers probe injection, ambiguity, multi-turn drift, and edge permissions that automation misses.
Playwright flows validate that model outputs actually drive the UI, APIs, and billing paths correctly.
We leave you with a repeatable eval suite wired to your CI so the next prompt change does not silently regress.
| Metric | Definition | Typical target | Tool source |
|---|---|---|---|
| Eval pass rate | % of golden prompts that meet quality criteria | >90% pre-launch; >95% post-hardening | LangSmith, Braintrust |
| Faithfulness (RAG) | Answer grounded in retrieved context, not invented | >0.85 on held-out doc set | Ragas |
| Context recall | Relevant docs actually retrieved for the question | >0.80 for in-scope queries | Ragas |
| Attack success rate | % of adversarial prompts that bypass guardrails | <5% on standard injection suite | Promptfoo |
| Tool call success | Agent actions complete without schema or auth errors | >98% on defined workflows | LangSmith traces |
| p95 latency | Time to first token / full response at staging load | Within your stated SLA | k6, Langfuse |
| Cost per session | Token spend for a typical user journey | Within budget; no retry loops | Langfuse, provider dashboards |
Share your product type, model stack, and launch timeline. We'll map which pillars matter most and what evidence you need before go-live.