This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
Faithfulness on 50 golden questions looked strong. Staging signup emails were dead, SSO duplicated users, and the copilot sat behind a paywall that did not unlock after payment.
Go/no-go used the eval number because it was the only automated signal.
LLM evals and product QA answer different questions. A golden-prompt pass is not a test plan. Demos hide fallbacks, permissions, and billing.
Highlights
- Two-layer go/no-go
- Severity on user impact
- Named residual-risk owner
Typical launch bar: Wait until journeys and evals agree — do not ship on demo scores alone.
Challenges
The ML team and the QA team did not share a launch checklist.
Payment was tested with a mock that always succeeded.
Nobody wanted to be the person who said Wait when the demo was scheduled.
How we tested
Pair evals with journey tests: signup, billing, permissions, fallback.
Severity on product impact, not on model-score deltas.
Ship / wait / block with an owner for residual risk.
Regression pack in CI so prompt changes cannot silently break checkout.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| The ML team and the QA team did not share a launch checklist. | Pair evals with journey tests: signup, billing, permissions, fallback. |
| Payment was tested with a mock that always succeeded. | Severity on product impact, not on model-score deltas. |
| Nobody wanted to be the person who said Wait when the demo was scheduled. | Ship / wait / block with an owner for residual risk. |
What we delivered
Journeys / UI
- Playwright — Signup, billing, entitlement
Adversarial evals
- Promptfoo — Small golden + adversarial set
API / contract
- Postman — SSO and role checks
LLM traces / evals
- LangSmith — Prompt regression
CI regression
- GitHub Actions — Both layers on PR
Outcomes
We reported two layers: model behavior and product journeys. Entitlement-after-pay was High. Eval green + broken auth is Wait.
Playwright critical path plus a small Promptfoo pack plus a written gate. The launch owner signed the residual risk list — we did not invent urgency, and we did not hide the auth bugs behind a faithfulness score.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| Journeys / UI | Playwright | Signup, billing, entitlement |
| Adversarial evals | Promptfoo | Small golden + adversarial set |
| API / contract | Postman | SSO and role checks |
| LLM traces / evals | LangSmith | Prompt regression |
| CI regression | GitHub Actions | Both layers on PR |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.