This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
Order-status lookups worked. Policy questions hit stale help-center chunks and then the model’s general “good CX” prior.
The copilot also issued refunds through a tool with no amount cap when the user said “as we agreed in chat.”
Air Canada’s chatbot invented a bereavement fare process; the tribunal held the airline responsible for website information, chatbot included. Retail return policies are the same commitment class.
Highlights
- Policy citation required
- Write-tool caps and confirmation
- Cart tenancy beside LLM tests
Typical launch bar: Wait until journeys and evals agree — do not ship on demo scores alone.
Challenges
Help center had three conflicting return pages (EU, US, outlet).
The model treated an expired promo banner as live.
Guest vs logged-in carts merged into another customer’s lines after account switch — a tenancy bug next to the LLM work.
How we tested
Policy answers cite the current help article. Conflicts between articles fail the case.
Refund/cancel tools use the same authorization as a human agent, with confirmation and caps.
Money path in Playwright: search → PDP → cart → pay → order visible.
Promo claims must match the cart engine.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| Help center had three conflicting return pages (EU, US, outlet). | Policy answers cite the current help article. Conflicts between articles fail the case. |
| The model treated an expired promo banner as live. | Refund/cancel tools use the same authorization as a human agent, with confirmation and caps. |
| Guest vs logged-in carts merged into another customer’s lines after account switch — a tenancy bug next to the LLM work. | Money path in Playwright: search → PDP → cart → pay → order visible. |
What we delivered
Journeys / UI
- Playwright — Checkout, returns, guest cart merge
Adversarial evals
- Promptfoo — Invented policy and social-engineered refunds
LLM traces / evals
- LangSmith — Which help article was retrieved
API / contract
- Postman — Refund tool authz and PSP webhooks
Outcomes
Region-aware retrieval and a “no policy statement without a citation” rule. Refund tool required the same role as human support.
We would not ship policy evals green while guest-cart merge was red. Both were in the launch pack.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| Journeys / UI | Playwright | Checkout, returns, guest cart merge |
| Adversarial evals | Promptfoo | Invented policy and social-engineered refunds |
| LLM traces / evals | LangSmith | Which help article was retrieved |
| API / contract | Postman | Refund tool authz and PSP webhooks |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.