This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
Chitchat and navigation worked. Eligibility, deadlines, and “do I qualify” questions were abbreviated or wrong, with no citation.
Identity-proofed case status leaked another resident’s ticket when ids were sequential.
Canada’s Auditor General reported that the CRA’s Charlie chatbot answered test questions incorrectly most of the time, after large spend. IRS chatbot self-serve rates have also declined in public reporting. Accuracy against a frozen question set is the product.
Highlights
- Gold-set accuracy over containment
- Shared gold answers across voice and chat
- Eval-set ownership when rules change
Typical launch bar: Block until pass rules hold on the money, identity, or safety path.
Challenges
Teams celebrated containment rate (users who did not escalate) which can rise when the bot is confidently wrong.
Gold answers changed when regulations changed; the eval set was stale.
Voicebot and chatbot shared intents but not the same refusal behavior.
How we tested
A scored question bank from the real call-center top intents, with gold answers from the policy team.
“I don’t know / see this page” beats a short wrong answer.
Case status IDOR and language/accessibility on the filing path.
Records retention and what gets logged from a resident chat.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| Teams celebrated containment rate (users who did not escalate) which can rise when the bot is confidently wrong. | A scored question bank from the real call-center top intents, with gold answers from the policy team. |
| Gold answers changed when regulations changed; the eval set was stale. | “I don’t know / see this page” beats a short wrong answer. |
| Voicebot and chatbot shared intents but not the same refusal behavior. | Case status IDOR and language/accessibility on the filing path. |
What we delivered
Adversarial evals
- Promptfoo — Gold question bank
LLM traces / evals
- LangSmith — Trace and citation presence
API / contract
- Postman — Case-status IDOR
Journeys / UI
- Playwright + axe — Identity and filing UX
Outcomes
We reported accuracy on the gold set, not containment. Containment without accuracy was marked as a misleading metric in the report.
A dated eval set with an owner, plus IDOR tests, became the release gate. Voice and chat had to share the same gold answers.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| Adversarial evals | Promptfoo | Gold question bank |
| LLM traces / evals | LangSmith | Trace and citation presence |
| API / contract | Postman | Case-status IDOR |
| Journeys / UI | Playwright + axe | Identity and filing UX |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.