This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
Fare rules lived in a GDS-ish API and in a messy help center. The model preferred the messy pages and then improvised timelines (“within 90 days”).
Visa and entry-rule questions were answered from training data, not from a current source.
Air Canada was ordered to honor a bereavement process its chatbot invented (B.C. Civil Resolution Tribunal, 2024). The company’s argument that the bot was a separate legal entity did not work. Travel assistants that speak policy are the company.
Highlights
- High-stakes intent routing to humans
- Fare-tool grounding
- Policy hallucination as Block
Typical launch bar: Wait until journeys and evals agree — do not ship on demo scores alone.
Challenges
Help articles contradicted the fare brand.
Codeshares meant two carriers’ rules; the bot picked the nicer one.
Emotionally charged bereavement and disruption chats are exactly when improvisation is most harmful.
How we tested
Fare, refund, and baggage answers cite the fare rules tool or refuse.
No invented dates or “you will be refunded.”
Booking confirmation amounts match checkout; supplier failure rolls back.
Cancellation policy math on modify/cancel.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| Help articles contradicted the fare brand. | Fare, refund, and baggage answers cite the fare rules tool or refuse. |
| Codeshares meant two carriers’ rules; the bot picked the nicer one. | No invented dates or “you will be refunded.” |
| Emotionally charged bereavement and disruption chats are exactly when improvisation is most harmful. | Booking confirmation amounts match checkout; supplier failure rolls back. |
What we delivered
Adversarial evals
- Promptfoo — Bereavement, IRROPS, visa inventions
RAG quality
- Ragas — Grounding in fare rules
Journeys / UI
- Playwright — Book, modify, cancel
LLM traces / evals
- LangSmith — Rules tool vs help-center improvisation
Outcomes
High-stakes intents (death in family, IRROPS, visa) required retrieval plus a human handoff offer. Invented timelines failed the pack.
Playwright booking/cancel math stayed in the same report. We treated policy hallucination as Block, not Medium “content.”
Stack
| Layer | Tool | What we validated |
|---|---|---|
| Adversarial evals | Promptfoo | Bereavement, IRROPS, visa inventions |
| RAG quality | Ragas | Grounding in fare rules |
| Journeys / UI | Playwright | Book, modify, cancel |
| LLM traces / evals | LangSmith | Rules tool vs help-center improvisation |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.