This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
Intent routing worked in a headset. On speakerphone, “send four hundred to Jordan” became a different amount or name.
When confidence was low, the agent still executed instead of repeating back.
Voice adds ASR error on top of tool-call risk. Confirmation has to be heard, not assumed. DTMF fallbacks still need to exist.
Highlights
- Repeat-back as a control
- Confidence thresholds tied to writes
- Recording-store review
Typical launch bar: Wait until journeys and evals agree — do not ship on demo scores alone.
Challenges
We could not call real customers. Fixture audio had to cover accents, overlap, and noise.
The NLU vendor and the LLM vendor scored “success” differently.
Call recordings in a third-party bucket were a data-residency surprise.
How we tested
High-stakes actions require repeat-back of amount, destination, and a second confirmation.
Low ASR confidence → clarify or DTMF, never write.
Same IDOR and idempotency tests as the non-voice API.
Barge-in and timeout UX.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| We could not call real customers. Fixture audio had to cover accents, overlap, and noise. | High-stakes actions require repeat-back of amount, destination, and a second confirmation. |
| The NLU vendor and the LLM vendor scored “success” differently. | Low ASR confidence → clarify or DTMF, never write. |
| Call recordings in a third-party bucket were a data-residency surprise. | Same IDOR and idempotency tests as the non-voice API. |
What we delivered
LLM traces / evals
- LangSmith — ASR text vs tool args vs confirmation
API / contract
- Postman — Same transfer API from voice tokens
Journeys / UI
- Playwright — Account-linked voice settings and receipts
Adversarial evals
- Promptfoo — Social-engineering on the speech-to-text transcript
Outcomes
Transfers without a confirmed repeat-back failed. DTMF path was tested as a first-class journey. Recording bucket policy went in the report.
Idempotency keys on the voice path used the same ledger tests as the app.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| LLM traces / evals | LangSmith | ASR text vs tool args vs confirmation |
| API / contract | Postman | Same transfer API from voice tokens |
| Journeys / UI | Playwright | Account-linked voice settings and receipts |
| Adversarial evals | Promptfoo | Social-engineering on the speech-to-text transcript |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.