This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
The happy path sent one payout. When the tool timed out, the agent retried with a new idempotency key because the model “helpfully” generated a fresh reference.
A second issue: the model called refund and payout in one turn when the ticket text was ambiguous.
Double-pay from retries is a classic payments defect. Putting an LLM in front of the same APIs adds a second retry loop the PSP never saw.
Highlights
- Ledger as source of truth for agent success
- Server-issued idempotency keys
- Confirmation gates on write tools
Typical launch bar: Block until pass rules hold on the money, identity, or safety path.
Challenges
Staging PSP and production PSP behaved differently on timeout.
The agent’s planner issued parallel tool calls; our first eval only looked at the final chat text.
Support wanted speed. Confirmation felt slow until we showed the duplicate payout in the ledger.
How we tested
Tool layer owns idempotency. The model must not mint payment references.
Write tools require confirmation for amount, beneficiary, and reason. Ambiguous tickets must not execute.
Replay duplicate webhooks and out-of-order PSP events.
Trace every tool call: arguments, HTTP status, whether money moved.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| Staging PSP and production PSP behaved differently on timeout. | Tool layer owns idempotency. The model must not mint payment references. |
| The agent’s planner issued parallel tool calls; our first eval only looked at the final chat text. | Write tools require confirmation for amount, beneficiary, and reason. Ambiguous tickets must not execute. |
| Support wanted speed. Confirmation felt slow until we showed the duplicate payout in the ledger. | Replay duplicate webhooks and out-of-order PSP events. |
What we delivered
LLM traces / evals
- LangSmith — Multi-step tool traces
API / contract
- Postman / Newman — Webhook duplicates and idempotency
Journeys / UI
- Playwright — Confirmation UI and cancel-back
Adversarial evals
- Promptfoo — Ambiguous tickets that must not pay
Failure telemetry
- Sentry — Timeout and retry storms
Outcomes
Pass rule: chat can look successful and the case still fails if two movements exist. We tested the ledger, not the bubble.
Schema lock: payment tools reject missing idempotency keys generated server-side. Promptfoo plus Newman collections ran on every agent prompt change.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| LLM traces / evals | LangSmith | Multi-step tool traces |
| API / contract | Postman / Newman | Webhook duplicates and idempotency |
| Journeys / UI | Playwright | Confirmation UI and cancel-back |
| Adversarial evals | Promptfoo | Ambiguous tickets that must not pay |
| Failure telemetry | Sentry | Timeout and retry storms |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.