This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
A slightly different enum (“US” vs “USA”) 500’d the tool. The agent retried three times and charged the user for all four calls.
A second tool ran without approval because the planner issued it in parallel with a read.
Broken tool calls are a top real-world agent failure. If the model “answers safely” but the tool still executes — or loops — you missed the actual defect.
Highlights
- Side-effect oracles
- Forced provider failure
- Schema drift as a first-class suite
Typical launch bar: Wait until journeys and evals agree — do not ship on demo scores alone.
Challenges
Eval harnesses scored the final natural-language answer, which apologized while the refund still went out.
Staging tools were more permissive than production schemas.
n8n/LangChain glue swallowed errors into “ok.”
How we tested
Schema-strict fixtures: missing field, extra field, wrong type, enum drift.
Write tools require approval; parallel writes are in scope.
Cost/loop caps; provider errors must not infinite-retry.
Fallback UX when the tool fails: no blank thread, no silent success.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| Eval harnesses scored the final natural-language answer, which apologized while the refund still went out. | Schema-strict fixtures: missing field, extra field, wrong type, enum drift. |
| Staging tools were more permissive than production schemas. | Write tools require approval; parallel writes are in scope. |
| n8n/LangChain glue swallowed errors into “ok.” | Cost/loop caps; provider errors must not infinite-retry. |
What we delivered
LLM traces / evals
- LangSmith — Tool traces, retries, cost
Adversarial evals
- Promptfoo — Argument drift and injection into tool args
API / contract
- Postman — Tool API contract
Journeys / UI
- Playwright — Approval and fallback UX
Failure telemetry
- Sentry — Swallowed 500s in glue
Outcomes
Success is defined on tool traces and side effects, not on the chat bubble. Loop/cost caps were tested by forcing provider 500s.
Approval was a product control, not a system-prompt suggestion. GitHub Actions re-ran the pack on schema changes.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| LLM traces / evals | LangSmith | Tool traces, retries, cost |
| Adversarial evals | Promptfoo | Argument drift and injection into tool args |
| API / contract | Postman | Tool API contract |
| Journeys / UI | Playwright | Approval and fallback UX |
| Failure telemetry | Sentry | Swallowed 500s in glue |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.