This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
The model answered from general training when the municipal corpus missed a clause. On a .gov origin, that reads as official interpretation.
Accessibility on the filing path failed keyboard checks. Deadline-day load dropped filings without an error the resident could see.
The Markup’s 2024 testing of New York City’s MyCity chatbot found systematic illegal guidance (tips, housing vouchers, cashless bans). Different users got contradictory answers. A disclaimer did not make the answers lawful.
Highlights
- Harm-question eval set owned by the agency
- Cross-session contradiction checks
- Translation fidelity to source law
Typical launch bar: Block until pass rules hold on the money, identity, or safety path.
Challenges
Agencies wanted coverage of thousands of pages without an evaluation set.
The model hedged (“generally you may”) which still reads as permission.
Translation into other languages drifted off the English legal meaning.
How we tested
Legal questions require retrieval from the agency’s published pages. No hit = refuse and link to the source page or a human.
Eval set of high-harm questions with answers checked by the policy owner — not by the vendor.
Contradiction test: same question, two sessions, answers must not conflict on a yes/no legal point.
axe + keyboard on the primary filing journey; spike test on deadline day.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| Agencies wanted coverage of thousands of pages without an evaluation set. | Legal questions require retrieval from the agency’s published pages. No hit = refuse and link to the source page or a human. |
| The model hedged (“generally you may”) which still reads as permission. | Eval set of high-harm questions with answers checked by the policy owner — not by the vendor. |
| Translation into other languages drifted off the English legal meaning. | Contradiction test: same question, two sessions, answers must not conflict on a yes/no legal point. |
What we delivered
Adversarial evals
- Promptfoo — High-harm legal questions
RAG quality
- Ragas — Grounding in agency pages
Journeys / UI
- Playwright + axe — Filing journey accessibility
Load / soak
- k6 — Deadline-day spike, no silent drop
LLM traces / evals
- LangSmith — Retrieval misses vs improvisation
Outcomes
We treated invented statute or eligibility as Block. Hedged illegal advice still failed. Translations were scored against the English source, not fluency.
Playwright + axe on filing, k6 on deadline shape, Promptfoo on the harm set. The city owner had to sign the residual risk for anything not in the corpus.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| Adversarial evals | Promptfoo | High-harm legal questions |
| RAG quality | Ragas | Grounding in agency pages |
| Journeys / UI | Playwright + axe | Filing journey accessibility |
| Load / soak | k6 | Deadline-day spike, no silent drop |
| LLM traces / evals | LangSmith | Retrieval misses vs improvisation |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.