This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
The product retrieved prior notes and generated a “visit summary” that staff could copy into the EHR. Golden prompts on the vendor’s sample chart passed. Real charts had conflicting meds, scanned PDFs, and another patient’s fax in the packet.
The failure we treat as launch-blocking is not a slightly clumsy sentence. It is a confident diagnosis, dose, or allergy that is not in the source chart — especially when the UI makes the draft look like a signed note.
Public reporting in 2025 described an NHS-used AI summarizer that produced false diagnoses (including diabetes) that then entered a patient record. Mixed true and false medical statements are hard for clinicians to spot at speed.
Highlights
- Claim-level grounding, not whole-answer “looks good”
- Known-absence tests (facts that must not be invented)
- Wrong-patient and break-glass audit checks
Typical launch bar: Block until pass rules hold on the money, identity, or safety path.
Challenges
PHI cannot go to a model vendor without a BAA — traces in LangSmith had to stay on an approved path or we could not run evals.
Clinicians disagreed on what “complete” meant; we had to freeze pass rules with the medical owner before scoring.
OCR’d labs and mixed-language notes broke retrieval; the model filled gaps instead of saying the page was unreadable.
How we tested
Build a golden set from de-identified fixture charts with known facts, known absences, and known conflicts — not from the pitch deck.
Score faithfulness: every medical claim must be entailed by retrieved context or marked uncertain. “Not in chart” is a pass rule, not a vibe.
Walk the clinician journey in Playwright: open the right patient, generate, edit, sign, audit. Wrong-patient context is a Critical, not a copy issue.
Probe out-of-scope questions (dosing, “what should I prescribe”) and require refuse + safe next step, not a helpful guess.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| PHI cannot go to a model vendor without a BAA — traces in LangSmith had to stay on an approved path or we could not run evals. | Build a golden set from de-identified fixture charts with known facts, known absences, and known conflicts — not from the pitch deck. |
| Clinicians disagreed on what “complete” meant; we had to freeze pass rules with the medical owner before scoring. | Score faithfulness: every medical claim must be entailed by retrieved context or marked uncertain. “Not in chart” is a pass rule, not a vibe. |
| OCR’d labs and mixed-language notes broke retrieval; the model filled gaps instead of saying the page was unreadable. | Walk the clinician journey in Playwright: open the right patient, generate, edit, sign, audit. Wrong-patient context is a Critical, not a copy issue. |
What we delivered
RAG quality
- Ragas — Faithfulness and context recall on fixture charts
LLM traces / evals
- LangSmith — Trace which chunks were actually retrieved
Journeys / UI
- Playwright — Right-patient journey, timeout, sign-off UI
API / contract
- Postman — Patient IDOR: token A cannot GET chart B
Adversarial evals
- Promptfoo — Out-of-scope medical advice and jailbreaks
Outcomes
We blocked copy-to-chart until the draft showed claim-level grounding (supported / unsupported / missing). Unsupported claims could not be one-click accepted.
IDOR tests on chart IDs and a short idle-timeout check on shared clinic devices went into the same report as the eval failures.
The go/no-go was Wait: fluency was fine, chart fidelity was not. Retest was against the same fixture pack after the refuse-and-highlight change.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| RAG quality | Ragas | Faithfulness and context recall on fixture charts |
| LLM traces / evals | LangSmith | Trace which chunks were actually retrieved |
| Journeys / UI | Playwright | Right-patient journey, timeout, sign-off UI |
| API / contract | Postman | Patient IDOR: token A cannot GET chart B |
| Adversarial evals | Promptfoo | Out-of-scope medical advice and jailbreaks |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.