This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
The brief sounded helpful: strengths, gaps, suggested questions. One gap was “no leadership” for a candidate whose CV listed a team lead role the retriever missed. Another brief mixed companies from a different applicant.
Interviewers pasting the brief into personal Gmail was a DPA issue the product had not designed for (copy button with no watermark or warning).
Hallucinated employment history is an adverse action waiting to happen. Isolation between requisitions is as important as summary quality.
Highlights
- Canary mixup tests
- Disallowed-question filters
- Copy/export as a privacy control
Typical launch bar: Block until pass rules hold on the money, identity, or safety path.
Challenges
PDF CVs parsed poorly; the model filled missing sections.
Calendar tools pulled attendee names from the wrong event.
Managers wanted “probing questions” that implied medical or family status — illegal in some jurisdictions.
How we tested
Ground every biographical claim in the application packet.
Canary fields per candidate to detect mixups.
Interviewer sees only assigned reqs.
Export and copy UX that does not pretend the brief is a system of record.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| PDF CVs parsed poorly; the model filled missing sections. | Ground every biographical claim in the application packet. |
| Calendar tools pulled attendee names from the wrong event. | Canary fields per candidate to detect mixups. |
| Managers wanted “probing questions” that implied medical or family status — illegal in some jurisdictions. | Interviewer sees only assigned reqs. |
What we delivered
RAG quality
- Ragas — Faithfulness to the packet
Adversarial evals
- Promptfoo — Disallowed interview questions
API / contract
- Postman — Interviewer RBAC
Journeys / UI
- Playwright — Brief, notes, no cross-req leak
Outcomes
Unsupported bio claims failed. Question suggestions were filtered against a disallowed-topic list. Mixup canaries failed the suite.
We added a “from application, not inferred” label and blocked auto-copy of full packets to mailto links.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| RAG quality | Ragas | Faithfulness to the packet |
| Adversarial evals | Promptfoo | Disallowed interview questions |
| API / contract | Postman | Interviewer RBAC |
| Journeys / UI | Playwright | Brief, notes, no cross-req leak |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.