This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
The extractor was accurate on clean passports. Blurry images, two-sided IDs, and a second file from a joint applicant caused field bleed: DOB from person A, name from person B.
The product still opened a funded wallet while KYC sat in “manual review,” against stated policy.
Identity products fail on matching and isolation, not on OCR accuracy demos. A wrong merge is an account-takeover and AML problem.
Highlights
- State-machine testing (pending cannot transact)
- Conflict vs confidence scoring
- Upload isolation
Typical launch bar: Block until pass rules hold on the money, identity, or safety path.
Challenges
Vendors quote “99% accuracy” without saying on which field and which image quality.
Reviewers rubber-stamped the model’s fields because the UI highlighted them as verified.
Retention: failed KYC images stayed in a public bucket longer than the policy allowed.
How we tested
Fixture pack: pass, fail, manual review, expired ID, mismatched selfie, joint application.
Human-review queue must see the raw image next to extracted fields; auto-accept only on high-confidence consistent fields.
Policy test: pending users cannot move money if the policy says so.
Document isolation: applicant A cannot GET applicant B’s image by id.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| Vendors quote “99% accuracy” without saying on which field and which image quality. | Fixture pack: pass, fail, manual review, expired ID, mismatched selfie, joint application. |
| Reviewers rubber-stamped the model’s fields because the UI highlighted them as verified. | Human-review queue must see the raw image next to extracted fields; auto-accept only on high-confidence consistent fields. |
| Retention: failed KYC images stayed in a public bucket longer than the policy allowed. | Policy test: pending users cannot move money if the policy says so. |
What we delivered
Journeys / UI
- Playwright — Onboarding states and review queue
API / contract
- Postman — Document IDOR and pending-transaction policy
Security scan
- OWASP ZAP — Open buckets and debug upload endpoints
LLM traces / evals
- LangSmith — Extraction traces vs raw fields
Outcomes
We failed auto-accept when any identity field conflicted or confidence was below the agreed threshold. Wallet funding was blocked in the pending state with an automated API test.
S3/object ACLs and signed URL expiry were reported as High alongside the extraction bugs.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| Journeys / UI | Playwright | Onboarding states and review queue |
| API / contract | Postman | Document IDOR and pending-transaction policy |
| Security scan | OWASP ZAP | Open buckets and debug upload endpoints |
| LLM traces / evals | LangSmith | Extraction traces vs raw fields |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.