This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
ACL filters existed on the search UI. The embedding index was a shared collection with metadata filters that did not apply on the chat path.
An answer cited “the indemnity clause we used last month” — that clause lived in a different matter the user should not see.
Tenant and matter isolation failures are the legal version of a cross-customer leak. Privilege and confidentiality do not survive a shared vector index with weak filters.
Highlights
- Canary phrases per tenant/matter
- Same authz on UI, API, and RAG
- Privilege preservation in summaries
Typical launch bar: Block until pass rules hold on the money, identity, or safety path.
Challenges
Engineering believed UI filters equaled retrieval filters.
Summaries dropped privilege markings so the output looked shareable.
Partners wanted a firm-wide “similar deals” mode that is incompatible with default matter isolation unless separately authorized.
How we tested
Two-user fixtures: matter A and matter B with distinctive phrases. Chat on A must never emit B’s phrases.
Test list, search, export, and AI answer with the same IDs.
External share links expire; privilege tags survive summarization (do not strip CONFIDENTIAL).
Audit who retrieved what.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| Engineering believed UI filters equaled retrieval filters. | Two-user fixtures: matter A and matter B with distinctive phrases. Chat on A must never emit B’s phrases. |
| Summaries dropped privilege markings so the output looked shareable. | Test list, search, export, and AI answer with the same IDs. |
| Partners wanted a firm-wide “similar deals” mode that is incompatible with default matter isolation unless separately authorized. | External share links expire; privilege tags survive summarization (do not strip CONFIDENTIAL). |
What we delivered
API / contract
- Postman — Search and chat APIs with swapped matter ids
LLM traces / evals
- LangSmith — Retrieved document ids vs allowed set
Journeys / UI
- Playwright — Share-link expiry and privilege labels
RAG quality
- Ragas — Answers only from allowed context
Outcomes
Chat retrieval used the same authorization check as GET /documents. A distinctive-phrase eval failed the suite on any leak.
Firm-wide similarity was a separate product flag, off by default, with an explicit client-consent path — not a hidden prompt.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| API / contract | Postman | Search and chat APIs with swapped matter ids |
| LLM traces / evals | LangSmith | Retrieved document ids vs allowed set |
| Journeys / UI | Playwright | Share-link expiry and privilege labels |
| RAG quality | Ragas | Answers only from allowed context |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.