This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
List APIs filtered by org. The digest job queried a warehouse view that used org_id incorrectly for one report type.
API keys for scanners were visible in a debug panel the copilot could quote when asked “how do I rotate secrets?”
Security products are judged on tenant isolation of findings. An LLM digest is still an export path.
Highlights
- Canaries on every export path
- One authz function
- Secret non-echo tests
Typical launch bar: Wait until journeys and evals agree — do not ship on demo scores alone.
Challenges
The warehouse team did not consider themselves on the copilot’s authz path.
“Explain this finding” retrieved similar write-ups globally to “improve quality.”
Export CSV and AI digest used different access code paths.
How we tested
Canary finding titles per tenant in every digest, export, and chat path.
Keys stored hashed/wrapped; UI and copilot must not echo secrets.
RBAC on destructive actions (mute all, delete integration).
Ingest burst: lag within SLA, no silent drop if at-least-once is claimed.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| The warehouse team did not consider themselves on the copilot’s authz path. | Canary finding titles per tenant in every digest, export, and chat path. |
| “Explain this finding” retrieved similar write-ups globally to “improve quality.” | Keys stored hashed/wrapped; UI and copilot must not echo secrets. |
| Export CSV and AI digest used different access code paths. | RBAC on destructive actions (mute all, delete integration). |
What we delivered
API / contract
- Postman — Cross-tenant finding ids
LLM traces / evals
- LangSmith — Digest retrieval set
Adversarial evals
- Promptfoo — Secret-echo and jailbreaks
Load / soak
- k6 — Ingest burst and backpressure
Dependency review
- Snyk — Dependency issues in the scanner agents
Outcomes
Same authorization function for list, export, digest, and chat. Global “similar write-up” retrieval was disabled until it was org-scoped.
Secret-echo tests in Promptfoo. k6 on ingest with poison messages.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| API / contract | Postman | Cross-tenant finding ids |
| LLM traces / evals | LangSmith | Digest retrieval set |
| Adversarial evals | Promptfoo | Secret-echo and jailbreaks |
| Load / soak | k6 | Ingest burst and backpressure |
| Dependency review | Snyk | Dependency issues in the scanner agents |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.