This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
Buyer and seller UIs were separate. The embedding index was not. A seller prompt “summarize delayed orders” included another shop’s SKU and a buyer email.
Commission math in chat disagreed with the payout CSV.
Marketplaces fail on tenancy first. Adding a copilot without the same isolation as the REST API creates a new exfiltration path.
Highlights
- Canary tenancy in RAG
- Untrusted marketplace content
- Ledger vs chat reconciliation
Typical launch bar: Wait until journeys and evals agree — do not ship on demo scores alone.
Challenges
Support “god mode” for the copilot used a shared retrieval path.
CSV and chat used different timezone cutoffs for “this week’s payout.”
Storefront themes injected seller-written FAQs that contained prompt-injection payloads.
How we tested
Canary orders per shop. AI answers must not contain the other shop’s canary.
Payout figures from the ledger tool, not from model arithmetic.
Buyer cannot see seller notes; seller cannot see another seller’s PII.
Dispute/refund touching both sides.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| Support “god mode” for the copilot used a shared retrieval path. | Canary orders per shop. AI answers must not contain the other shop’s canary. |
| CSV and chat used different timezone cutoffs for “this week’s payout.” | Payout figures from the ledger tool, not from model arithmetic. |
| Storefront themes injected seller-written FAQs that contained prompt-injection payloads. | Buyer cannot see seller notes; seller cannot see another seller’s PII. |
What we delivered
API / contract
- Postman — Cross-shop IDOR on orders and payouts
Adversarial evals
- Promptfoo — Indirect injection in seller FAQs
LLM traces / evals
- LangSmith — Retrieved shop ids
Journeys / UI
- Playwright — Buyer vs seller vs support
Outcomes
Support mode still filtered retrieval by ticket shop unless a break-glass role was used and audited. FAQ chunks were treated as untrusted content (indirect injection tests).
Payout evals compared chat numbers to CSV with an explicit timezone.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| API / contract | Postman | Cross-shop IDOR on orders and payouts |
| Adversarial evals | Promptfoo | Indirect injection in seller FAQs |
| LLM traces / evals | LangSmith | Retrieved shop ids |
| Journeys / UI | Playwright | Buyer vs seller vs support |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.