This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
Spam detection was decent. Borderline defamation was auto-cleared because the model optimized for “keep the conversation.”
A copyright takedown in the legal queue was summarized to editors without the attached evidence file, then marked resolved.
Moderation products fail when automation has write access without the same policy as humans — especially around defamation, CSAM, and copyright claims.
Highlights
- Separate scoring for rare severe classes
- No auto-close on legal/child-safety
- PII minimization in moderator copilots
Typical launch bar: Wait until journeys and evals agree — do not ship on demo scores alone.
Challenges
Precision/recall dashboards hid the legal-hold misses because volume was small.
Multilingual comments bypassed an English-only classifier then hit a generative “translate and decide” step that changed meaning.
The copilot could see reporter email addresses in the queue.
How we tested
Policy fixtures: spam, harassment, legal hold, child-safety (must hard-fail to human).
Auto-publish allow-list only. Legal hold cannot be closed by the model.
Comments on drafts stay unpublished.
Moderator RBAC.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| Precision/recall dashboards hid the legal-hold misses because volume was small. | Policy fixtures: spam, harassment, legal hold, child-safety (must hard-fail to human). |
| Multilingual comments bypassed an English-only classifier then hit a generative “translate and decide” step that changed meaning. | Auto-publish allow-list only. Legal hold cannot be closed by the model. |
| The copilot could see reporter email addresses in the queue. | Comments on drafts stay unpublished. |
What we delivered
Adversarial evals
- Promptfoo — Policy fixtures including jailbroken UGC
Journeys / UI
- Playwright — Queue, hold, publish, unpublish
API / contract
- Postman — Moderator RBAC
LLM traces / evals
- LangSmith — What the model saw vs what was auto-written
Outcomes
Rare-but-severe queues were scored separately from spam accuracy. Child-safety and legal hold had no auto-close tool.
Reporter PII stripped from the model context. Translation-then-decide required a human on legal categories.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| Adversarial evals | Promptfoo | Policy fixtures including jailbroken UGC |
| Journeys / UI | Playwright | Queue, hold, publish, unpublish |
| API / contract | Postman | Moderator RBAC |
| LLM traces / evals | LangSmith | What the model saw vs what was auto-written |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.