This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
Summaries included quotation marks around paraphrases. Video captions named a speaker who was not on screen.
Scheduled posts went live from a draft slug that was never gated. The model’s SEO title for a draft was indexable.
Publishers already fight training-data and attribution disputes. Invented quotes and captions presented as if they were in the asset are an editorial-standards failure, not a “creative” feature.
Highlights
- Substring quotes vs paraphrase labels
- Preview-token cache checks
- Rights as a tool, not a prompt
Typical launch bar: Wait until journeys and evals agree — do not ship on demo scores alone.
Challenges
Editors liked punchy quotes more than accurate ones.
CDN cached a draft HTML once it had been fetched with a preview token.
Rights metadata was in a DAM the retriever did not see, so the assistant cleared a clip for social that was embargoed.
How we tested
Quoted text must be a substring of the source (or marked paraphrase).
Captions labeled generated; people tags from a verified roster or omitted.
Logged-out request to draft slug is 404/401.
Paywall/entitlement: the assistant must not summarize behind the paywall for anonymous users.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| Editors liked punchy quotes more than accurate ones. | Quoted text must be a substring of the source (or marked paraphrase). |
| CDN cached a draft HTML once it had been fetched with a preview token. | Captions labeled generated; people tags from a verified roster or omitted. |
| Rights metadata was in a DAM the retriever did not see, so the assistant cleared a clip for social that was embargoed. | Logged-out request to draft slug is 404/401. |
What we delivered
Journeys / UI
- Playwright — Draft leak, paywall, publish pipeline
RAG quality
- Ragas — Summary faithfulness
Adversarial evals
- Promptfoo — Invented quote probes
Front-end performance
- Lighthouse — Popular article start time — secondary to leak tests
Outcomes
Quote-substring evals and draft-URL tests were launch-blocking. Entitlement applied to the summarize API, not only the article page.
Embargo/rights became a tool check before social scheduling.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| Journeys / UI | Playwright | Draft leak, paywall, publish pipeline |
| RAG quality | Ragas | Summary faithfulness |
| Adversarial evals | Promptfoo | Invented quote probes |
| Front-end performance | Lighthouse | Popular article start time — secondary to leak tests |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.