A plan has owners, edges, failure states, and written pass rules. Golden prompts have familiarity.
Why teams lean on golden prompts
They are easy to demo. They regress quickly. They make stakeholders feel progress. In early building, that is rational.
The problem starts when the same five prompts become the entire definition of “tested” for an AI feature that can read private data or take actions.
At that point you are regression-checking the demo, not the product.
What golden prompts usually miss
Ambiguity. Real users do not speak like your README.
Out-of-scope asks. Products need a refusal and redirect story, not improvisation.
Retrieval misses. If you use RAG, “I don’t know” and citation honesty matter as much as happy hits.
Adversarial pressure. Instruction overrides and data exfil attempts are not insults. They are foreseeable inputs.
Structured output breakage. Missing fields and invalid JSON break downstream UI and tools.
Multi-turn drift. Answer one looked fine. Turn seven quietly changed the user’s goal.
The product around the model. Auth, permissions, billing gates, and fallbacks never appear in a prompt table.
If your “AI QA” is replaying the same friendly prompts after every change, those gaps stay invisible until production.
What a real prompt case looks like
Upgrade from:
- Prompt: “Summarize this doc”
- Note: “Looks good”
To:
- Setup: member role, tenant A doc only
- Prompt: ambiguous summarize request on a doc the user should not see
- Pass rule: refuse or block; no content from tenant B; no system prompt leakage
- On fail: severity, owner, retest note
Ten cases like that beat fifty vague “try this” lines.
Build a starter set in one afternoon
If you have nothing beyond golden prompts today, create four buckets:
- Core job — three cases users pay for
- Ambiguity — three messy phrasings of the same job
- Miss / refuse — three asks that should not be answered confidently
- Pressure — three injection or permission-boundary attempts appropriate to your threat model
Add setup + pass rule to each. Run them on staging with a non-author reviewer. You will learn more in two hours than in another week of polishing demo copy.
Keep the golden set. Promote it from “the plan” to “one layer of the plan.”
Put model behavior on the same checklist as auth
Answer quality is not a side quest. It belongs next to login and billing because users experience one product.
We explain what belongs on that combined list, and how to assign owners, in this guide:
AI app launch checklist (model behavior included)
The interactive list is here if you want to start today: 24-point launch checklist.
A simple weekly habit
After any material prompt, model, tool, or retrieval change:
- Re-run the golden set (keep it).
- Re-run five edge / adversarial / miss cases with pass rules.
- Force one failure path (timeout or empty retrieval).
- Update open severity items.
That is still lightweight. It is just no longer theater.
Where teams get stuck
They argue about whether an answer is “good” without a pass rule. They treat vendor model cards as product tests. They let the people who wrote the prompts also be the only judges.
Split the roles when you can. At minimum, have one skeptical teammate answer the checklist honestly.
What “looks good” hides
Without a pass rule, “looks good” usually means:
- Fluent tone
- Roughly on topic
- No obvious crash
It often ignores:
- Silent permission mistakes
- Missing uncertainty
- Wrong structured fields that break the next tool call
- Citations that sound real and are not
Fluency is not correctness. Your cases have to say which one you are scoring.
If you need the longer sequence
Golden prompts sit inside a broader launch method:
How to test an AI app before launch
We test. We do not build. If you want an independent read of your cases and gaps, book a free triage.