This is not a theory essay. It is the order we use when time is limited and the cost of being wrong is high. You can run it yourself with staging access. You can also use it as a shared language with whoever owns the release.
- Test the product journey and the model behavior as one system, not two separate hobbies.
- Start with the highest-cost failure modes: wrong answers with confidence, unsafe tool calls, data exposure, and weak fallbacks.
- Golden prompts and a polished demo are not coverage.
- Use a simple order: map risk → probe journeys → probe model edges → check safety → check failure states → rank severity → decide ship / wait / block.
- If you cannot answer basic coverage questions, that is a finding, not a vibe.
- Write down what “good enough” means before you run cases. Otherwise every result becomes a debate.
Who this guide is for
You will get the most from this if:
- An AI feature is live, or about to be, for real users
- The product can read private data, take actions, or influence money / access / trust
- Your team is small or moving fast, and dedicated AI QA coverage is thin
- You have staging (or a realistic preview) and at least one person who did not write all the prompts
You may not need a full independent engagement yet. You still need a sequence that is more serious than “the demo looked good.”
Who this is not for
- Weekend experiments with no users and no sensitive data
- Teams that only want someone to rebuild the product (we test; we do not build)
- Releases that refuse to delay even when Critical issues are open and unnamed
Why AI launches fail after a clean demo
AI demos impress fast. Production fails quieter.
A common pattern looks like this:
- Leadership sees a walkthrough with three known prompts and a friendly account.
- Engineering confirms the happy path in staging.
- Someone notes “we should add more evals later.”
- Launch happens.
- Real users paste messy language, ask outside the corpus, trigger a tool on a weird permission path, or hit a timeout that shows a blank state.
None of that means the model vendor failed you. It means coverage was thin. The demo optimized for clarity. Production optimizes for whatever a distracted person types on a phone.
Classical QA still matters here. Auth bugs, billing bugs, and permission bugs do not become less important because there is a model in the middle. AI adds another layer: the model can be fluent while the product is wrong.
The principle: one system, not two hobbies
Teams often split work badly:
- “Prompt people” own answer quality in a notebook
- “QA people” own buttons and login
- Nobody owns what happens when a wrong answer drives a write action
Treat the product as one system:
- The UI promises something
- The model produces something
- Tools / retrieval / policies constrain something
- The user sees and acts on the result
A finding can live in any of those layers. Your test plan has to touch all of them.
What to test (in order)
1. Map what “bad” costs you
Before writing cases, spend thirty honest minutes on a half page:
- Actions: read, write, send, delete, pay, change permissions, call external APIs
- Data: user content, tenant data, system instructions, retrieved documents, logs
- Blast radius: support load, user trust, legal exposure, money, safety, other tenants
Rank the top three “if this goes wrong, we have a bad week” scenarios. Those go first. Everything else waits.
Example (generic product pattern, not a client story): if the assistant can send email on the user’s behalf, “wrong recipient / wrong body / no confirmation” outranks “tone is slightly off.”
If you skip this step, you will spend hours polishing low-cost issues and miss the expensive ones.
2. Walk the product journeys that matter
Treat this like serious product QA, not only prompt play.
Minimum journey set:
- Signup / login / role boundaries (admin vs member vs guest)
- The main AI-backed action users actually pay for
- Empty states, loading, retries, and “model said no”
- Billing or plan limits if they gate model features
- A second device or narrow viewport if that is where users live
What to watch for:
- The UI claims the model did something it did not do
- Errors are swallowed into a generic “something went wrong”
- Role A can trigger tools meant for role B
- A free plan can reach a paid model path without a clear gate
If the UI lies about what the model did, that is a product bug even when a raw model response looked “fine” in isolation.
3. Probe model behavior on purpose
Golden prompts are necessary. They are not sufficient.
Build a small set you can re-run after every prompt or model change. Include:
- Ambiguous asks (“fix this,” “make it better,” “what should I do?”)
- Out-of-scope asks (things you explicitly do not support)
- Contradictory instructions in one thread
- Long, messy, multi-turn context with typos and half thoughts
- Retrieval misses (if RAG): questions your corpus cannot answer
- Structured output breaks: missing fields, invalid JSON, wrong enums
- “Confident wrong” traps: questions where a plausible answer is worse than a refusal
Before you run the set, write the pass rule in one line per case. Example: “Must refuse and offer human handoff” or “Must cite a source or say it does not know.” If you skip that, every answer becomes a meeting.
Depth tip: ten well-specified cases beat fifty vague ones.
4. Check safety and permission edges
At minimum, for products that see private data or can act:
- Prompt injection / instruction-override attempts appropriate to your threat model
- Attempts to pull system instructions or another user’s / tenant’s data
- Tool calls that should require confirmation (send, delete, pay, share)
- Role or tenant boundaries around retrieved context
- Logging: secrets and raw PII should not land in places you would not put in a support ticket
You do not need a novel-length red team. You need a documented set you can re-run. Undocumented “we tried a few jailbreaks once” disappears the week someone changes the system prompt.
5. Failures and fallbacks
Ask one blunt question: when the model times out, refuses, or returns junk, what does the user see?
Strong patterns:
- Clear retry with state preserved
- Degraded mode that still lets the user finish the job another way
- Human handoff with context
Weak patterns:
- Blank screen or infinite spinner
- A confident wrong answer with no escape hatch
- “Sorry” with no next step
- Silent retries that burn cost and latency
Fallbacks are features. Test them like features. Force timeouts and empty retrieval in staging on purpose.
6. Rank what you found
Not every issue is a launch blocker. Ranking prevents two failure modes: shipping Critical issues because “everything is a finding,” and delaying forever because “everything is Critical.”
A simple language that works:
- Critical: realistic path to data exposure, unsafe write, or major user harm
- High: broken core journey or repeated wrong answers on primary use cases
- Medium: real issue with limited blast radius or workaround
- Low: polish, edge copy, unlikely path
If you want a shared helper for that language, use our severity classifier.
7. Decide with a gate, not a feeling
End with one of three outcomes:
- Ship — critical probes done; open issues ranked with owners
- Wait — coverage partial; keep blast radius small while you close gaps
- Block — data exposure, ungated writes, or missing fallbacks still unknown on a user-facing flow
“We feel ready” is not a gate. Neither is “sales needs it Friday” without a named person accepting residual risk.
Helper: go / no-go tool. Longer write-up: go / no-go criteria guide.
A one-week self-run schedule (if you are alone)
If you have five focused half-days before launch:
- Day 1: Risk map + journey list + pass rules for ten model cases
- Day 2: Journey testing (auth, core AI action, billing gates, mobile)
- Day 3: Model set + RAG miss cases + structured output
- Day 4: Safety / permission probes + forced fallback tests
- Day 5: Severity rank, owners, ship / wait / block, and a short written note to the team
If Day 5 still has Critical items open, the calendar is the problem, not the method.
What “good enough” evidence looks like
You do not need a hundred-page report to trust a launch. You do need artifacts someone else can re-check:
- The risk half-page
- The journey list with pass/fail notes
- The model/safety case set with pass rules
- A severity list with owners and dates
- A written ship / wait / block call
If those five things exist, you have a conversation. If they do not, you have optimism.
How GenCodeQA approaches it
We test. We do not build. That boundary matters: the job is an accurate picture of risk, not a larger build contract.
In practice we combine:
- Product journeys (auth, permissions, billing, device)
- AI-native checks (prompts, retrieval, tools, guardrails)
- Severity-ranked findings with location and what “fixed” looks like
- Retest support as agreed
We start with minimum necessary access: usually read-only repo or staging. NDA available first. Method overview: AI evaluation framework. Services: what we test.
Common false confidence
- “The demo went well.”
- “Our eval set is green” (when the eval set is ten friendly prompts).
- “We use a safe model” (model choice is not product QA).
- “We’ll watch production” (monitoring is not a substitute for pre-launch probes).
- “It’s only an internal tool” (internal users still hit permissions and data edges).
- “The vendor ran red teaming” (their model card is not your product test).
- “We’ll fix it in the next sprint” (without a gate, next sprint becomes never).
Free tools to run before you hire anyone
- Coverage gap finder — tick what you already cover
- Launch risk score — urgency self-check
- 24-point launch checklist — owners and tracking
- Go / no-go — ship / wait / block language
No signup. Use them as homework for your launch thread.
When to get a second pair of eyes
Consider an independent pass when:
- You are days from launch and Critical questions are still unanswered
- The product handles logins, payments, personal data, or model-driven writes
- The people who built the prompts are also the only people “testing” them
- Your team disagrees on Ship vs Wait and needs an outside read
Book a free testing triage. Bring the product type, model setup, what you have already tested, and the date you are trying to protect. We come back with scope, timeline, and the first things we would check. No invented urgency from us.
FAQ
Do I need production access to start testing?
No. Staging plus clear roles is enough for a first pass. Read-only repo access helps when flows are hard to infer from the UI alone.
Is this only for LLM chat UIs?
No. Copilots, agents with tools, RAG assistants, and AI features inside SaaS all need journey + model coverage together.
How is this different from running Promptfoo or LangSmith?
Eval tooling helps automate checks and regressions. It does not replace product QA, permission checks, or fallback design. Use both. See our testing tools notes.
What if we built the app with AI coding tools?
Same failure modes still apply. Speed of build does not create test coverage. Start with the gap finder and this sequence; a dedicated AI-built apps guide is on the later calendar.
How many prompts is “enough”?
Enough to cover your top-cost failure modes with written pass rules, plus a few adversarial and miss cases. Quality of specification beats raw count.
Can we ship with open Medium findings?
Sometimes, with named owners and monitoring. Critical / High on data exposure or ungated writes usually means Wait or Block.