The AI app launch checklist that covers model behavior

Why generic QA checklists fall short

Classic QA still matters. Keep auth, permissions, billing, empty states, and mobile.

What classic lists often skip:

• Answer quality outside the demo set
• Retrieval misses and citation honesty (if you use RAG)
• Tool calls and approval gates
• Prompt injection and data exposure attempts
• Fallback when the model fails
• Regression after someone “just changed the prompt”

If your checklist has none of those rows, you are launching an AI feature with a non-AI test plan. That is how confident wrong answers and ungated writes reach production while the UI checklist looks green.

What belongs on an AI launch checklist

Group items so ownership is obvious.

A. Product journeys

• Critical path works for a real role (not only admin)
• Permissions match what the model can see and do
• Billing / plan gates behave if they control model access
• Error and empty states are intentional
• Mobile / narrow layouts do not hide confirmations on write actions

B. Model behavior

• Documented prompt / eval set beyond golden demos
• Ambiguous and out-of-scope asks handled in an agreed way
• Structured outputs validated when the product depends on them
• Multi-turn drift checked on the flows you care about
• “Say you don’t know” behavior tested where guessing is expensive

C. Safety and data

• Injection / exfil style probes appropriate to your product
• Tenant / role boundaries around context
• Logging does not store secrets you would not want in a ticket
• Tool permissions reviewed for least privilege

D. Failures and operations

• Timeouts, refusals, and junk outputs have a designed UX
• Cost / retry loops will not silently melt the budget on a bad day
• You know how you will retest after model or prompt changes
• On-call knows what “AI feature broken” looks like in logs and user reports

You do not need a hundred rows. You need honest coverage of the rows that match your risk map. Cut what does not apply. Do not cut safety, permissions, and fallbacks just to feel faster.

Ownership: the part most teams skip

A checklist without owners is a wish list.

Suggested split:

• Product owns journey pass criteria and “good enough” language for answers
• Engineering owns tools, permissions, fallbacks, logging, and regressions
• Whoever owns the model config owns the eval / adversarial set and re-runs after prompt changes

Write the name next to the group, not “the team.” If two people own it, nobody owns it.

How to use the GenCodeQA 24-point checklist

We published the interactive list we work through before AI products go live:

24-point AI launch checklist (free, no signup)

How to use it well:

1. Copy it into your issue tracker or doc the same day you open it.
2. Assign an owner per group.
3. Mark items you cannot answer as open, not done.
4. Anything open that touches write actions, private data, or unsafe tools is launch-critical until resolved or explicitly accepted by a named person.
5. Re-open AI rows after every material prompt, model, tool, or retrieval change.

The note on the tool page is intentional: people often get stuck around item four or five. That stuck feeling is the point. It shows where confidence ends.

Example of honest vs optimistic checking

Optimistic: “Safety — done” because someone tried one jailbreak joke in chat.

Honest: “Safety — open. We have three injection cases written; two pass; tenant isolation not probed yet. Owner: A. Revisit Thursday.”

Only the second version helps a release decision.

A sample week using the checklist

Assume you have a support-style assistant that drafts replies and can optionally insert them into a ticket.

Monday: Copy the 24 points. Cut rows that truly do not apply. Assign Product to journeys and answer quality, Eng to tools/fallbacks/logging, Model owner to eval/adversarial set.

Tuesday: Run journey rows for member vs admin. Note any place the UI claims a draft was “saved” when the model call failed.

Wednesday: Expand the model set beyond golden prompts. Add ambiguous asks, out-of-scope asks, and one retrieval miss if docs are involved. Write pass rules before scoring.

Thursday: Safety and write gates. If insert-to-ticket is one click, treat confirmation and undo as first-class tests. Probe whether another tenant’s context can appear.

Friday: Mark open rows honestly. Severity-rank anything still open. Make a ship / wait / block call for the intended audience. Put the note in the launch channel.

If Friday still has Critical data or write-path unknowns, do not negotiate the checklist. Change the launch.

Evidence worth attaching to a checked row

A checked row should point to something another person can re-check:

• Link to an eval case ID or spreadsheet row
• Staging notes with role used and date
• Screenshot or trace for a fallback path
• PR that added an approval gate
• Short “retested after prompt change on DATE” note

“Discussed in standup” is not evidence. Neither is “seems fine.”

How the checklist fits the rest of the launch system

Use tools in this order when you are short on time:

1. Gap finder — see what is missing
2. This checklist — track and assign
3. Risk score — urgency conversation
4. Go / no-go — decide

Full sequence: How to test an AI app before launch.

How GenCodeQA uses checklists in engagements

A checklist is a map, not the report.

In a validation pass we:

• Confirm which journeys and AI behaviors are in scope
• Probe the highest-cost gaps first
• Write severity-ranked findings with location and fix guidance
• Retest what you claim is fixed, when that is part of the engagement

We do not pad the list to look busy. If an item is not relevant to your product, it stays out of scope in writing. We also do not treat a client-completed checklist as evidence that probes happened. Boxes are claims. We verify.

Independence matters here. Because we do not build the product, we are not incentivized to keep rows open to sell more implementation. The goal is a clear picture, including “this area looks solid.”

Red flags we see on “completed” checklists

• Every AI row checked the morning of launch with no attachments
• Safety checked by the same person who wrote the system prompt, with no second reader
• Fallbacks marked done because “the model rarely fails”
• Mobile unchecked on a product where confirmations hide behind overflow menus
• No re-open rule after prompt changes

If your checklist shows those patterns, treat the document as a draft, not a gate.

What to do with open items near a launch date

Open items need one of three treatments:

1. Close with evidence (retest note, screenshot, eval row, PR link)
2. Accept residual risk in writing (named person, date, why blast radius is limited)
3. Change the launch (narrow audience, feature flag, remove write actions, delay)

What does not count:

• “We’ll watch it”
• “It’s probably fine”
• Moving the row to a backlog with no owner

If several AI safety or write-action rows are still open and the date is fixed, the honest move is Wait or Block, not a quieter Ship.

Common false confidence

• Checking a box because “we talked about it once”
• Treating vendor safety claims as your product test
• Only testing with the team that built the prompts
• Skipping fallbacks because “the model is reliable”
• Calling the checklist done the morning of launch with half the AI rows blank
• Using the checklist as a customer-facing certificate

Free tool

Open the 24-point checklist

Related: coverage gap finder for a weighted view of what is missing, and severity classifier when open items need ranking.

When to get a second pair of eyes

If your launch date is fixed and several AI rows are still open, an independent pass is often the fastest way to turn “we think it’s fine” into a scoped list with severity and owners.

Book a free testing triage. Bring the checklist export or screenshots of open rows. That is enough to start a useful conversation.

FAQ