This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
Row-level security was “on” in a screenshot. The generated client called `/api/items/:id` without a server check beyond “is logged in.”
Prompt-built checkout created an order on redirect, not on PSP confirmation.
AI app builders ship UI fast and often leave authorization, empty states, and webhook idempotency as leftovers. We test those leftovers first.
Highlights
- API-first tests that survive UI regen
- Webhook confirmation vs redirect
- Client-bundle secret scan
Typical launch bar: Wait until journeys and evals agree — do not ship on demo scores alone.
Challenges
No one owned the generated auth code. “The tool made it” is not a control.
Environment keys for the AI builder were in client bundles.
Each regenerate of the app shuffled routes and broke the first Playwright suite — we had to target roles and APIs, not brittle CSS.
How we tested
Auth, password reset, session expiry.
IDOR on every object type; tenant isolation if multi-workspace.
Empty, loading, error states. Mobile keyboard covering CTA.
PSP webhook out-of-order and duplicate.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| No one owned the generated auth code. “The tool made it” is not a control. | Auth, password reset, session expiry. |
| Environment keys for the AI builder were in client bundles. | IDOR on every object type; tenant isolation if multi-workspace. |
| Each regenerate of the app shuffled routes and broke the first Playwright suite — we had to target roles and APIs, not brittle CSS. | Empty, loading, error states. Mobile keyboard covering CTA. |
What we delivered
Journeys / UI
- Playwright — Signup, happy path, mobile CTA
API / contract
- Postman — IDOR, webhook idempotency
Security scan
- OWASP ZAP — Exposed debug and .env-ish endpoints
Dependency review
- Snyk — Dependency surprises in the generated lockfile
Accessibility
- axe DevTools — Public landing and checkout
Outcomes
Critical IDOR and double-entitle findings blocked launch. We left Newman collections that survive UI regen, plus a thin Playwright happy path.
Secret-in-bundle was High. The team moved keys server-side before we would call the app launch-ready.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| Journeys / UI | Playwright | Signup, happy path, mobile CTA |
| API / contract | Postman | IDOR, webhook idempotency |
| Security scan | OWASP ZAP | Exposed debug and .env-ish endpoints |
| Dependency review | Snyk | Dependency surprises in the generated lockfile |
| Accessibility | axe DevTools | Public landing and checkout |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.