This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
Card price and detail price drifted after a feed update. The assistant used the stale card embedding.
“Good schools” and crime commentary appeared without a data source — both are fair-housing-sensitive.
Sold listings still had an “ask for a tour” CTA in chat.
Wrong price, stale status, or invented property facts destroy trust and create fair-housing and misrepresentation risk. Map/list desync is the classic non-AI twin of the same problem.
Highlights
- Field-grounded facts
- Status-aware CTAs
- As-of timestamps on laggy feeds
Typical launch bar: Wait until journeys and evals agree — do not ship on demo scores alone.
Challenges
MLS lag meant the truth was minutes behind; the assistant spoke in the present tense.
Geocodes for new builds put pins in the wrong neighborhood, and the model narrated that neighborhood.
Agents wanted the bot to “sell the listing.”
How we tested
Facts (price, beds, HOA, status) from listing fields or refuse.
Neighborhood commentary marked non-authoritative or omitted.
Map bounds, filters, and list pane stay in sync; sold/off-market cannot be treated as available.
Lead routing isolation; buyer A cannot see buyer B’s inquiry.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| MLS lag meant the truth was minutes behind; the assistant spoke in the present tense. | Facts (price, beds, HOA, status) from listing fields or refuse. |
| Geocodes for new builds put pins in the wrong neighborhood, and the model narrated that neighborhood. | Neighborhood commentary marked non-authoritative or omitted. |
| Agents wanted the bot to “sell the listing.” | Map bounds, filters, and list pane stay in sync; sold/off-market cannot be treated as available. |
What we delivered
Journeys / UI
- Playwright — Search, map, listing, lead form
API / contract
- Postman — Feed ingest, status, lead IDOR
Adversarial evals
- Promptfoo — Invented HOA/flood/school facts
RAG quality
- Ragas — Answers grounded in listing JSON
Outcomes
Present-tense facts required a fresh field read. Stale feed showed “as of” in the UI. Chat CTAs respected status.
Playwright: filter → open listing → attributes match. API: agent B cannot PATCH agent A’s listing. Promptfoo: invented HOA.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| Journeys / UI | Playwright | Search, map, listing, lead form |
| API / contract | Postman | Feed ingest, status, lead IDOR |
| Adversarial evals | Promptfoo | Invented HOA/flood/school facts |
| RAG quality | Ragas | Answers grounded in listing JSON |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.