This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
On well-known CVEs the prose was fine. On recent or tenant-specific findings, the model invented exploitability and “upgrade to X.Y.Z” that was not in the advisory.
Remediation steps included another tenant’s hostname from a retrieved similar ticket.
Benchmarks such as HalluCVE and surveys of “AI slop” in vulnerability assessment document pervasive fabricated CVE detail — especially after knowledge cutoff — delivered with high confidence. That wastes analyst time and can drive the wrong change.
Highlights
- External resolve for CVE ids
- Inventory-grounded impact
- Alert freshness vs prose
Typical launch bar: Wait until journeys and evals agree — do not ship on demo scores alone.
Challenges
Analysts started trusting the write-up format (CVSS-looking numbers).
The model recommended blocking a port that would have taken down a payment path — it had no change-window awareness.
Ingest lag meant the copilot described an alert as active after it was already auto-closed.
How we tested
CVE ids must resolve to NVD or the vendor advisory tool. Unresolved = refuse.
Impact on “your stack” only from the asset inventory tool, not from analogy.
Cross-tenant finding IDs return 404/403, including in summaries.
Analysts can mute/close/assign; the copilot cannot close Sev-1 without the same role.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| Analysts started trusting the write-up format (CVSS-looking numbers). | CVE ids must resolve to NVD or the vendor advisory tool. Unresolved = refuse. |
| The model recommended blocking a port that would have taken down a payment path — it had no change-window awareness. | Impact on “your stack” only from the asset inventory tool, not from analogy. |
| Ingest lag meant the copilot described an alert as active after it was already auto-closed. | Cross-tenant finding IDs return 404/403, including in summaries. |
What we delivered
LLM traces / evals
- LangSmith — Advisory tool vs invented CVSS
Adversarial evals
- Promptfoo — Unknown CVE and post-cutoff ids
API / contract
- Postman — Cross-tenant findings
Journeys / UI
- Playwright — Ticket lifecycle and gated close
Outcomes
Citator-style CVE resolve, inventory-grounded impact, and tenant canaries. Close-ticket is a gated tool.
We added freshness: status from the detection API with a timestamp in the UI.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| LLM traces / evals | LangSmith | Advisory tool vs invented CVSS |
| Adversarial evals | Promptfoo | Unknown CVE and post-cutoff ids |
| API / contract | Postman | Cross-tenant findings |
| Journeys / UI | Playwright | Ticket lifecycle and gated close |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.