How to test for RAG hallucinations (and not fool yourself)

It can reduce them.

It can also create a new failure mode: confident answers that are grounded in the wrong documents, stale documents, or missing context—and the UI still looks “helpful.”

This guide is a practical sequence to test for RAG hallucinations in a way that you can re-run after retrieval, prompts, or corpora change.

What counts as a “RAG hallucination”?

In a RAG product, a “hallucination” is not only “the model invented facts.”

It is also:

• The model answers something the system cannot justify with retrieved sources.
• The model answers with retrieval that is wrong for the user (wrong tenant, wrong doc set, wrong role).
• The model answers with stale information that should not be used anymore.
• The model generates confident citations that are misleading (citation text exists, but does not support the claim).
• The model refuses when it should answer, or answers when it should say it does not know.

If your UI shows citations (or “sources”), hallucination testing is mostly about whether citations actually support the claims.

Why this breaks in production

RAG failures are easy to miss in demos because demos often:

• Use clean queries that retrieve well.
• Use a small subset of documents.
• Use a single tenant or a single role.
• Avoid the edge cases (empty retrieval, conflicting docs, ambiguous questions).
• Skip the “what happens when retrieval is wrong” UX.

Production introduces messy input:

• Users ask in natural language and with partial context.
• They ask outside the corpus.
• They ask for documents they should not access.
• They run the same conversation days later, after docs have changed.

If the product’s fallback is weak, users will still get an answer—confidently wrong, just with a citation block.

The testing sequence

1) Start with a quick risk map

Write down what can go wrong in four categories:

1. Wrong grounding (citations do not support claims)
2. Missing grounding (no citations, or citations exist but are empty/irrelevant)
3. Access mistakes (user sees or is influenced by documents they should not)
4. UX/fallback failures (user gets an answer when the product should refuse/redirect)

Then decide which journeys matter:

• the primary user question flow
• any “search inside the chat” flow
• document upload or “connect your docs” setup flows (if applicable)

2) Test citation honesty and grounding separately

Many teams test only one thing: “does it sound right?”

For RAG, you want two checks:

• Citation honesty: are the sources present and relevant?
• Claim grounding: do the sources actually support the answer text?

You can do this with human review for a small set and then codify the pass rule for regression.

Pass rule examples you can write:

• “If the answer references a source, each key claim must be supported by at least one retrieved chunk.”
• “If no retrieved chunks match the question intent, answer must either refuse or explicitly say it cannot find support.”
• “Citations must not be fabricated when retrieval is empty.”

3) Force retrieval miss cases

Do not wait for retrieval misses to happen in production.

Write cases where:

• the question is intentionally outside your corpus
• the query uses ambiguous phrasing that causes low similarity
• the query is correct but the relevant document was not yet indexed
• the query requires a doc the product should exclude based on access rules

Then check behavior:

• Does the system stop and say it does not know (or ask a clarifying question)?
• Does it “guess” anyway?
• Does it cite something unrelated?

These miss cases are where trust is either earned or broken.

4) Test multi-turn drift and contradiction

RAG products often re-rank or re-retrieve on each turn. That is good.

It is also where drift can appear:

• The model changes its story mid-thread after retrieval changes.
• The model continues using earlier retrieved context even when retrieval would change.

Test simple multi-turn sequences:

• Ask a question, then ask a follow-up that shifts the intent.
• Ask for a correction of a prior claim (“are you sure?”).
• Ask the model to reconcile two partially conflicting answers.

Watch for:

• contradictions without updated citations
• refusal when citations exist (UX failure)
• confident correction that is not supported by newly retrieved sources

5) Test access boundaries (tenant + role isolation)

If your corpus is multi-tenant or role-gated:

• Ensure retrieval never pulls across boundaries.
• Ensure the model never gets “helpful” context it should not see.

Write probes like:

• “What is in this user’s private doc?” (role A vs role B)
• “Compare tenant A policy vs tenant B policy.”
• “Summarize a doc ID that role B should not access.”

Pass rule examples:

• “No answer should include any tokens drawn from unauthorized documents.”
• “If unauthorized documents are requested, the system must refuse or redirect without citing.”

This is not a theoretical requirement. It’s a RAG core requirement.

6) Verify fallback UX when retrieval is empty or low confidence

Fallback is part of RAG quality.

Test explicitly:

• empty retrieval (no chunks)
• low similarity retrieval (chunks exist but may not support the claims)
• conflicting chunks (two docs disagree)

Then decide what “good” looks like:

• refusal with a next step
• clarifying question with what the system can search
• a response that states uncertainty and avoids confident invention

If the UI always renders an answer bubble, users will treat it as truth.

How GenCodeQA approaches it

We test RAG as a whole system:

• product journey (auth, permissions, data boundaries, UX)
• retrieval checks (coverage and access)
• generation checks (grounding + citation honesty)
• failure UX (timeouts, empty retrieval, inconsistent outputs)

Findings are severity-ranked with clear “fix guidance” and re-test support.

Method overview: AI evaluation framework.

If you want a quick weighted view of gaps: coverage gap finder.

Free tool: start with gaps

Coverage gap finder (no signup)

Use it with a teammate who did not write your retrieval prompt set.

When to get a second pair of eyes

If you are days from launch, handle login + private data, or you cannot honestly answer “what do we do when retrieval misses?”, an independent pass can turn uncertainty into a scoped test plan.

Book a free testing triage.

FAQ