It can.
But RAG also makes a specific kind of failure easy to miss: answers that feel grounded because there are citations—while the citations do not actually support the claims, or the system retrieved the wrong documents for the user.
If you want public trust, you need public checks. Here are the ones you can verify without pretending.
What to verify (not just what to feel)
1) Are the citations honest?
Ask a reviewer question:
“If we removed the citations, would the same claims still be supported by the retrieved chunks?”
If the answer is no, you do not have citation honesty. You have citation decoration.
2) Do the claims map to the retrieved content?
Separate the test:
- citations exist
- claims are supported
Sometimes both are true. Sometimes citations exist but claims are not supported. Sometimes the system cites something else entirely.
The product needs the second check.
3) What happens on retrieval misses?
Create cases where the system should not find anything relevant:
- out-of-corpus questions
- ambiguous queries that retrieve low-quality chunks
- “correct question, missing doc”
Then verify the fallback:
- does the system refuse or ask a clarifying question?
- or does it guess confidently anyway?
4) Can a user see or be influenced by the wrong tenant/doc set?
RAG is still a data boundary system.
If your retrieval layer is wrong, you can get both confidentiality problems and hallucinations that look “reasonable” because they are grounded in the wrong context.
A simple way to structure your tests this week
If you only have one afternoon, do this:
- Pick one core user journey (the main question path).
- Write 8 test cases:
- 3 that retrieve correctly
- 2 retrieval misses (empty or low-confidence)
- 1 conflicting-docs case
- 2 access boundary probes (tenant/role)
- For each case, write one sentence as the pass rule:
- “Must refuse when no retrieved support exists.”
- “Must cite chunks that actually support the claim.”
- “Must not use unauthorized documents.”
That is real evidence. It is also small enough to re-run.
Where to go next
If you want the bigger launch sequence (journeys + model probes + safety + fallbacks), use the guide:
How to test for RAG hallucinations
If you want a quick view of what you might be missing, start with the free coverage gap finder.
And if you are days from launch and cannot be honest about retrieval misses and fallbacks, book a free triage.