You built a RAG system. You embedded your docs, wired up a vector store, stuffed the top chunks into the prompt, and shipped. Then the questions came back wrong — confidently, fluently wrong — and your first instinct was to blame the model or reach for a bigger one.
Here's the uncomfortable truth after debugging a lot of these: the model is almost never the problem. In a Retrieval-Augmented Generation pipeline, the model can only answer from what you put in its context. If retrieval hands it the wrong chunks, even a frontier model will produce a beautifully written wrong answer. RAG failures are retrieval failures wearing a generation costume.
This is a field guide to the seven failure modes that cause most bad RAG answers, how to recognize each one, and the concrete fix. No new framework to adopt — just the debugging order that actually finds the bug.
First, the one diagnostic that saves you hours
Before touching anything, answer one question: was the correct information in the retrieved chunks at all?
Log the actual chunks your retriever returned for a failing query and read them with your own eyes. This splits every RAG bug into two worlds:
- The answer was not in the retrieved chunks → you have a retrieval problem (failure modes 1–5 below). The generator never stood a chance.
- The answer was in the chunks but the model ignored or mangled it → you have a generation problem (failure modes 6–7).
Skipping this step is why people waste days tuning prompts when the real bug is that the right paragraph was never retrieved. Do this first, every time.
One more split worth making early: is the question single-hop ("what is our refund window?") or multi-hop ("how does our refund window compare to the EU default?"). Multi-hop questions need facts from several documents at once, and a top-k retriever tuned for single-hop will reliably fetch one of the two needed pieces and miss the other. If your failures cluster on comparative or multi-part questions, the fix isn't a better embedding — it's query decomposition (break the question into sub-queries, retrieve for each, then synthesize), which is a different lever than everything below.
Failure mode 1: Your chunking destroys meaning
The single most common cause of bad retrieval is naive chunking — splitting documents every N characters with no regard for structure. You end up with chunks that cut a sentence in half, separate a heading from its content, or split a table from the row that gives it meaning. An embedding of half a thought retrieves poorly because it is half a thought.
Symptoms: retrieved chunks are topically close but never contain the complete fact; answers are vague or stitched from fragments.
The fix: chunk along the document's natural structure, not a fixed byte count.
- Split on semantic boundaries — paragraphs, sections, Markdown headings — not blind character windows.
- Add overlap (e.g. 10–20%) so a fact straddling a boundary survives in at least one chunk.
- Keep chunks coherent: one chunk should answer one thing. For structured docs, keep a table or list with its caption.
- Attach metadata to each chunk (source, section title, date) — you'll use it for filtering and citations later.
There's no universal chunk size; it depends on your documents and your embedding model's sweet spot. Measure it (see failure mode 7), don't guess. Chunking is deceptively deep — it's where a structured course on RAG and retrieval-augmented generation spends real time, because getting it right fixes more bugs than any prompt tweak.
Failure mode 2: The wrong embedding model (or a mismatch)
Your embedding model decides what "similar" means. Two failures hide here.
First, a weak or generic embedding model can't tell apart concepts that matter in your domain — legal clauses, medical terms, internal product names all collapse into the same neighborhood. Second, and more insidious: a mismatch between query and document embeddings. If you embed documents with one model and queries with another, or you embed a terse 6-word question and expect it to land near a dense 400-token passage, the vectors live in different worlds.
Symptoms: retrieval returns plausibly-related-but-not-relevant chunks; performance is mediocre across the board rather than failing on specific queries.
The fix:
- Use a strong, current embedding model suited to your domain and language — for non-English content, verify it was actually trained for it.
- Embed queries and documents with the same model and the same preprocessing.
- Consider query transformation: rewrite the short user question into a richer form (or generate a hypothetical answer and embed that — the HyDE technique) so it sits closer to real passages.
Failure mode 3: Pure vector search misses exact terms
Dense vector search is great at semantics and surprisingly bad at exact matches — error codes, SKUs, function names, dates, proper nouns. A user asking about ERR_CONN_4021 doesn't want "semantically similar" connection errors; they want that string. Embeddings smear it into a neighborhood and the exact doc ranks fifth.
Symptoms: great on conceptual questions, weak on specific identifiers, names, or numbers.
The fix: hybrid search. Combine dense vector search with classic keyword search (BM25/full-text) and fuse the results (Reciprocal Rank Fusion is a simple, strong default). You get semantics and exact-match precision. Most production-grade vector stores support hybrid retrieval now — turn it on.
Failure mode 4: No re-ranking, so the best chunk isn't on top
First-stage retrieval optimizes for recall — cast a wide net and return, say, the top 50 candidates. But you only feed a handful to the model, and the ordering of that first stage is coarse. The genuinely best chunk is often in your top 50 but not your top 5, so it never reaches the prompt.
Symptoms: when you log the top 20–50 candidates, the right chunk is in there — just ranked too low to make the cut.
The fix: add a re-ranking stage. Retrieve a generous candidate set, then run a cross-encoder reranker that scores each candidate against the query directly (far more accurate than the initial vector distance) and keep the top few after re-ranking. This two-stage "retrieve wide, re-rank narrow" pattern is one of the highest-ROI upgrades in RAG and often fixes the "it was almost right" class of bugs outright. Designing this multi-stage retrieval well is core to a deeper course on advanced LLM integration.
Failure mode 5: Stale, duplicated, or contradictory sources
Retrieval is only as good as the corpus. If your index contains last quarter's pricing alongside this quarter's, three near-duplicate copies of the same policy, or contradictory drafts, retrieval will faithfully surface the wrong-but-similar one. The pipeline isn't broken — your data is.
Symptoms: answers are outdated, contradict each other across runs, or cite a deprecated document.
The fix:
- Deduplicate before indexing, and keep the index fresh — re-index or invalidate on source changes, don't let it drift.
- Store a
date/version in chunk metadata and filter or boost by recency at query time.
- Have a single source of truth per fact; archive superseded docs out of the active index.
- Garbage in, garbage retrieved. Data hygiene is unglamorous and it outperforms most clever tricks.
Failure mode 6: The context is right but the prompt undermines it
Now the generation side. Suppose the correct chunk is in context but the answer is still wrong. Often the prompt is the culprit: it doesn't tell the model to ground its answer in the provided context, doesn't give it permission to say "I don't know," or buries the relevant chunk among ten irrelevant ones so the signal drowns (the "lost in the middle" effect, where models attend least to the middle of a long context).
Symptoms: the right info is in the prompt, yet the model invents, ignores it, or pads with prior knowledge.
The fix:
- Instruct explicitly: "Answer only using the context below. If the answer isn't there, say you don't know." This single line kills a surprising amount of hallucination.
- Pass fewer, better chunks — re-ranking (mode 4) makes this possible. More context is not more better; irrelevant chunks actively hurt.
- Put the most relevant chunks at the start or end, not buried in the middle.
- Ask for citations — make the model name which chunk each claim came from. It both reduces fabrication and gives you a debugging trail.
Deciding what deserves a place in the context window — and what to leave out — is its own discipline, increasingly called context engineering, and it's the focus of a dedicated course on context engineering for AI agents.
Failure mode 7: You have no evals, so you're debugging blind
This is the meta-failure that makes all the others permanent. If "is it working?" is answered by eyeballing a few queries, you can't tell whether a change helped, hurt, or just moved the failures around. Every fix above becomes a coin flip.
Symptoms: you "improve" the pipeline and have no idea if it actually got better; regressions ship silently.
The fix: build an evaluation set and measure, separately, the two halves of the pipeline.
- Retrieval metrics — on a set of questions with known relevant documents, measure recall@k and precision@k. This tells you whether the right chunk is even being retrieved (the diagnostic from the intro, made systematic).
- Generation metrics — given the retrieved context, measure faithfulness (does the answer stay grounded in the context?) and answer relevance. LLM-as-judge scoring works well here when calibrated.
Even 30–50 representative question/answer pairs turn RAG tuning from vibes into engineering: you change one thing, you re-run, you see the number move. Treating evals as first-class is exactly the muscle a course on AI evals in production builds, and it's the difference between "I think the reranker helped" and "recall@5 went from 0.61 to 0.84."
The debugging order, in one list
When a RAG answer is wrong, resist the urge to swap models. Work the pipeline in this order:
- Log the retrieved chunks. Was the answer even in there? This alone routes you to retrieval vs. generation.
- If not retrieved → check chunking (1), embeddings (2), hybrid search (3), re-ranking (4), and data freshness (5), roughly in that order of likelihood.
- If retrieved but ignored → fix the prompt and context layout (6).
- Throughout, measure with evals (7) so every change is verified, not hoped.
Nine times out of ten the bug is in steps 1–2, not the model. Working the steps in order also stops you from stacking fixes blindly — change one variable, re-measure, and you'll usually find a single failure mode is responsible for most of your bad answers.
Conclusion
RAG looks simple in the tutorial — embed, search, prompt — and that simplicity is exactly why broken RAG is so common. The hard parts are invisible until you debug them: how you split documents, how you embed and search, how you re-rank, how clean your corpus is, and whether you can measure any of it. None of those are model problems, and none are solved by paying for a bigger model.
The teams whose RAG actually works aren't using a secret model. They're disciplined about retrieval and they measure everything. Start with the diagnostic — log your chunks — fix the failure mode it points to, and put an eval set behind every change. That's how a RAG demo becomes a RAG system people can trust.
The courses linked throughout are part of Cursuri-AI.ro, an AI-learning platform with hands-on, current tracks on RAG, LLM integration, context engineering, and evaluating AI systems in production.
Sources & further reading:
This article is educational content. Techniques and tooling evolve quickly; validate approaches against your own data and current library documentation.
