Your RAG isn't broken. Your chunks are.

Question

Everyone wants to blame the model. "GPT-4 is hallucinating." "Claude isn't following instructions." "The embeddings are bad."Nine times out of ten, it's the chunks.I've been auditing RAG pipelines lately and the same patterns keep showing up:Fixed-size chunking that splits sentences in half. The model gets a fragment that starts with "...and therefore the policy applies only when" and you wonder why the answer is confidently wrong.No overlap between chunks. Context that spans a paragraph boundary is gone. The model sees the question but not the answer, because the answer lives in the chunk after the one that matched.Chunking before cleaning. Headers, nav menus, footer junk, and "Click here to subscribe" all get embedded as if they're content. Then they show up in retrieval and pollute the context window.Embedding the wrong thing. People embed the raw chunk but the user query is phrased completely differently. No wonder cosine similarity is low — they're not semantically close, they just happen to be about the same topic.Before you blame the model, look at what you're actually retrieving. Log the top 5 chunks for 10 real user queries and read them like a human. If you can't answer the question from those chunks, the model can't either.The model is a mirror. It reflects the quality of what you feed it.