RAG (Retrieval-Augmented Generation - where an LLM answers questions using retrieved document chunks) breaks most often before retrieval even starts. The question going in is underspecified, and no retrieval system can compensate for a bad query.
The Idea: A Small Loop Before You Search
Most RAG pipelines treat the user's raw question as retrieval-ready. It usually isn't. A question like "what are the risks?" has no anchor - no product, no time range, no document scope. The retrieval step fetches loosely related chunks, the LLM hallucinates a coherent answer from noise, and everyone blames the model.
Loop engineering for question parsing inserts a lightweight reasoning step before retrieval runs: parse the raw question, identify what context is missing to make it answerable, then re-parse with that gap filled - either by asking the user, inferring from conversation history, or pulling from a lightweight document index. The loop is intentionally small: one or two passes, not an autonomous agent spinning indefinitely. The goal is a query that names specific entities, a scope, and an intent - not a philosophical question the retriever has to guess at.
Real Example
Here's a minimal implementation pattern using Python and any chat-completion API:
def parse_and_refine(raw_question: str, doc_summary: str, llm) -> str:
audit_prompt = f"""
User question: "{raw_question}"
Document summary: "{doc_summary}"
Identify what is ambiguous or missing (entity, time range, scope).
Return a refined, retrieval-ready question. If nothing is missing, return the original.
"""
refined = llm.chat(audit_prompt)
return refined.strip()
# Usage
raw = "What are the risks?"
doc_summary = "Q3 2024 earnings report for Acme Corp, covering supply chain and FX exposure."
ready_query = parse_and_refine(raw, doc_summary, llm)
# → "What supply chain and FX risks does Acme Corp report in Q3 2024?"
The doc_summary doesn't have to be a full index - a one-paragraph description of the document set is enough to ground the refinement. For multi-turn chat, you'd also pass conversation history into the audit prompt so the loop can resolve pronouns and carry-forward references ("the risks from last quarter").
The cost is one extra LLM call per query. On most inference APIs that's under 50ms added latency and a fraction of a cent - easily worth it if it prevents a hallucinated answer that sends someone down the wrong path.
Key Takeaways
- Retrieval quality is bounded by query quality - fixing the query upstream is more reliable than tuning chunking or embeddings downstream.
- A two-pass parse loop (audit then refine) catches the most common failure mode: questions that lack entity, scope, or time anchors.
- Keeping the loop to one or two LLM calls maintains low latency while meaningfully improving retrieval precision.
What's the most common type of underspecified question your users actually send into your RAG system - missing entity, missing time range, or something else entirely?
Sources referenced: Towards Data Science - Loop Engineering for RAG Question Parsing