A search-augmented agent looks like RAG until the citation list breaks. The first user clicks through, lands on a 404, and the trust budget spends in one impression. Three months of eval work, groundedness and context adherence and factual accuracy all green, did not catch it.
The reason is structural. Generic RAG eval was built for a frozen vector store and a stable ground truth. Search agents have neither. The corpus is the live web, the agent writes the literal query string that drives what comes back, sources span government filings and LinkedIn takes in one result set, and users click the URLs. Skip the four axes unique to this shape (query quality, source freshness, source diversity, citation validity) and you ship an agent that confidently cites stale or fabricated URLs. The rest of RAG eval still applies, but the rest of RAG eval was already passing. There is a longer version with the rubric code if you want it.
Three differences pull this out of the RAG bucket. The corpus is not yours: two runs of the same query twelve hours apart return different URLs, so reproducibility lives in a snapshot, not the results page. The query is the lever: search is sensitive to the literal string the agent writes, and a rewrite that drops a constraint returns a clean-looking set that misses it, which no synthesis prompt recovers from. And the user verifies: a citation is a URL the user clicks, so a fabricated or 404 link is a one-click trust failure that groundedness against the retrieved set never sees.
Score the four axes per query, not aggregated. A single number hides which one collapsed, and the diagnostic value is in the vector.
Axis 1: query quality
The agent turns the user's prompt into a search string, and that string is what the engine literally reads. Three failure modes show up in almost every run.
- Constraint drop. "Laptops with battery over 10 hours" compresses to "best laptops 2026 for ML," and the results average five-hour batteries. Groundedness is fine because the brief grounded itself in the retrieved set. The constraint is gone.
- Entity loss. A question about a specific company or regulation abstracts to the category, and the brief answers a different question well.
- Over-broad rewrite. A narrow factual lookup expands into a topic, the search returns a Wikipedia overview, and the brief paraphrases it and misses the fact.
Score it with a judge that takes the original prompt and the rewrite as a pair and rates constraint preservation and entity recall 1 to 5, penalizing unnecessary expansion as much as a drop. Run this over a week of traffic and the first cluster is usually "rewriter dropped the year, version, or region" on roughly one in eight queries. Cheapest fix in the loop, and the one most teams skip.
Axis 2: source freshness
Freshness is a function of the question, not a global setting. "Capital of France" is insensitive. "Current price of X" is real-time. "FDA guidance on AI as a medical device" is quarter-sensitive. The agent has to decide the regime and behave: prefer recent sources, refuse stale ones, and add a recency caveat when the answer depends on current data.
Two checks carry it. Classify the freshness regime per query (insensitive, slow, fast, real-time) and attach it as a span attribute. Then attach a recency value per source, take the median recency of the cited sources, and fail when it exceeds the floor for that regime. The stale-fact bug is the one users notice last and complain about loudest: correct the day the source was indexed, wrong every day after, and invisible to every other metric.
Axis 3: source diversity
A general search returns Wikipedia, vendor blogs, news, government docs, and SEO spam in one set, and the agent cites three to five. The failure is monoculture: every citation from one domain, or a brief that reads like consensus when it summarizes one tab.
Three checks cover it, and the first two are deterministic and run in microseconds:
- Unique-domain count per citation set, with a floor by question class. A date lookup can be tight; a synthesis question needs four or more.
- Per-domain cap. No domain contributes more than two citations, unless the question is about that domain (an EU AI Act question can cite eur-lex repeatedly).
- Primary-vs-secondary ratio. Classify each citation as primary (a filing, paper, or announcement) or secondary (commentary, aggregator). A regulatory question with zero primary citations is a failure however grounded the brief is.
Axis 4: citation validity
The highest-signal cheap check in the whole stack, and the one most teams find only after the first complaint. Three sub-checks compose it.
- The URL resolves. HEAD-request every citation; a non-2xx or a redirect to a different domain is a fail. Tens of milliseconds, and it rejects fabricated and stale links before the user sees them.
- The passage exists on the page. Fetch the body and confirm the quoted or paraphrased passage is actually there. A paraphrase that drifts past a similarity threshold is a fail. This catches a real URL with an invented quote.
- The claim is supported by that passage. Groundedness and chunk attribution, but per claim, not per answer. A brief can be 0.94 grounded as a whole and 0.61 aligned per claim, and users only see the per-claim view.
Wire it as two stages: the deterministic checks (resolves, passage present) as a hard gate, the claim-passage entailment as a judge on the synthesis span. A citation that fails any check is a hard reject: log it, regenerate the answer with that URL excluded, and re-run the gate. Teams that wire this stop seeing the "user clicked a 404" failure almost immediately.
Cache the snapshot so CI does not flake
Two runs of the same prompt hours apart return different URLs, so without a per-test-case cache the gate flakes for reasons unrelated to the agent and the team stops trusting it. Cache three layers: the search call (query and full result JSON), the page fetch (URL, body, timestamp), and the extraction (cleaned text, date, anchors). Replay every run from cache, and re-snapshot deliberately on a schedule that fits the domain (weekly for news, monthly for stable how-to agents), treating each re-snapshot as a new golden-set version with its own baseline.
Trace it so a bad answer is debuggable
The eval scores attach to spans, and the spans are how you debug. Three span kinds carry the load: a retriever span per search call, carrying the literal query the agent sent plus per-source URL, recency, and authority; a tool span for the wire call to the search API, so latency and cost land on the right node; and an LLM span for the synthesis. Splitting the search call from the retriever span keeps the query-quality input and the network cost on separate, queryable nodes.
Run the four axes per query and read the vector. The axis that drops is the bug, and the standard RAG metrics keep running alongside because they are necessary, just not sufficient.
