We support multiple LLM providers, and each one behaves differently when wired into the same QueryAgent pipeline. We wanted to understand what that difference actually looks like, in accuracy, in cost, and across query types. So we ran a structured benchmark across five models on the same domain, same questions, and same retrieval stack. This post shares what we found.
Here's what we tested and what we found.
What Is the QueryAgent?
Synthadoc compiles raw source documents — PDFs, spreadsheets, web pages, Word files — into a structured, cross-referenced wiki at ingest time. Knowledge is written once, not re-summarized on every query.
The QueryAgent handles every question: it decomposes the query, runs parallel BM25 retrieval across the compiled wiki, builds a proportional context window from the retrieved pages, and calls the configured LLM to synthesize a cited answer. It's the most token-intensive component in the system, which makes model choice directly visible in both output quality and monthly spend.
Synthadoc supports Anthropic, MiniMax, DeepSeek, Qwen, Gemini, Groq, and local Ollama. We wanted to know which provider gave the best return on tokens.
The Test Setup
Domain: AquaFlow Capital, a private equity M&A due diligence wiki built from eight deal documents (LBO model mechanics, quality of earnings, covenant framework, ESG standards, legal DD process, exit valuation benchmarks).
15 questions across three tiers:
- English - Medium (Q1–Q5): factual retrieval, single-page lookups
- English - High (Q6–Q10): multi-source synthesis, numerical reasoning
- Cross-lingual Chinese (Q11–Q15): the same questions posed in Chinese, requiring CJK retrieval
Grading: 0–100 per question against a ground-truth rubric. PASS = 85% or above.
Models: MiniMax-Think (M3), Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1 (V3), Qwen Plus.
Results
MiniMax-Think (M3) lands in the top-left of the scatter - highest accuracy, lowest cost. No other model comes close to that position.
Claude Opus 4.8 scores 89%, just 3 points behind MiniMax, but costs 26× more per 1K queries. For most workloads, that delta doesn't buy enough accuracy to justify the spend.
Claude Sonnet 4.6 at 86% is a solid choice if you're on the Anthropic API and need consistent citation quality without paying Opus prices.
DeepSeek-R1 at ~$40/1K queries punches above its cost on synthesis-heavy questions, making it a reasonable budget pick for exploratory queries where some precision loss is acceptable.
What This Means in Practice
For knowledge base query workloads - dense source documents, multilingual queries, multi-hop numerical reasoning - the instinct to reach for the most expensive frontier model doesn't hold. MiniMax-Think (M3) wins on both axes simultaneously, which almost never happens in cost-quality tradeoffs.
If you're building with a QueryAgent pattern (BM25 or vector retrieval + LLM synthesis), the model that reads context well and reasons over retrieved chunks matters more than raw benchmark scores on reasoning tasks. That's where MiniMax-Think's architecture pays off.
At scale, model choice is a cost architecture decision as much as a quality one. A 26× cost gap compounds fast.
What This Benchmark Doesn't Cover
Accuracy and cost are two dimensions, not the whole picture. Several factors that matter in production were outside the scope of this evaluation:
- API availability and uptime: a model that scores well in testing but has reliability issues in production is a different calculus entirely
- Response latency: we measured answer quality, not time-to-first-token or end-to-end streaming latency, which affects user experience significantly in interactive query scenarios
- Rate limits and throughput: free tiers and paid tiers differ widely; sustained batch workloads can hit rate ceilings that don't show up in a 15-question test
- Regional availability: some providers have geographic restrictions or data residency constraints that may be relevant for compliance-sensitive deployments
- Support and SLA: enterprise use cases may weight vendor support, contractual SLAs, and long-term model stability (version pinning, deprecation policy) heavily
These factors can shift the ranking for a given deployment. The benchmark gives you a starting point on quality and cost layer in your own constraints from there.
The full report covers per-question scoring, WARN pattern analysis (where each model specifically fails and why), token cost math, and a per-question performance chart across all 15 questions:
→ Full evaluation report on GitHub
