Civil engineering background. I am not an AI researcher, and I don't write synthetic benchmarks for a living.
I build an open-source Model Context Protocol (MCP) server called MSCodeBase. Its primary job is to give coding agents structured context about a codebase: semantic vector search, symbol resolution, AST call graphs, persistent memory layers, and sandboxed execution.
Over months of iteration, the project grew organically. Every time an agent struggled with an edge case, I added a specialized tool. Before long, the surface area expanded to 62 registered tools.
Then I sat down and asked the boring engineering question I had been avoiding: Which of these tools does an agent actually use when solving real problems?
To find out, I instrumented runtime telemetry and let the system run.
A. The Inventory Disconnect
When I audited the telemetry logs, the gap between what was engineered and what was executed was embarrassing.
- Registered Tools: 62 total (32 core engine tools, 14 memory modules, 12 inline tools, 4 dev utilities)
- Visible by Default: 16 of the 32 core tools
- Tools with Metric Records: 5 of 62
Five tools. Out of sixty-two.
Not "five tools used frequently" — five tools that had ever logged a single invocation in the entire history of the telemetry file.
The other 57 tools were not necessarily broken. They were assumed tools. They were implemented, documented, tested in isolation, and registered — but never reached by the model during real task execution.
Worse yet, the narrow surface the agent did reach happened to be the most fragile part of the pipeline:
- LSP Toolkit: 8 invocations, 3 hard runtime errors (a 37.5 error rate on live calls).
- search_code: 10 timeouts recorded, hanging for 15 to 31 seconds per call with repeated retry loops.
The primary entry point the agent picked was precisely the tool most likely to fail mid-search.
B. The "Smart" Search Illusion
To understand why the agent was struggling, I ran a single-symbol retrieval test against the exact same repository index using four different paths:
- grep -c: 28ms latency → Exact file, exact line hit.
- search_code(fast): 75ms latency → Top-1 exact match.
- search_code(quality): 3666ms latency → Miss (returned JSON log files and CHANGELOG history).
- get_symbol_info: Failed → Returned "not found" for a symbol that existed in the source file I was explicitly querying.
The heavy semantic "quality" mode was 49x slower than exact matching and returned pure context noise instead of the function definition.
Furthermore, get_symbol_info — a tool explicitly designed to resolve symbol names via AST parsing — reported "not found" twice in a single debugging session for a symbol clearly visible in the tree.
In tool demos, you only see the happy path at page-load speed. In real agent runs, you encounter the live path: slow, silent, confident, and wrong.
C. Agent Execution Patterns in the Wild
I ran two subagents against the same broken repository branch. One was instructed to be "MCP-first, diagnose only," and the other was given no specific tool guidance (Natural):
- MCP-First Agent: Executed 9 MCP calls (wasting 4 on an unindexed semantic layer). It correctly identified the bug location, but modified zero files.
- Natural Agent (Read + Grep + Edit): Executed 0 MCP calls. It relied entirely on standard file reading, exact searching, and direct edits, resolving all failing tests in a single pass.
The model itself was rarely the bottleneck. The complex, slow machinery built around it was blocking execution.
D. What Actually Wins
I benchmarked search strategies head-to-head on real codebase queries with verified ground truth:
- Quality (Semantic Only): Recall@5 = 0.133 | Median Latency = 2145ms
- Fast (Exact Only): Recall@5 = 0.167 | Median Latency = 167ms
- Cascade Routing (Fast → Quality): Recall@5 = 0.233 | Median Latency = 564ms
- Cascade + AST Graph: Recall@5 = 0.433 | Median Latency = 177ms
The cascade approach outperformed every single-mode search strategy. Adding the structural AST graph layer doubled accuracy again while remaining faster than the heavy semantic search.
The winning pattern is simple: cheap and precise matching first, heavy semantic search only as a fallback when exact matching returns nothing.
Per-query-type routing reinforced this lesson: semantic scoring scored 0.00 recall on exact bug-hunting tasks where string matching excelled. Meanwhile, the AST graph caught structural dependencies (find_impact) that vector search missed completely (1.00 vs 0.00 recall).
E. Key Takeaways
- Measure actual invocation: Your documented tool surface and your actual usage telemetry are two different graphs. Without runtime measurement, you are maintaining dead code.
- Build fallback chains: Do not rely on single "smart" paths. Use exact-and-cheap before semantic-and-expensive.
- Structure over vectors for code: An AST graph layer answers structural relationships that vector embeddings cannot resolve.
Start with one search path that works fast. Measure what gets called, observe where it fails, and only add tool number two when real runtime data proves you need it.