Most of us have shipped an agent that worked perfectly on a laptop and then watched it behave like a different program the first time real traffic arrived. This is a writeup of where those systems actually break, aimed at the people here who keep them running rather than the ones demoing them.
The gap is not code quality. It is that agent workloads scale along axes ordinary web services do not, so the usual playbook of adding instances and tuning queries points at the wrong layer.
Why the Demo Conditions Never Survive
In development you have one or two concurrent users, rate limits feel infinite because nobody else is spending your quota, and error handling is a developer noticing a stack trace and restarting the process.
Then 200 tickets land in the same minute. Each one needs the same 5 to 8 second loop. The provider caps you at 100 requests per minute. The vector store was written for sequential access, not 200 concurrent searches. Nothing in that list is a bug you can find by reading the agent code.
Vertical Scaling Runs Out Fast
For a normal service a bigger box buys real headroom. For an agent it mostly does not, because the time goes to waiting on someone else's inference endpoint rather than to local compute. Agent runtimes commonly sit at 10 to 15 percent CPU while being saturated in every way that matters.
Vertical scaling still earns its keep when the agent does real local work: parsing large documents, generating embeddings locally, running a small model in process. Outside those cases horizontal scaling is the lever, and the hard part is externalizing conversation history and task progress so any worker can pick up any task.
The Bottleneck Is Rarely Where You Think
The model call is the longest single step, which is why everyone blames it, but it is the actual constraint maybe 40 percent of the time. If you can issue 100 concurrent calls and your limit allows it, the model is not your ceiling. The dispatcher that only moves 20 tasks a second is, or the write that blocks after every response.
"Add more instances" is the right answer only when the bottleneck lives in the processing layer. If it is a rate limit or a row lock, more instances make it worse by adding contention. The full guide to scaling AI agents from development to production walks through the instrumentation that tells these cases apart, along with the worker pool, router and specialist, and circuit breaker patterns that each cover a different failure.
Queues Are the Control Surface, Not Just a Buffer
Once a queue exists it stops being a holding pen and becomes where system behavior is defined. Priority lanes decide what degrades first. Dead letter queues keep one poisoned task from stalling the line. Queue depth also turns out to be a far better autoscaling signal than CPU or memory here, because it tracks the thing that is actually backing up.
It gives you somewhere to put graceful degradation too. Past a depth threshold you can stop accepting low priority work, route pending tasks to a cheaper and faster model, or tell users about the delay. All three beat silent timeouts and half written task state.
The Takeaway
Measure before you scale, because for agent systems the intuitive answer is wrong more often than it is right. Instrument each stage of the request lifecycle, work out which of the three resources you are actually out of (runtime, provider quota, or state), and scale that one. Everything else is spending money to make contention worse.