In high-volume logistics, silence is the scariest sound there is.
Years ago, I was dropped into a sorting facility that processed 8,000 scanner requests per minute. An ISP failure had cut the line for an hour. When the connection came back, thousands of scanners tried to flush their queued transactions simultaneously.
The dashboard said "Green." The floor was silent.
The system hadn't crashed; it had entered a Garbage Collection Death Spiral.
We diagnosed the "Thundering Herd" that killed the facility without changing a line of application code. We just had to respect the physics of the runtime.
The Smoking Gun: Premature Promotion The servers were running -XX:+UseParallelGC (a Stop-the-World collector) with -XX:MaxTenuringThreshold=2.
Because of the burst in traffic, the "Young Generation" (capped at 1GB) filled instantly. With a threshold of only 2, short-lived XML payload objects were being promoted to the "Old Generation" almost immediately.
The JVM was spending more time pausing to clean the Old Gen than it was processing packages. It wasn't a crash; it was a series of freezes.
The Configuration Thrash
On top of the GC pause, we found watt.server.jms.trigger.reuseSession=false. For every single request in that thundering herd, the server was forcing a full session handshake. We were burning CPU cycles just to say "Hello."
The Fix
We didn't rewrite the app. We tuned the engine:
Swapped the GC: Moved to CMS (-XX:+UseConcMarkSweepGC) to clean memory without stopping threads.
Enabled Reuse: Set reuseSession=true and enabled Producer Caching.
Throttled the Flood: Reduced trigger threads to stop context switching.
Why This Matters for AI
I wrote this retrospective because I see AI Architects making the same mistakes today.
Context Window Thrashing: If you spin up a new LLM Context for every interaction, you are recreating the reuseSession=false bug.
Token Garbage: If you don't manage RAG retrieval lifecycles, you will hit token limits — the new OutOfMemoryError.
Read the full breakdown of the JVM flags and the "Physics of Latency" here:
The Day the Scanners Stopped