Async/Await in Production: The Bug That Only Appeared Under Load

Question

Async/Await in Production: The Bug That Only Showed Up Under Load

Most people learn async/await in a clean little test project.

await GetDataAsync();

Looks simple. Feels safe.

Then you push to production, traffic hits, and async/await quietly wrecks your backend. This is a real failure pattern: thread starvation mixed with blocking async calls causing cascading latency spikes.

---

It Worked Fine Locally

Typical .NET backend API:

• ASP.NET Core
• SQL Server
• External HTTP calls
• Async/await everywhere (or so we thought)

Code looked like this:

[HttpGet]
public IActionResult GetUser(int id)
{
    var user = _userService.GetUserAsync(id).Result;
    return Ok(user);
}

Nothing obviously wrong if you come from a sync background.

Locally? Fast. Stable. No issues.

---

First Symptom: Random Slow Requests

In production under moderate traffic:

• Some requests take 200ms
• Others randomly spike to 5–10 seconds
• CPU is fine
• DB is fine

Nothing obvious blocking. That's the problem.

---

The Real Issue

This line is the killer:

_userService.GetUserAsync(id).Result;

Or .GetAwaiter().GetResult().

What happens:

1. Async method starts
2. Hits an await
3. Thread frees up
4. You immediately block waiting for the result

Now combine with ASP.NET thread pool behavior.

---

Thread Pool Starvation

Under load:

• Thread pool has limited threads
• Each request blocks a thread waiting on async work
• But that async work needs threads to resume

So threads wait for threads that are already blocked. Not a classic deadlock, but the system acts like one. Latency explodes.

---

Why Local Didn't Catch It

Locally: low concurrency, plenty of threads, no pressure.

Production: many concurrent requests, real API latency, real DB spikes. Everything amplifies the problem.

---

The Fix

[HttpGet]
public async Task<IActionResult> GetUser(int id)
{
    var user = await _userService.GetUserAsync(id);
    return Ok(user);
}

And inside services:

public async Task<User> GetUserAsync(int id)
{
    return await _repository.GetUserAsync(id);
}

Async all the way up. No exceptions.

---

Hidden Trap

Sync over async in libraries:

Task.Run(() => _service.CallAsync()).Result

Or legacy code forcing sync wrappers. Spreads the problem silently across layers.

---

Another Silent Killer

Context capture:

await SomethingAsync(); // captures context

Fix for backend services:

await SomethingAsync().ConfigureAwait(false);

Avoids unnecessary context switching overhead.

---

How We Found It

We eventually saw:

• Thread pool queue length spikes
• High blocked thread count
• Requests stuck in "waiting" state

After removing all .Result usage:

• Latency dropped
• Throughput roughly doubled
• CPU normalized

---

Bottom line: Never mix .Result or .Wait() with async code. Async has to be end to end. Thread pool starvation is silent but deadly. Production is not local.