Why cloud outages last for hours when traffic never grew: what to change in your load balancer

Question

On Thursday 12 November 2026, Outlook stopped working for a large share of Microsoft 365 users. People could not open their mailboxes in the browser, in the desktop app or on their phones. Microsoft's final record of the incident gives the impact window as 10:05 UTC on 12 November to 16:10 UTC on 13 November, about thirty hours, and its first public notice went up at 16:20 UTC on the 12th.

The most useful sentence Microsoft published during those thirty hours was not about the cause. It came three hours into its public updates, at 19:18 UTC:

"We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure."

Traffic never grew. The same users were sending the same requests. What changed was how many servers were taking them, and that number kept falling for reasons that had little to do with the original fault.

This article is for engineers who run services behind a load balancer. I want to use this outage to explain one failure pattern that turns a short problem into a long one, and then go through the exact settings in Envoy, AWS Application Load Balancer and your own client code that stop it. I also built a small model of the pattern so you can see the numbers move, and I will be clear about where the model ends and Microsoft's facts begin.

What Microsoft said, in the order it said it

Microsoft posts incident updates in the Microsoft 365 admin center, which needs a work account. The University of Pennsylvania's mail team republishes the same text with UTC times on a public page, and Microsoft's own public status page carries a shorter version. Every quote below comes from incident EX1550543 in that record. I am quoting Microsoft, not news reports.

Here is the sequence, cut down to the updates that matter for an engineer:

• 18:19 UTC, 12 November. The affected infrastructure "is experiencing elevated load across multiple regions."
• 18:43. "Affected servers are becoming intermittently unavailable, which places additional strain on the remaining infrastructure."
• 19:18. The quote above: not more traffic, fewer servers.
• 19:43. "Affected servers are failing automated health checks, which is causing them to become unavailable, and this may be related to a recent change to a component involved in processing those checks."
• 19:57. Microsoft has "taken steps to prevent additional servers from being removed from service."
• 20:56. "We've validated an alternative health check method on a subset of affected infrastructure, which allowed the servers in that subset to successfully return to service."
• 22:00. "A core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding." The plan is to isolate part of the service, restart it, and gradually reintroduce traffic.
• 23:59. "While the servers we've returned to service initially remain healthy, we've observed that they can become unresponsive again once they begin receiving traffic."
• 01:06, 13 November. "A process on the affected servers responsible for establishing secure connections becomes overwhelmed, causing the servers to stop processing requests." Microsoft also begins reverting a recent network configuration change that correlates with the start of impact.
• 04:59. Microsoft observes "a surge in outbound call volume from our routing services to our backend servers," partly "due to traffic rebalancing actions that we have taken throughout this event."
• 09:55. "Engineering teams are cycling affected infrastructure through traffic drain, restart, remediation, and staged return-to-service."
• 17:57. Microsoft gives a preliminary root cause: "The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change." It adds: "Given the scale of demand, we restored services gradually, validating stability at each stage. While this took longer, it helped ensure a stable restoration."

Microsoft has said a full Post-Incident Report will follow. Until it does, nobody outside Microsoft knows exactly which component did what. What the updates do show is the shape of the problem, and that shape is common enough that you can recognise it in your own systems.

The loop that keeps a service down

Read the updates again and you can see four steps that feed each other:

1. Some servers fail their health checks and are taken out of service.
2. Their traffic moves to the servers that are left. Those servers now carry more than they were sized for.
3. Overloaded servers answer slowly, so they fail their health checks too, and are removed as well.
4. When servers are repaired and put back, they receive a large share of the traffic all at once, become overloaded straight away, and fail again.

Meanwhile users and apps keep retrying, which adds requests on top of the same demand. Load can also come from the system itself: Microsoft's 04:59 update describes a surge of calls from its own routing services to its backend servers, partly caused by the traffic moves it had made during the event.

Google's SRE book names this pattern. In its chapter on addressing cascading failures it says: "A cascading failure is a failure that grows over time as a result of positive feedback." It also warns about the exact mechanism in step 3, that health checking can turn on the service it is meant to protect: "This practice may create a failure mode in which health-checking itself makes the service unhealthy."

Researchers call the worst version of this a metastable failure: the first trigger knocks capacity down, and then the system's own reactions keep it down even after the trigger is gone. I am not saying that is what happened inside Exchange Online. Microsoft's later updates suggest the network change may have kept causing trouble until it was reverted. But the 23:59 update, servers that are healthy until traffic reaches them, is the symptom that this loop produces.

A small model you can run

To show the loop with numbers, I wrote a short Python model, ejection_cascade.py. It is not Microsoft's system and the numbers are mine. I chose them to be easy to follow.

• A pool of 20 servers. Each can complete 100 requests a second, so the pool can do 2,000.
• Users send 1,500 requests a second, every second, for the whole run. The pool runs at 75 percent. Traffic never changes.
• At 60 seconds, a bad change breaks 6 servers. They fail every health check and serve nothing.
• At 240 seconds, the fault is fixed and those 6 are healthy again.
• Every 10 seconds the load balancer runs a health check. A server fails if it is broken, or if it was asked for more than it can do, because an overloaded server answers its check late.
• A failed server is removed for 30 seconds times the number of times it has been removed, up to 300 seconds. That is how Envoy's outlier detection counts: "The real time is equal to the base time multiplied by the number of times the host has been ejected."
• An overloaded server wastes work on requests that time out, so the further past capacity it goes, the fewer requests it completes. This is how real overloaded servers behave, and it is the reason a busy server is worse than a slow one.

To keep the picture clean, the model has no retries. Everything you see comes from removals alone.

Six servers break. The load balancer correctly removes them. Now 1,500 requests a second land on 14 servers, about 107 each, and each can only do 100. At the next health check all 14 answer late and all 14 are removed. Ten seconds after a fault that broke 30 percent of the pool, the pool is empty.

The fault is fixed at 240 seconds, and it does not matter. The removed servers come back when their removal timers run out. Because they were removed together, they come back in groups: the 6 fixed servers at 240 seconds, the 14 others at 270. Whichever group comes back first is handed all 1,500 requests a second. They fail the next check and are removed again, for longer this time. That is the 23:59 update in miniature: healthy until traffic reaches them.

Four settings, four outcomes

Next I ran the same fault and the same traffic through four load balancer setups.

• No cap, no floor. The load balancer removes any server that fails a check and has no rule about how many it may remove. This is what you get from a home-made balancer, or from a commercial one where the safety settings are turned off.
• Cap removals at a third. Never remove more than 6 of the 20 at once. Envoy has this cap, max_ejection_percent, but only for outlier detection, its removals for errors and failed connections it sees on live traffic. It has no equivalent cap for active health checks; for those, the panic threshold below is the guard.
• Panic threshold at 50 percent. When fewer than half the servers are healthy, stop trusting the checks and send traffic to every server, broken ones included. This is called failing open. In Envoy this is the panic threshold, and in AWS ALB it is routing failover.
• Cap, shed and slow start. The cap at a third, plus each server turning away anything above its own capacity with a fast rejection, plus servers that come back getting a small share of traffic that grows over 30 seconds.

The model printed these results, as requests completed per second during the fault and the time the pool was back to full service:

• No cap, no floor: 217 requests a second on average during the fault, 0 for most of it, and not back to full service within the 600-second run.
• Cap removals at a third: 1,300 a second during the fault, and not back to full service within the run.
• Panic threshold 50%: about 1,090 a second on average during the fault, and back to full service at 240 seconds, when the fault is fixed.
• Cap, shed and slow start: 1,400 a second during the fault, and back to full service at 245 seconds, 5 seconds after the fix.

The number to compare is the recovery time. A setup that serves more during the fault but never recovers is worse than one that serves less and recovers.

The fault is identical in every row. Six servers broke for three minutes. Depending only on configuration, that fault cost either about seven percent of requests for three minutes, or almost every request for the rest of the ten-minute run.

The cap alone stopped the cascade but did not recover. This surprised me the first time I ran it. With the cap in place, the 6 broken servers fill the removal slots. The 14 working servers are still overloaded at 107 percent, so they keep failing their checks, and as soon as any slot opens one of them is removed and the others get busier. The pool settles at 1,300 a second and stays there, even after the fault is fixed, because the six removal slots are now always full of busy working servers and the pool never has all twenty in rotation at once. A cap protects you from losing the whole pool, but if the check fails servers for being busy, the pool never gets out of overload.

The panic threshold recovers, but it pays during the fault. It switched on even though only six of twenty servers broke, because the fourteen overloaded servers failed their checks too and took the healthy share below half. Sending traffic to every server means a share of requests lands on broken servers and fails. Envoy's documentation describes it as a mode where it "will disregard health status and balance either amongst all hosts or no hosts." You accept some errors on broken servers in exchange for not crushing the healthy ones.

The setup that did best changed the meaning of the health check. A server that turns excess requests away stays fast, so it passes its check and stays in rotation.

Remove the broken server, keep the busy one

There are two different reasons a server can fail a health check. It can be broken, or it can be busy. A load balancer that cannot tell them apart treats both the same way, by removing the server, and removing a busy server makes every other server busier.

The fix lives on the server, not in the load balancer. Give every server a limit on how much work it accepts at once, and reject anything above that limit immediately, with HTTP 503 and a Retry-After header. Amazon's Builders' Library article on load shedding describes what happens without it: "When these factors are combined, an overload creates its own feedback loop that results in overload as a steady state." Google's SRE book describes the goal of shedding the same way: to keep the server from "running out of RAM, failing health checks" while "still doing as much useful work as it can."

Two details make shedding work in practice.

Shed before the expensive step. A rejection only helps if it is cheap. Microsoft's 01:06 update said the process that was overwhelmed was the one "responsible for establishing secure connections." If the TLS handshake is your expensive step, a 503 sent after the handshake is too late, because the server has already paid for it. Limit new connections at the edge, before the handshake. When I probed Outlook's servers from India on the morning of 13 November, 138 of 186 attempts to Outlook addresses accepted the TCP connection and then did not finish the TLS handshake within 6 seconds. From outside I cannot tell which machine stalled, but a health check that only opens a TCP connection would have called those endpoints healthy.

Keep the health check cheap and honest. The check should test the step users need, such as completing the TLS handshake, and should not run an expensive query that a busy server will answer late. A separate, very light liveness check ("is the process running") is a good partner for it. The SRE book makes the same split. It calls process health checking and service health checking "two conceptually distinct operations": the first asks whether the program responds at all and belongs to the scheduler, the second asks whether it can take this kind of request right now and belongs to the load balancer.

What to change in your own configuration

Here are the settings, with their real names and defaults, from Envoy's and AWS's own documentation. Envoy is the open source proxy that many service meshes and API gateways are built on; ALB is the HTTP load balancer in AWS. Some of these settings are off unless you turn them on.

Envoy

Every value below is a field on an Envoy cluster. The comments give the default from the Envoy API docs.

clusters:
- name: mail_backend
  connect_timeout: 1s
  lb_policy: ROUND_ROBIN
  common_lb_config:
    healthy_panic_threshold:
      value: 50              # default 50: below this share of healthy hosts, use them all
  round_robin_lb_config:
    slow_start_config:
      slow_start_window: 60s # off unless set
      min_weight_percent:
        value: 10            # default 10: a returning host starts at 10% of its weight
  outlier_detection:
    split_external_local_origin_errors: true  # default false
    consecutive_local_origin_failure: 5       # default 5: connect failures, resets, timeouts
    enforcing_consecutive_5xx: 0              # default 100: do not eject for 5xx replies
    enforcing_success_rate: 0                 # default 100: nor for a low 5xx success rate
    interval: 10s            # default 10s
    base_ejection_time: 30s  # default 30s, multiplied by how often the host was ejected
    max_ejection_percent: 33 # default 10
  circuit_breakers:
    thresholds:
    - priority: DEFAULT
      max_connections: 1024  # default 1024
      max_pending_requests: 1024
      max_requests: 1024
      retry_budget:
        budget_percent:
          value: 20.0        # default 20 once a budget is set
        min_retry_concurrency: 3

A few notes on these, because the defaults are not all what people assume.

• Do not eject servers for shedding. By default Envoy ejects a host after 5 consecutive 5xx replies, and a 503 is a 5xx. A server that sheds load correctly would be ejected for it, which is the cascade again. The settings above tell Envoy to ignore 5xx replies for ejection: both the consecutive-5xx detector and the success-rate detector are switched off for errors the server returns. Envoy still ejects for failures it sees itself, such as refused connections, resets and timeouts, through consecutive_local_origin_failure and the local-origin success rate. (The separate consecutive_gateway_failure check counts 502, 503 and 504, so it does not help here; its enforcement defaults to 0.)
• max_ejection_percent defaults to 10. That is a sensible cap on a large pool; I set 33 only to match the model. On a small pool, check how your Envoy version rounds it, because a percentage cap can allow very few ejections. Envoy has an always_eject_one_host option, off by default, for that case. Remember that this cap applies only to outlier detection. Removals from active health checks are not capped by it; for those, your protection is the panic threshold.
• The panic threshold defaults to 50 percent. Leave it on. If you have set it to 0 to "never send traffic to unhealthy hosts", you have built the first setup in my model.
• Slow start is off by default. Envoy's slow start docs say a host enters slow start when it joins the cluster, or, with active health checking, when it goes from unhealthy to healthy. They also give an honest limit: it helps most when a few hosts come back, and helps little when every host is new at once, because they all get similar traffic.
• The retry budget caps retries as a share of requests in flight, meaning requests that have been sent and not yet answered. The circuit breaker docs give a default of 20 percent and a minimum of 3 concurrent retries, and setting a budget overrides the plain max_retries limit, whose default is 3.

AWS Application Load Balancer

ALB does not have an ejection cap, but it does fail open, and it has slow start. Both are target group attributes:

aws elbv2 modify-target-group-attributes \
  --target-group-arn "$TG_ARN" \
  --attributes \
    "Key=target_group_health.unhealthy_state_routing.minimum_healthy_targets.percentage,Value=50" \
    "Key=slow_start.duration_seconds,Value=60"

From the ALB target group documentation:

• Routing failover. "If the number of healthy targets is below this value, send traffic to all targets, including unhealthy targets." The count version defaults to 1, which means ALB fails open only when no target is healthy. The percentage version is off until you set it. AWS's own explanation of why you would: it "reduces the risk of overloading the healthy targets." For a large pool, a percentage is the setting you want.
• Slow start. The load balancer "linearly increases the number of requests that it can send to a target in slow start mode." The range is 30 to 900 seconds, and it is off by default. Two catches from the attributes page: slow start cannot be used with the least outstanding requests or weighted random algorithms, and targets registered into an empty target group in one operation do not enter slow start, because "Newly registered targets enter slow start mode only when there is at least one healthy target that is not in slow start mode." So if your recovery plan is to deregister every target and register them all again in one call, none of them will ramp. Targets that recover on their own do re-enter slow start: "When the target becomes healthy, it enters slow start mode again."
• Limiting requests per target. ALB's newer target optimizer feature lets you enforce a maximum number of concurrent requests on each target, using an agent you install on the targets. It can only be turned on when the target group is created.

ALB does not retry requests for you, so retries are your client's job, and that is a part of the loop you control even when someone else runs the load balancer.

Retries: the load you add yourself

Every client that retries a failed request takes a bigger share of a server that is already short of capacity. The Amazon Builders' Library article on timeouts, retries and backoff says: "Retries are 'selfish.'" It also shows how quickly they multiply in a deep call chain. With five layers of services that each retry independently, "the load on the database will increase 243x, making it unlikely to ever recover."

Backoff with jitter, which means waiting a growing and partly random time before each retry, spreads retries out in time, which helps, but it does not reduce how many there are. If almost every request is failing, the same number of retries still arrives, just later. What reduces the number is a budget. Google's SRE book, in the chapter on handling overload, uses two:

• a per-request limit of "up to three attempts", and
• a per-client limit: "A request will only be retried as long as this ratio is below 10%," where the ratio is retries to requests.

The book says that in a bad case, retries without the per-client budget grow load to "somewhere just below 3X," and the budget cuts that "to just 1.1x in the general case."

A client-side retry budget is about twenty lines. This one keeps a running count of requests and retries and refuses to retry once retries pass 10 percent of requests:

import random
import time


class RetryBudget:
    """Allow a retry only while retries stay under `ratio` of all requests."""

    def __init__(self, ratio: float = 0.1, min_retries_per_window: int = 3) -> None:
        self.ratio = ratio
        self.min_retries = min_retries_per_window
        self.requests = 0
        self.retries = 0

    def record_request(self) -> None:
        self.requests += 1

    def can_retry(self) -> bool:
        allowed = max(self.min_retries, self.requests * self.ratio)
        if self.retries < allowed:
            self.retries += 1
            return True
        return False


def call_with_budget(send, budget: RetryBudget, attempts: int = 3, base: float = 0.1):
    budget.record_request()
    for attempt in range(attempts):
        try:
            return send()
        except TimeoutError:
            if attempt == attempts - 1 or not budget.can_retry():
                raise
            time.sleep(random.uniform(0, base * 2 ** attempt))  # full jitter

In production you would reset or decay the counts over a time window, so one bad minute does not shape the next hour, and you would also retry on a 503 and wait for its Retry-After value instead of the jittered delay. The rule does not change: three attempts at most per request, and retries switched off for everyone once they become a real share of traffic.

Two more client rules follow from the outage itself:

• Honour Retry-After. If the server tells you when to come back, come back then. A server that sheds load is asking for exactly this.
• Use a circuit breaker on calls to a struggling dependency. When most recent calls to a service are failing, stop calling it for a short time and return a cached answer, a clear message, or the rest of the page without that part. Google's SRE book describes a client-side version it calls adaptive throttling, where each client tracks how many of its requests the backend accepted and starts rejecting locally once requests run well ahead of accepts. The book says, "We generally prefer the 2x multiplier."

Bring servers back slowly, and in stages

The longest stretch of the Microsoft incident was not finding the cause. It was getting servers back without them falling over again. The 22:00 update describes isolating part of the service, restarting it and "gradually reintroducing traffic." The 09:55 update on 13 November describes "traffic drain, restart, remediation, and staged return-to-service," and the 17:57 update explains that the slow, staged restore was deliberate.

You can build this into your tools before you need it. Slow start in the load balancer handles one server at a time. For a whole region or a whole pool coming back, you need a way to admit a fraction of traffic, watch error rates and latency, and raise the fraction step by step. That can be weighted routing at your DNS or edge layer, a feature flag that limits which clients may connect, or an admission limit you can raise by hand. The SRE book gives the same advice: "When adding load to a cluster, slowly increase the load," and, during recovery, "Gradually ramping up the load" so caches warm and connections are set up before traffic returns to normal.

Microsoft also did two other things that belong in every runbook. At 19:06 UTC, under three hours after its first notice, it "paused all deployments and configuration changes" so nobody changed a second thing while the first was still unexplained. And at 23:59 it began "limiting which types of traffic can reach the affected infrastructure," which both reduced load and helped it find out whether one type of request was making things worse.

The whole picture

Here is the request path with each guard in the place it belongs.

1. The client retries at most three times, with jittered backoff, inside a retry budget, and honours Retry-After.
2. The load balancer brings returning servers back with slow start.
3. Each server puts a circuit breaker around its own calls to the data store, so a slow store does not tie up all of its threads.
4. Each server sheds work above its own limit with a fast 503, before the expensive step, so it stays quick enough to pass its health check.

The two labels above the path cover the rest. The load balancer caps how many servers it removes for errors, and fails open with a panic threshold or routing failover before the pool gets small. And shedding happens on every server, not only the one the arrow is drawn from.

None of these is new; Envoy, ALB and the SRE book have described them for years. In the Microsoft incident, a network problem that took some servers out was followed by many hours of servers being removed, restarted and overloaded again. These settings are aimed at that second part.

A checklist for this week

If you run a service behind a load balancer, these are the questions I would answer before the next incident:

1. What happens if half my servers fail their health check at the same moment? If the answer is "they are all removed", set a panic threshold or routing failover.
2. Does my health check fail a server for being busy? If it does, add load shedding so busy servers stay fast, and keep the check cheap.
3. Do servers that come back get their full share at once? If they do, turn on slow start, and know how you would bring back a whole pool in stages.
4. How many times can one user request be tried across my whole call chain? Multiply the attempts at each layer. If the number is large, add a retry budget at one layer and stop retrying at the others.
5. Where is my most expensive step, and do I reject work before it or after it? For many services it is the TLS handshake or a database call.
6. Can I freeze changes and limit traffic types with one command during an incident? Microsoft did both. You should be able to as well.

Sources and limits

Microsoft's quotes come from its Service Health record for incident EX1550543, read on the University of Pennsylvania mirror on 13 and 14 November 2026, and from Microsoft's public status page. The preliminary root cause is Microsoft's own wording; its full Post-Incident Report had not been published when I wrote this. The probe numbers come from my own tests from one home connection in India on the morning of 13 November, with no account and no login. The model is mine, with numbers I chose to explain the mechanism. It is not a reconstruction of Exchange Online, and its outcomes depend on those numbers.

If you want to go further into the server side of this, I wrote a free lesson on load shedding at systemdesign.academy, with the diagrams and the trade-offs of each way to reject work.