Paper 08: Non-linear Dynamics of Retry Storms — When Self-Healing Mechanisms Break the System

I. Abstract #

Conventional intuition treats automatic retries upon network failure as a safety net that improves software reliability. In non-linear systems, this reflex leads directly to collapse. When load crosses a critical threshold, retries create a positive feedback loop with an amplification gain exceeding 1, turning a brief glitch into a self-sustaining storm. The system falls into metastable failure: a stuck overload state where all compute capacity is wasted on expired requests, driving useful goodput to zero even after the original trigger has disappeared. Hysteresis keeps the system locked in this overloaded state. Three architectural defenses are essential: capping retry budgets at 10%, dropping expired work early through deadline propagation, and shedding excess load right at the entry perimeter.

II. What the “Retry a Few Times Just in Case” Mental Model Misses #

1:00 PM UTC on September 20, 2015. Amazon Web Services engineers in Northern Virginia made a decisive operational move: they cut off all incoming traffic to the internal DynamoDB metadata service¹. Metadata is the directory information that tracks which storage node holds each slice of data. They did not cut traffic to patch a security flaw or replace broken hardware. They had to block connections to rescue the cluster from thousands of internal storage servers that were relentlessly resending failed requests.

My intuition — and that of many engineers — about retries (automatically resending a request after a failed attempt) starts from a comforting piece of arithmetic. Suppose a network call has a 1% chance of failing. If the software automatically retries twice, the probability that all three attempts fail drops to 0.01 cubed, or one in a million.

Many developers habitually drop max_retries: 3 into a service configuration on a Friday afternoon before heading home. They feel confident they have just raised service availability to near perfection. On paper, retrying three times cuts the failure probability from one percent to one in a million. On production on a Friday afternoon, retrying three times guarantees that an overloaded server receives three identical copies of the same heavy workload.

That one-in-a-million calculation holds under only one condition: the server on the second and third attempt must be completely independent of the first attempt. In practice, a slow network call or delayed server response is rarely caused by a stray cosmic ray hitting a fiber-optic cable. It is slow because it is overloaded. When a server is backed up behind a long queue, sending two extra retry requests does not help the user get an answer faster. That extra traffic only pushes the server over the edge faster.

Anyone who has used a mobile banking app in Ho Chi Minh City on Lunar New Year’s Eve has taken part in a retry storm like this. When lucky-money transfers spike, the system slows down and phone screens show a spinning wheel. The natural human reflex is to swipe down to refresh or tap the transfer button five or six times in a row. The banking server in the background does not cancel the earlier transactions still waiting in line. Every extra tap simply adds another heavy transaction into an already full queue.

Here I want to draw a clear line between two metrics that are easy to confuse on a monitoring dashboard: Throughput and Goodput. Throughput (total raw throughput) measures all work the hardware is processing across CPUs, RAM, and network links. Goodput (useful throughput) counts only those requests that finish successfully and return a result while the user is still waiting.

Every request has a maximum waiting limit called a timeout; once that time expires without an answer, the app shows an error and gives up. When a request sits in a queue past its timeout, every CPU cycle the server spends afterward running database queries or business logic is wasted work. During a retry storm, server CPU usage can hit 100% and Throughput stays at its peak, while Goodput drops to zero. The system runs at full speed just to finish requests that nobody is waiting for anymore.

In technical documentation, automatic retry is often described as a self-healing feature. Inside a distributed system running near its limit, persistently resending requests acts instead as a self-jamming mechanism.

III. Positive Feedback Loops and Metastable Failure: The Mechanics Behind the Trap #

Transient Triggers and Sustaining Mechanisms #

In 2022, a research team from Harvard University, Meta, Microsoft, and Carnegie Mellon University led by Nathan Bronson published Metastable Failures in Distributed Systems at OSDI². Their paper named and analyzed one of the toughest failure modes in system operations: metastable failure.

A metastable failure is a self-sustaining overload state where a system gets stuck and cannot recover on its own even after user traffic drops back to normal. At the exact same level of incoming user traffic, the system can sit in one of two very different equilibrium states. The first is the healthy state: fast responses, empty queues, and Goodput matching hardware capacity. The second is metastable failure: full queues, response times exceeding client timeouts, and Goodput near zero.

Sliding from the healthy state into metastable failure always involves two separate parts:

The first part is the trigger. This is a brief disturbance that slows the system down for a short period. It can be a ten-minute traffic spike, a brief pause for memory cleanup, or thirty seconds of network packet loss. Crucially, this trigger is temporary and goes away on its own.

The second part is the sustaining mechanism. This is a positive feedback loop that multiplies work and keeps the system jammed long after the trigger has ended. In most major outages, the sustaining mechanism is a retry storm coming from user devices or cascading between internal services.

Feedback Gain and the Critical Boundary #

The growth rate of a server queue equals total incoming requests minus the server’s actual processing capacity. Incoming requests consist of original user traffic plus retry traffic caused by errors. I refer to the number of new retry requests generated by each failed request as the feedback gain.

This feedback gain divides system behavior into two opposite regimes:

When the feedback gain is less than 1, the system stays in the safe regime. Each failed request produces, on average, less than one new retry. Excess load shrinks with each cycle, allowing the system to absorb the hiccup and return to its normal operating point.

When the feedback gain reaches 1 or higher, the system enters the self-amplifying regime. Each failed request spawns at least one new retry. Incoming work grows faster than the server can clear its queue, pushing the system straight into metastable failure.

When a system goes down after a fifteen-second network glitch, incident reports often blame those fifteen seconds. But to me, blaming the trigger is like blaming a match for burning down a fuel depot. What actually keeps the fire burning for the next five hours is the retry loop we carefully wrote into our code.

IV. From Dynamics to Your System: The Mapping Table #

System outages do not happen in an idealized environment. As queues grow longer, a server’s actual processing capacity does not stay flat; it drops sharply.

This drop comes from physical hardware limits. The CPU spends more and more time on context switching (pausing one task thread to switch to another) across thousands of open worker threads (independent execution paths inside a program). At the same time, database connections are held hostage by requests waiting in line. As RAM fills up, garbage collection (the automatic process that reclaims unused memory) runs repeatedly and freezes the application. The more tasks the server takes on at once, the fewer tasks it actually finishes per second.

Watching the Hang Xanh roundabout in Ho Chi Minh City at 6:00 PM, I see motorcyclists riding onto the sidewalk not because they are in more of a hurry than usual. They do it because everyone figures creeping forward one extra meter will get them through. On a server, thousands of client apps similarly assume one more retry cannot hurt, turning the intersection into an open-air parking lot.

The table below connects the concepts of non-linear dynamics to familiar traffic patterns in Ho Chi Minh City and components of a distributed software system:

The four-step slide from a small hiccup into metastable failure follows a predictable path:

1. Brief glitch: A minor storage slowdown raises response time from 20 milliseconds to 200 milliseconds for 15 seconds.
2. Retries kick in: Client apps hit their 150-millisecond timeout limit. Thousands of clients resend requests at once, pushing the feedback gain past 1.
3. Capacity drops: Server queues fill up. CPU usage hits 100%, but 80% of that time goes into opening and closing connections, logging errors, and running garbage collection, cutting real processing capacity in half.
4. Locked in metastable failure: Even though the storage slowdown ended after 15 seconds, the backlog of retries keeps feeding itself. Goodput drops to nearly zero.

No memory stick burned out, and no code had a syntax bug. The system went down because of the feedback loop built right into its design.

V. The Hysteresis Loop: Why Systems Cannot Recover on Their Own #

Two Stable Equilibria and the Hysteresis Loop #

After a retry storm, the hardest question operations engineers face is simple. Why did the servers stay down even after the flash sale ended, users went to sleep, and incoming traffic dropped below normal levels?

The answer lies in the two stable equilibria and the hysteresis loop (the phenomenon where a system stays stuck in a bad state even after external conditions return to normal) found in non-linear systems³.

Think of the Hang Xanh roundabout or the Dan Chu six-way intersection in Ho Chi Minh City during rush hour. When the number of vehicles entering the roundabout crosses the intersection’s capacity limit, traffic slows down. Instead of waiting behind the line, motorcyclists edge onto the curb and fill every gap in the middle of the junction. A cross-locking loop forms: vehicles from one direction block the path of vehicles from the cross street, locking everyone into a rigid jam.

At that point, even if the lights turn green or traffic from distant streets thins out below normal levels, the center of the roundabout stays frozen. Bikes pressed wheel-to-wheel have no room to move; that is the clearest physical picture of a hysteresis loop. To get traffic moving again, traffic police have to temporarily block incoming roads from afar. Once new entries into the junction drop near zero and the trapped vehicles clear out, police can let traffic flow back in waves.

In distributed software, the hysteresis loop works the exact same way:

To break this self-sustaining loop, operators cannot simply wait for user traffic to return to pre-outage levels. They have to pull actual load well below the recovery threshold by blocking connections at the edge, turning on maintenance mode, or restarting clusters in stages. Only after flushing out expired requests in the queue can the system let user traffic back in.

Suppose you are on call that night watching CPU usage sit at 100%: it is easy to think the server is busy fulfilling customer orders. In reality, it is just answering requests whose senders closed their apps two hours ago.

Two Forensic Case Studies: DynamoDB and Slack #

The 2015 DynamoDB outage mentioned at the start of this post is a classic example of hysteresis. An internal metadata service slowed down because partition data payloads grew larger. When this service took too long to respond and timed out, thousands of storage servers assumed they had lost contact with the cluster.

Automated recovery code on those storage servers immediately triggered waves of reconnects and retries against the metadata service. The internal retry storm brought DynamoDB to a halt, which then cascaded into EC2 Auto Scaling and other AWS services. To save the system, AWS engineers had to block all traffic to the metadata service and clear out the backed-up queues. Only then could they slowly let connections back in step by step.

Five years later, on January 4, 2021, the messaging app Slack suffered a global outage⁴. It lasted nearly five hours on the first workday after the New Year holiday. Millions of people opening the app at once created a wave of reconnects over WebSocket (a protocol that keeps a two-way communication line continuously open between an app and a server). At the same time, the backend had to read data from disk because the cache was not warmed up yet, overloading an AWS Transit Gateway and leaving web worker threads stuck waiting for network data.

Right then, Slack’s autoscaling system (the mechanism that automatically adds or removes servers based on load) looked at average CPU usage to make scaling decisions. Because worker threads were sitting idle waiting on network read/write operations (I/O) rather than crunching numbers, actual CPU usage dropped very low, tricking the controller into shutting down active web servers. Moments later, thread exhaustion triggered an emergency order to spin up 1,200 new servers at once. That sudden wave overloaded internal configuration services and hit AWS API rate limits. Slack recovered only after engineers switched their load balancers into an emergency load-shedding mode and manually expanded network gateway capacity.

VI. Three Quantitative Simulations #

To test these rules with concrete numbers, I built three independent numerical simulations. The full Python code used to generate the charts is included in the deep-dive box at the end of this section.

Simulation A: Exponential Amplification on Call Trees #

The first simulation measures how requests multiply across a chain of services calling one after another. In this scenario, the failure rate at the bottom service rises from 0 to 95%, and each tier is allowed up to 3 retries.

The simulation numbers show how fast load escalates across a chain of 5 connected services. When the bottom service fails only 10% of the time, load amplification is just 1.52x and user success rate stays at 99.5%. Once the failure rate crosses 50% and reaches 70%, load amplification jumps to 22.99x, moving toward the theoretical ceiling of 3 to the 4th power, or 81x. At that point, user transaction success drops to 12.28%. Retrying at every tier does not save the transaction; it turns a local slowdown into a system-wide overload.

Simulation B: The Hysteresis Loop and Goodput Collapse #

The second simulation solves the differential equation to track Goodput across varying user traffic. I ramped incoming load up from 100 to 1,350 requests/second during loading, and then back down from 1,350 to 100 requests/second during unloading.

The chart traces a clear hysteresis loop. As load increases, the system keeps Goodput high until it hits the overload threshold around 950 requests/second. At that point, latency crosses the 50-millisecond timeout mark, a retry storm kicks in, and Goodput drops straight from 900 requests/second to nearly zero. When incoming load is dialed back down, the system does not recover at 950 requests/second; it stays stuck in metastable failure. Operators have to cut incoming load all the way down to the recovery threshold around 320 requests/second (nearly a 70% reduction) before the system can clear its backlog and recover.

Simulation C: Breaking Phase Synchronization Among Clients #

The third simulation tracks 4,000 client devices that hit a network glitch at second 0. From that point, I compared four retry waiting strategies across a 35-second window.

Simulation C highlights an ironic trap. Many developers faithfully implement textbook Exponential Backoff (doubling the wait time after each failed attempt, such as waiting 1, 2, 4, and 8 seconds). Yet if they forget to add jitter (a random time offset added or subtracted so devices do not retry at the same moment), all those clients wake up on the exact same second. Instead of spreading out the load, I find we accidentally train thousands of clients to punch the same punching bag on the exact same beat.

Fixed exponential backoff causes independent clients to line up into synchronized waves that hit the server. Those load spikes arrive right at seconds 1, 3, 7, and 15, with each peak exceeding 3,000 requests/second and knocking down the server’s recovery attempts. By contrast, Marc Brooker’s Full Jitter algorithm⁵ spreads retry calls evenly across the timeline. By flattening the spikes and keeping peak load below 400 requests/second, this algorithm gives the server enough breathing room to drain its queue.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

import numpy as np

# 1. Microservices tree amplification calculation (Sim A)
def retry_amplification(N, k, p):
    E_C1 = np.where(np.isclose(p, 1.0), float(k), (1.0 - p**k) / (1.0 - p))
    return E_C1 ** (N - 1)

# 2. Full Jitter and Decorrelated Jitter algorithms (Sim C)
def simulate_jitters(num_clients, max_retries=4, t_base=1.0, t_cap=20.0):
    full_jitter_times = []
    decorr_jitter_times = []

    for _ in range(num_clients):
        # Full Jitter
        t = 0.0
        for i in range(max_retries):
            t += np.random.uniform(0.0, min(t_cap, t_base * (2 ** i)))
            full_jitter_times.append(t)

        # Decorrelated Jitter
        t = 0.0
        prev_interval = t_base
        for _ in range(max_retries):
            interval = min(t_cap, np.random.uniform(t_base, prev_interval * 3.0))
            t += interval
            prev_interval = interval
            decorr_jitter_times.append(t)

    return np.array(full_jitter_times), np.array(decorr_jitter_times)

VII. Three Things to Do and What Each Costs #

Banning retries completely is an extreme overreaction: a two-millisecond network blip would unnecessarily fail user transactions. Yet allowing unlimited retries is reckless, because it sets up a positive feedback loop just waiting for a trigger.

In my view, a system designer’s job is not to eliminate retries. Instead, we need to install control valves so the feedback gain never crosses 1. Here are three practical defenses proven in large-scale systems:

1. Retry Budgets #

Instead of giving every single request a fixed quota of retry attempts, the system should manage retry permission as a shared allowance across the entire client app.

Following Google SRE practices and the internal service communication framework gRPC⁶, each client monitors a recent sliding time window. A failed request is allowed to retry only if retries currently make up 10% or less of all outgoing traffic from that client.

In the AWS Software Development Kit (SDK), this works through a token bucket model (a counter that acts like a prepaid wallet, adding tokens on success and spending them on failures). Each client starts with a bucket holding up to 500 tokens. Every request that succeeds on the first try adds 1 token, while every retry after an error spends 14 tokens. When the bucket runs empty, all retries are blocked right at the client, forcing the app to return an error immediately instead of piling more work onto the server.

This acts like an automatic fuse. During normal operations, occasional network glitches are retried without issue. When servers face widespread overload, the bucket drains quickly and shuts the retry valve at the source.

2. Deadline Propagation and Cancelling Expired Work #

To eliminate wasted work, every service in a call chain must share a single countdown clock for each user request.

When a request enters the API Gateway (the front-door service that receives and routes incoming user requests), it gets a maximum wait budget such as 500 milliseconds. The gateway stamps an exact expiration time onto the request header.

In gRPC, this expiration timestamp travels automatically via the grpc-timeout field. When service A calls service B, and B calls service C, that original expiration time is passed along. The remaining time ticks down rather than resetting to a fresh 500 milliseconds at each hop.

Before pulling a task from its queue, each service checks whether the remaining time is still greater than zero. If the deadline has passed or the remaining time is too short for the minimum step, the service drops the request immediately without calling the next tier.

In addition, when a user closes their browser tab or cancels an action, the system sends a cancellation signal (such as context.WithCancel in Go). This signal travels down the call chain to stop in-flight database queries right away.

3. Adaptive Concurrency Limits and Fast Load Shedding #

Fixed capacity limits (like a hardcoded 1,000-item queue or a static pool of 200 connections) easily backfire when conditions change. A queue size that works smoothly on a normal day becomes a holding pen for expired requests when database response times triple.

Modern architectures use adaptive concurrency limits based on continuous measurements of RTT (round-trip time — the time it takes to send a request and receive a reply). Netflix’s Gradient algorithm is a prime example: the server continuously divides its baseline no-load response time by its actual current response time.

When queues start building up, actual response time grows and pushes that ratio below 1. The system automatically shrinks the number of requests allowed to run at the same time, protecting the work already in progress.

When incoming traffic exceeds that safe limit, the entry gateway immediately rejects excess requests using HTTP status codes 429 Too Many Requests or 503 Service Unavailable. Rejecting a request at the front door takes less than a thousandth of the resources required to queue it until it times out. Serving 70% of users cleanly is always far better than accepting 100% and making everyone stare at an error screen.

When teams discuss active load shedding, many engineers feel reluctant, as if they are turning customers away. Yet I find returning an HTTP 503 error at the front door in 2 milliseconds is far kinder. It is much better than letting users into a queue, making them watch a spinning wheel for 30 seconds, and then timing out anyway.

Architectural Trade-off Matrix #

Every defense above comes with concrete implementation and operational trade-offs:

VIII. References #

• Bronson, N., Aghayev, A., Charapko, A., Stavrinos, T., & Zhu, T. (2022). Metastable Failures in Distributed Systems. Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI ‘22), 393–408.
• Strogatz, S. H. (2014). Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering (2nd ed.). Westview Press.
• Brooker, M. (2015). Exponential Backoff And Jitter. AWS Architecture Blog.
• Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (2016). Site Reliability Engineering: How Google Runs Production Systems. O’Reilly Media.
• Amazon Web Services. (2015). Summary of the Amazon DynamoDB Service Disruption in the US East Region. AWS Post-Event Summaries.
• Slack Engineering. (2021). A walkthrough of each incident from January 4th. Slack Engineering Blog.