Paper 05: Self-Organized Criticality — Why Systems Drift Toward the Edge of Collapse

I. Abstract #

The monitoring screen reads 80% CPU and zero errors; intuition says 20% to spare, so the system is safe. The Bak, Tang and Wiesenfeld sandpile model and Highly Optimized Tolerance, a theory of what optimization costs, say otherwise. A multi-server system does not stay in the safe zone; it drifts toward a critical state, where the same small fault is harmless today and spreads across the whole cluster tomorrow. There, small incidents are common, and large ones are rare but never rare enough to ignore. What drives the drift is not a law of nature, but round after round of cost-cutting. This paper explains the mechanism, and three ways to hold a system back from the edge, with the price of each.

II. Systems Drift to the Edge Through Decisions That Look Right #

Suppose you work at a food-delivery app in Ho Chi Minh City. At 11:45 on a Friday, the dashboard shows 80% CPU and zero errors; looking at that screen, my first instinct too is that 20% free means safe. Five minutes later the cache restarts, and the app freezes for the rest of the lunch rush.

A dashboard is a screen of a system’s health numbers. A cache holds frequently used data so the database is not asked every time; when it is empty, every lookup hits the database. The cache restarts a few times a week and has never caused trouble. The trigger did not change. The system did.

Six months earlier, the cluster ran at 50% CPU at peak, when requests pour in fastest. A request is one message from an app to a server, like one tap on “Place order”. Every sprint, the team cut a few servers because there was “plenty to spare”: 50% became 60%, 70%, then 80%, with no errors. A sprint is a short work cycle, often two weeks.

That busy share is utilization; the free part is headroom. In the quarterly review, each cut is a green line in the cost report. In the real system, it is a withdrawal from headroom that nobody had to sign for.

Behind those decisions sits linear intuition: effects scale with causes, and while load stays below capacity, everything is under control. To be safer, add protective layers: retries, cache, autoscaling. A retry resends a request when no answer comes back. Autoscaling adds servers automatically when load rises.

Jens Rasmussen (1997), a safety researcher, redrew that picture with three boundaries around every system run by people:

• The economic boundary: too expensive and too slow, and the business dies.
• The workload boundary: push people too hard, and the team breaks.
• The safety boundary: cross it, and the system collapses.

The system’s current position between those boundaries is its operating point. Business pressure pushes it away from the economic boundary; the wish for less effort pushes it away from the workload boundary. Neither force lets up, so the point is squeezed toward the third boundary, though nobody decides that. At the food-delivery app, the server cuts are the first force.

Sidney Dekker (2011) calls this drift into failure: sliding toward an incident through small steps that each look reasonable. The drift, he argues, comes from success, not carelessness. Each time you load more work onto a machine or drop a check to save a few milliseconds, the system looks cheaper. In return, you are spending spare capacity that no dashboard displays.

Then a trigger arrives: a few seconds of slow network, an empty cache after a holiday, an oversized config file. One time it is a ripple, the next a multi-hour outage. Effects out of proportion to causes. That is the sign of a critical state: a small disturbance may die out at once or spread through the whole system, with no telling which. Why a system walks into it on its own is Section III’s job.

III. Two Models: A Sandpile That Steepens Itself, and the Price of Optimization #

The Sandpile That Finds the Edge on Its Own #

Bak, Tang and Wiesenfeld (1987) described this with a sandpile. Anyone who has poured sand or rice from a bag has seen it: the pile steepens to a certain slope and holds there. Any extra grain can set off a slide, big or small, with no telling which. Their computer version needs only a square grid and the ability to count to four:

• Drop grains of sand one at a time onto random cells.
• A cell that reaches 4 grains topples: it gives 1 grain to each neighbor. Grains that fall off the edge are lost.
• A neighbor may reach 4 grains and topple too. That chain is an avalanche; its size is the total number of topplings.

Nobody tunes anything, yet the grid always fills to a certain level and stays there, wherever it started. That level is the critical state, and the grid finds it by itself. Physicists call this Self-Organized Criticality, or SOC, the phrase in this paper’s title.

In the critical state, avalanche sizes follow a power law. A power law is a distribution with many small events and few large ones, where the large ones are never rare enough to ignore. Human height is the opposite: most people are near average, and nobody is ten times taller than anyone else. By the slope measured in Section VI, a 1,000-toppling avalanche is only about 160 times rarer than a 10-toppling one.

So there is no typical size. The same grain disturbs 2 cells today and brings down the whole grid tomorrow. For software, “how big is a typical incident” has no good answer, so capacity planning built on the average incident tends to fall short. Capacity planning is working out in advance how many machines you need.

Two conditions keep the pile in this state:

• Slow loading, fast release. Sand arrives one grain at a time and avalanches happen almost instantly, so they do not overlap and the grid returns to the critical level.
• A branching factor of 1. The branching factor is the average number of new topplings one toppling causes. Below 1, incidents die out: 0.8, then 0.64, then nothing. Above 1, they blow up: 1.2, then 1.44, then no stopping. At exactly 1, a small disturbance can spread to any size.

In a server cluster, the branching factor is the average number of servers one failing server takes down with it.

Nobody designed the sandpile¹. Software systems have designers. If one also sits at the edge, who put it there?

HOT: The Edge Is a Product of Design #

Carlson and Doyle (1999) proposed Highly Optimized Tolerance, HOT for short, as a counterweight to SOC. In HOT, power laws in engineered systems come from the optimization problem itself. Teams must run as efficiently as possible within a cost limit while defending against a known list of risks. Its solutions are usually sturdy yet brittle: sturdy against the disturbances that were planned for, and very brittle against anything off the list.

The mechanism: everyday defenses gather small risks and shift them toward rare, large incidents. Retries make scattered network errors vanish from the statistics, and in exchange open the door to a retry storm, when masses of requests are resent at once. Autoscaling hides small load spikes, and in exchange opens the door to a huge bill, or a mass scale-out that flattens the database. The risk does not go away; it is bought now and paid later, many small incidents traded for a few very large ones.

This is the main point I want to emphasize: a system does not drift to the edge because a law of nature forces it, but because operating teams push it there, one cost-cutting sprint and one utilization bump at a time. SOC describes how the system drifts; HOT names the push, such as the server cuts in Section II, each one right on its own.

IV. From Sandpile to Your System: The Mapping #

The mapping needs one more model from the SOC family: Drossel and Schwabl’s forest-fire model (1992). Trees grow on empty cells, lightning strikes now and then, and a strike instantly burns the whole cluster of trees connected to the tree it hits. The rarer the lightning, the denser the forest grows between fires, and the larger the biggest fire gets.

In software, “dry firewood” builds up each time a small fault is hidden instead of fixed at the root. That happens when a memory-leaking service restarts whenever it fills up, code silently swallows exceptions, or retries keep firing over a dead link. A service is software on a server that does one job, such as payments. Technical debt is the shortcuts taken to hit a deadline and never revisited; it joins the small clusters into one mass, waiting for a big enough strike.

A month with zero alerts looks like a quiet month. By this model, it may also be a month spent stacking firewood.

The table maps both models onto a real system. A cascading failure is a chain reaction: one server fails, its work shifts to another, and that one fails too. A deploy is putting a new version of the code into live use.

V. When a Small Fault Spreads Across the Whole System #

The sandpile’s two conditions translate directly into operations. Slow loading, fast release holds while recovery is much faster than the gap between incidents. Once incidents arrive faster than the team can resolve them, the system never gets back to baseline. The branching factor moves toward 1 each time utilization is raised to save money: the less free room, the more easily one failing server pushes its neighbor over the ceiling.

A small calculation shows why. Suppose the app has 10 servers sharing load evenly and one fails; its work is split among the other 9. At 50%, each goes to about 56%; at 80%, to about 89%, still running but with almost no cushion. At 90%, they land on exactly 100%: the best number a utilization report can show, and the last one before collapse. At 95%, the 9 must carry about 106%, another fails, the last 8 carry about 119%, and the whole cluster goes down.

Call the utilization at which the queue starts lengthening without end the critical threshold. Far from it, a cascading failure is stopped quickly, because some server always has room for the leftover work. The closer to the threshold, the looser that limit; once the branching factor reaches 1, the only limit left is the size of the whole cluster.

So the tipping point here is not a hard capacity threshold being crossed. It is the moment large incidents become common enough for system-wide outages to turn routine. Before and after it, the dashboard looks identical, because it measures today’s load, not the distribution of incident sizes. The diagram sums up the three states and what moves the system between them.

A distributed system is many servers working together over a network. In such systems, the clearest mechanism pushing the branching factor above 1 is the retry storm. With 3 retries, each original request can become up to 4 sends. Near the ceiling, that extra load makes other people’s requests fail too, they resend as well, and load feeds itself after the trigger is gone.

This kind of incident is called a metastable failure: the system stays broken even after the original cause is gone². The September 2015 outage of DynamoDB, a database service on Amazon’s cloud (AWS), is one example: after a brief network disruption, retries kept the service overloaded for nearly five hours. In Slack’s January 2021 outage, autoscaling itself played a part: a burst requesting 1,200 servers within about 15 minutes overloaded the service that sets up new servers. A safety layer became part of the incident, the sturdy-yet-brittle shape from Section III.

Why a system stays stuck after the trigger is gone is the subject of Paper 06. Here, one conclusion is enough: to stabilize, bring the branching factor below 1 and limit how far each cascading failure can spread. Whether the sandpile really finds the edge on its own is what Section VI tests.

VI. Simulation: The Sandpile Finds the Edge on Its Own #

I rebuilt the game from Section III: a 64 × 64 grid, 60,000 grains, counting the topplings in every avalanche. The run produced 23,398 avalanches. Half had 24 topplings or fewer. But roughly 1 in 10 reached 1,000 or more, and the largest had 16,480, about four times the number of cells on the grid.

Every grain is the same. If I had to write the incident report for the 16,480-toppling avalanche, the “root cause” section would be one line: a grain of sand, just like the other 59,999.

How to read the chart: the horizontal axis is avalanche size, the vertical axis (labeled P(s)) is the probability of an avalanche that size, and both use a log scale where each major gridline is 10 times the one before. On this scale a power law is a straight line. The orange dots follow the dashed line up to about 1,000 topplings with a slope of about 1.1, within the range other studies report for two-dimensional grids³.

At the end, the dots drop below the dashed line: the largest avalanche is capped by the edge of the grid, just as an incident’s reach is capped by the size of the cluster. More important, no parameter was tuned to get that straight line; after roughly the first 9,000 grains, the grid reached the critical state on its own and stayed. That is the “self-organized” half of SOC. For software, it means no config switch turns off the drift toward the edge, only constant pushing back.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

import numpy as np

def run_sandpile(size=64, grains=60_000, z_c=4, seed=42):
    rng = np.random.default_rng(seed)
    grid = np.zeros((size, size), dtype=np.int64)
    avalanche_sizes = []

    for _ in range(grains):
        x, y = rng.integers(0, size, size=2)
        grid[x, y] += 1
        toppled = 0

        # A cell holding 4 grains topples, one grain to each neighbor.
        # Sand falling off the boundary is lost - that is the release mechanism.
        while np.any(grid >= z_c):
            xs, ys = np.where(grid >= z_c)
            toppled += len(xs)
            grid[xs, ys] -= z_c
            for dx, dy in ((1, 0), (-1, 0), (0, 1), (0, -1)):
                nx, ny = xs + dx, ys + dy
                inside = (nx >= 0) & (nx < size) & (ny >= 0) & (ny < size)
                np.add.at(grid, (nx[inside], ny[inside]), 1)

        if toppled:
            avalanche_sizes.append(toppled)

    return np.array(avalanche_sizes)

sizes = run_sandpile()
print(f"Avalanche count:  {len(sizes)}")
print(f"Median size:      {np.median(sizes):.0f}")
print(f"Largest:          {sizes.max()}")

# Slope: bin sizes on a log scale, fit a straight line up to 1,000 topplings.
edges = np.unique(np.logspace(0, np.log10(sizes.max() + 1), 30).astype(int))
density, _ = np.histogram(sizes, bins=edges, density=True)
centers = np.sqrt(edges[:-1] * edges[1:])
keep = (density > 0) & (centers <= 1_000)
slope, _ = np.polyfit(np.log10(centers[keep]), np.log10(density[keep]), 1)
print(f"Slope tau:        {-slope:.2f}")

VII. Three Ways to Hold a System Back From the Edge, and What Each Costs #

If economic pressure stays constant and HOT holds, no architecture stops a system from moving toward the edge. What you can do is keep distance from the threshold and limit how far each cascading failure spreads. Measures aimed straight at the retry loop, such as capping the total number of resends or adjusting how many requests a service takes at once, belong to a later paper in this series.

1. Keep headroom; do not run at the critical level. Headroom is the cushion that absorbs triggers, not waste. Some teams cap utilization at around 60% for services that must respond quickly. At that level, the 10-server cluster from Section V can lose 3 servers in a row and the other 7 only reach about 86%. This is a budget decision, not a code decision, because it runs against the very force that pushed the system to the edge in Section II. And headroom is the hardest line item to defend in a budget meeting, because at a glance it looks exactly like paying for machines to sit idle.

   A question for your engineering team: what is our peak utilization, and where does it go if we lose one server?

2. Admission control: limit how fast sand comes in. Admission control means deciding at the door whether to accept a request. Once queues pile up, taking more work only drives the system deeper into the critical zone. Rejecting non-essential traffic early with a 503, the error code for “service temporarily unavailable”, keeps most traffic flowing and stops the cascading failure at its source. At the food-delivery app, that means switching off meal recommendations so ordering keeps working; the price is dropping some real requests on purpose, so the priority order must be right.

3. Chaos engineering as a controlled burn. Chaos engineering means causing failures on purpose, such as shutting down a random server or slowing the network, to see how much the system can take. By the forest-fire model in Section IV, putting out every small failure only stacks firewood for the big fire. Deliberate failures burn the fragile clusters early, at a scale you still control, and force recovery mechanisms to run for real before a real crisis.

VIII. References #

• Amazon Web Services (2015). Summary of the Amazon DynamoDB Service Disruption and Related Impacts in the US-East Region.
• Bak, P., Tang, C., & Wiesenfeld, K. (1987). Self-organized criticality: An explanation of the 1/f noise.
• Bronson, N., Aghayev, A., Charapko, A., & Zhu, T. (2021). Metastable Failures in Distributed Systems.
• Carlson, J. M., & Doyle, J. (1999). Highly optimized tolerance: A mechanism for power laws in designed systems.
• Dekker, S. (2011). Drift into Failure: From Hunting Broken Components to Understanding Complex Systems. Ashgate.
• Dhar, D. (1990). Self-organized critical state of sandpile automaton models.
• Drossel, B., & Schwabl, F. (1992). Self-organized critical forest-fire model.
• Frigg, R. (2003). Self-organised criticality — what it is and what it isn’t.
• Huang, L., Magnusson, M., Muralikrishna, A. B., et al. (2022). Metastable Failures in the Wild.
• Nolan, L. (2021). Slack’s Outage on January 4th 2021. Slack Engineering.
• Rasmussen, J. (1997). Risk management in a dynamic society: A modelling problem.