Paper 03: Normal Accidents Theory & The Fallacy of Root Cause Analysis

I. Abstract: Systems No Longer Break Like Clocks #

Engineers learn that a system breaks because a part breaks: find the part, replace it, done. Charles Perrow showed that when parts interact in tangled ways and no slack remains between them, accidents become an unavoidable feature of the structure instead. This paper separates those two properties. It shows why “human error” is usually a convenient place to close the incident file. From there, we swap the question: which safety constraint was violated, and which mechanism should have held it?

Many incident report templates have a “Root cause” box, and it is rarely left empty, even when no single cause could fill it. Root Cause Analysis (RCA) is how teams fill it: trace back, find one root cause, fix it, close the file. The method assumes a mechanical watch: take the parts apart and you understand the whole, and every failure is a cracked gear. For simple systems that holds.

It fails for a complex adaptive system. In this kind of system, parts keep acting on each other and adjusting their behavior. For example, cloud infrastructure adds machines as traffic rises while apps resend requests that wait too long. This paper argues that in such systems, the root-cause hunt usually comes back empty, and accidents are the normal thing. To see why, separate two events that look alike.

II. Failure vs. Accident: Two Things That Look Alike #

I use one hypothetical scenario throughout this paper: at noon, an e-commerce platform opens a flash sale, and tens of thousands of people tap “Buy”. Each tap sends a request. A request is one message from an app to a server.

The platform is split into services. A service is software that does one job, such as the cart or payments. Each service runs as several copies on many machines; Kubernetes is the software that runs those copies and restarts any that get shut down.

The two events below look alike but unfold very differently:

• Expected failure. Local, bounded, within the design. If a copy of the cart service exceeds its memory allowance and is shut down, Kubernetes restarts it while other copies take the requests. It ends there.
• Systemic accident. Out of proportion to its cause and across every design boundary, yet every component behaves exactly as programmed. When thousands of machines protect themselves correctly at once, those mechanisms reinforce each other into a self-amplifying loop. The accident comes from the interaction of safe behaviors, through joints coupled too tightly.

The textbook case is the retry storm from Paper 02. A retry is the app resending a request after waiting too long. Slow requests trigger retries, retries slow the system further, and apps retry even more.

For the first kind, hunting the broken part is the right work. For the second, I could read the code until morning and every line would still be correct. With no broken part to find, look at the structure.

III. Perrow’s Two Properties That Decide Risk #

Charles Perrow, an American sociologist, analyzed the 1979 accident at the Three Mile Island nuclear plant. In Normal Accidents (1984), he classified systems by two properties. “Normal” here means unavoidable given the structure, not frequent.

Interactive Complexity: Influence Along Paths Nobody Drew #

Interactive complexity is how much parts affect each other along paths nobody planned or can easily see, however many parts there are. The tell: changing A breaks D, though A and D never call each other. In software it comes from indirect dependencies, data several services modify, and intermediary layers between calls. An indirect dependency means A needs B and B needs C, so when C fails, A fails too.

Coupling: How Much Slack Sits Between Parts #

Coupling measures the slack between two parts: spare time, holding space, alternate routes. Tight coupling means almost none is left; loose coupling means plenty.

Anyone who has seen delivery riders lined up at a Vietnamese broken-rice shop at noon has seen tight coupling: when the kitchen runs ten minutes late, so do the riders and their next orders. A shelf where finished orders wait is loose coupling: whichever side is slow, the other keeps working. The comparison has a limit: the shelf does not make food come out faster, and a late order is still delivered late; it only keeps the slow side from holding the other one in place.

In software, five services calling each other synchronously are tightly coupled. A synchronous call means the caller waits for the answer before doing anything else, like the rider at the counter, so one slow service stalls the chain. A queue between two services is loose coupling. A queue holds pending work the way the shelf holds orders: the receiver can be down for ten minutes and the work still waits.

Risk Rises With the Product of the Two #

Perrow gave no equation for this; the model below is this paper’s sketch of the shape of risk, not a forecasting tool. In it, risk builds up over operating time as the product of the two properties, not their sum. A tangled system with plenty of slack survives, and so does a simple one with little slack. Only when both are high does accident probability approach 100%, turning “will it happen” into “when”. For example, at the same complexity of 1 and over one time unit, loose coupling gives about a 39% chance of an accident and tight coupling about 95%.

Mapping to a Real System #

One question remains: how does a small event become a big incident?

IV. Why a System Gets Stuck Instead of Recovering #

Back to the flash sale. At 12:01 the payment service slows and many requests pass their timeout. A timeout is the longest a caller waits before giving up; then the app retries. Retries exist to make apps more resilient, and here they do it by multiplying the load on the jammed spot: with up to two retries, each slow request can become three.

Local usually turns global at a shared resource that runs out, such as processing threads, network buffers or database connections. The easiest to picture is a thread pool. A thread pool is a fixed set of workers in a server, each handling one request at a time, like the shop’s cooks.

With free slots, a retry is one extra request. With the pool full, it waits in line and pushes another request past its timeout, which triggers another retry. Behavior that is safe on one machine becomes a storm across a thousand, not because it changed, but because the slack ran out.

A system falls into this loop when failures spread faster than it can absorb them. Paper 02 calls the end state a metastable failure. That is a failure that sustains itself after the push is gone: buyers have thinned out, yet the payment page keeps spinning. The diagram traces the path from top to bottom.

Every component still behaves correctly, but the root-cause box needs filling, and the nearest candidate is usually a person.

V. “Human Error” Is a Stopping Place, Not an Answer #

Suppose that at 12:05 the on-call engineer restarts the overloaded database, and the system slows another notch while the database reloads frequently used data into memory. The report almost writes itself: “Cause: operator error.”

That conclusion is usually a stop rule¹. A stop rule is where an investigation halts. Here it is a convenient place to close the file, often chosen because it costs less than admitting an architectural problem.

The STELLA report (2017) from SNAFUcatchers, a group of researchers and tech companies, describes the on-call engineer’s position with a line. Below it is the running system, which nobody sees directly. Above it are people whose mental picture of that system comes only from screens such as dashboards and alerts. That picture never fully matches the real thing.

Reading an incident back with the ending known, I always see what should have been done; the engineer at 12:05 had only the picture. Two concepts explain why “human error” is nearly always the wrong label:

• Local rationality. Safety researchers Sidney Dekker and Richard Cook argue that nobody comes to work to break the system. Local rationality means that from where the person stood, with what the dashboard showed, the action made sense. It looks wrong only in hindsight, with information nobody had at the time. Blaming people for what “they should have known” punishes them for lacking what the system never gave them.
• Graceful extensibility². Safety researcher David Woods uses this name for a system’s ability to stretch its responses beyond its design. In big incidents, operators are usually the source of recovery, not failure. Someone sheds load in a way no runbook describes; someone else turns off product recommendations to keep checkout alive. A runbook is a step-by-step guide for one type of incident. Automation struggles here, as it only handles scenarios someone foresaw.

So where should the investigation go instead?

VI. Stop Hunting the First Domino: STAMP and CAST #

RCA pictures a row of dominoes: find the first to fall, stand it up, done. In the flash sale, the dominoes form a mesh, and several fall at once.

Nancy Leveson, an MIT professor, proposed STAMP, an accident model that replaces the chain of events with a control structure. A control structure is a system’s safety constraints plus the people and mechanisms responsible for holding them.

A safety constraint is something the system must never violate, written as a specific sentence. In STAMP, an accident happens when a constraint is not enforced well enough, whether because a part broke or because correct parts interacted badly.

CAST is the investigation method that goes with STAMP. Instead of stopping at “what broke” or “who erred”, it audits the whole control structure. The technical side includes Kubernetes, circuit breakers and alerts; the organizational side includes release approval, on-call policy and alert thresholds³. A circuit breaker cuts off calls to a service that keeps failing, like a household breaker tripping.

Three questions replace the root-cause question:

1. Which safety constraint was violated? For example: “never put more than 30% of a service’s copies on one machine”.
2. Which mechanism should have held it, and why did it not?
3. Which feedback loop broke: what should the operator have seen, and did not? A feedback loop is the path signals take from the system back to whoever controls it.

RCA yields “fix the bug in service X”, or for the flash sale, “remind the on-call engineer to be careful”. CAST yields “nothing detects or blocks violations of this constraint; build it”. CAST’s list is longer and harder to hand to one person, which is the point.

VII. The Chart: How Tight Coupling Pulls Risk Forward #

The chart plots the Section III model at three coupling levels, to see, not to predict. The horizontal axis is interactive complexity, the vertical axis accident probability from 0 to 1. Gray is loose coupling (0.5), light orange is medium (1.5), dark orange is tight (3); watch the dark orange line near the top while the gray one is only about a third of the way up.

Tighten a complex system and risk shoots up early: at tight coupling, risk passes 90% when complexity is only about 0.8; at loose coupling, not until about 4.6. Changing only how parts wait on each other makes the complexity budget before that mark six times wider. That six is no discovery, just 3 divided by 0.5 built into the product. A chart drawn from a formula cannot argue with the formula.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

import numpy as np
import matplotlib.pyplot as plt

def model_system_risk(complexity, coupling):
    # Sketch model: risk rises with the product of both properties, approaching 1
    return 1 - np.exp(-(complexity * coupling))

complexity_range = np.linspace(0.1, 5, 100)
coupling_levels = [0.5, 1.5, 3.0]  # loose (queues), medium, tight (sync calls)
colors = ['#D4D4D4', '#FB923C', '#F97316']

plt.figure(figsize=(10, 5))
for level, color in zip(coupling_levels, colors):
    risk = [model_system_risk(c, level) for c in complexity_range]
    plt.plot(complexity_range, risk, label=f'Coupling τ = {level}', color=color)

plt.title('Metaphorical Risk of Systemic Collapse')
plt.xlabel('Interactive Complexity (χ)')
plt.ylabel('P(Accident)')
plt.legend()
plt.show()

VIII. Four Things You Can Do Now #

1. Create slack in time. Where the business allows, replace synchronous calls with work sent to a queue, like the shop’s shelf. In the Section III model, cutting coupling helps as much as cutting complexity by the same share, and in my view it is usually far easier. Ask your engineering team: when a user taps “Pay”, how many services wait on each other?

2. Invest in observability, not just data collection. Observability means understanding from the outside what a system is doing, which is what narrows the gap STELLA describes. A dashboard of 200 charts does not help if you need chart 201. Ask not “do we collect this metric?” but “at 2 a.m., can the on-call engineer see which constraint is being violated?”

3. Leave room for humans to act. At the limit, the architecture needs ready levers instead of locking into the stuck state: manual circuit breakers, feature switches, load-shedding valves. A load-shedding valve deliberately rejects some requests so the rest keep working.

4. Protect constraints, not just components. Write safety constraints down as specific sentences and build detection for their violation. This applies to organizational rules too, such as “no new code goes live in the two hours before a flash sale”.

IX. We Can Hold Constraints, Not the Whole System #

Normal Accidents Theory and STAMP offer me a modest, practical view: nobody fully controls a complex system. We can only set and hold the constraints that govern it. The effort is better spent on making the system survive what nobody foresaw than on chasing one root cause. Next time you face a “Root cause” box, try writing down which constraint nobody was holding.

X. References & Further Reading #

• Perrow, C. (1984). Normal Accidents: Living with High-Risk Technologies.
• Cook, R. I. (1998). How Complex Systems Fail.
• Leveson, N. (2012). Engineering a Safer World: Systems Thinking Applied to Safety.
• Dekker, S. (2006). The Field Guide to Understanding ‘Human Error’.
• Woods, D. D. (2018). The Theory of Graceful Extensibility: Basic Rules that Govern Adaptive Systems.
• SNAFUcatchers Consortium (2017). STELLA: Report from the SNAFUcatchers Workshop on Coping with Complexity.