Paper 03: Normal Accidents Theory & The Fallacy of Root Cause Analysis

I. The Clockwork Hangover #

Engineers are taught that systems break because a part breaks: find the part, replace it, done. For modern distributed systems that model fails in a way that has a name. Charles Perrow showed that when two properties are both high — components interacting in tangled ways, and no slack left between them — accidents stop being anomalies and become normal behavior of the structure itself. This paper takes those two properties apart, shows why the “human error” label is usually a convenient place to close the file, and replaces the root-cause hunt with a different question: which safety constraint was violated, and which control structure was supposed to hold it.

Engineers grow up with the mechanical-watch model. Something broke? A gear cracked, or somebody forgot to wind it. Take the parts apart and you understand the whole. Tidy, satisfying, and for simple systems, correct.

That model breaks the moment you step into a complex adaptive system — a system whose parts continuously act on each other and change their own behavior in response. A typical cloud-native architecture qualifies: the autoscaler reacts to load, the load balancer reacts to the autoscaler, the retry policy reacts to both.

The STELLA report from the SNAFUcatchers named the most important gap in systems of this kind: what sits below the line and what sits above the line. Below the line is the real system — codebase, hardware, network paths — which nobody observes directly. Above the line is what people actually touch: dashboards, logs, alerts. The two never match. An on-call engineer does not debug the system; they debug a picture of the system, drawn by whichever metrics somebody chose to collect six months ago.

So when an outage lands, the first reflex is to hunt a root cause. This paper argues that hunt usually comes back empty — and that the accident itself is the normal thing.

II. Two Events That Look Alike: Failure vs. Accident #

To reason about collapse at all, you have to separate two events that look identical from outside and behave nothing alike:

• Expected failure. Local, bounded, below threshold. A pod exhausts memory and gets killed. The system already has the absorption for it: the orchestrator notices, shifts traffic to another node, reschedules the pod. This is in the design, and it ends where it started.
• Systemic accident. Non-linear, unbounded, past threshold. The unnerving part: every component is behaving exactly as programmed. There is no bug. But across thousands of nodes, the self-protection mechanisms resonate into a self-amplifying loop — the retry storm from the previous paper is the standard case. The accident is produced by the interaction of safe behaviors, travelling through joints that are coupled too tightly.

The distinction decides how you investigate. For the first kind, finding the broken part is the right work. For the second, there is no broken part to find.

III. Perrow’s Frame: Two Deciding Variables #

Charles Perrow, after analyzing Three Mile Island, reduced the structural risk of any system to two variables.

Interactive complexity ( $\chi$) — how tangled the interactions are. Not “how many components the system has,” but “how many paths of influence exist that nobody has drawn.” The tell: changing A breaks D, and there is no direct call between A and D. In software architecture it comes from transitive dependencies, shared state, and the sidecar or service-mesh layers sitting between every call.

Tight coupling ($\tau$) — how little slack sits between parts. It is the inverse of buffer: how much time, how much holding space, how many alternate paths exist between two components. A five-tier synchronous gRPC chain is tightly coupled — the bottom tier stalls and all five wait. A queue between two services is loosely coupled — the consumer can die for ten minutes and the messages are still there.

1. Risk Is the Product of the Two #

The relationship can be written as an equation, with a warning attached right here: this is a heuristic for thinking, not an instrument for measuring. It shows the shape of risk; it does not produce a number to forecast with.

In words: risk accumulates over operating time, and it accumulates as the product of the two variables, not their sum. A tangled system with loose coupling survives; a tightly coupled but simple one survives too. Only when both are high does $P_{acc}$ run toward 1 — and once it has, the question stops being “will this happen” and becomes “when.”

Substitute real numbers: at $\chi = 1$, a loosely coupled system ($\tau = 0.5$) gives $P_{acc} \approx 39\%$, while the same system tightly coupled ($\tau = 3$) gives $P_{acc} \approx 95\%$. Same complexity, only the coupling changed, and risk moves from “occasionally” to “near certainty.”

2. Mapping to Architectural Metrics #

IV. The Path Into a Stuck State #

A system falls into a stuck state (metastability) when the rate of failure spread outruns whatever absorption capacity it has left. The trigger is nearly always the same: a shared resource saturates — thread pool, network buffer, database connection queue.

The mechanism worth noticing is how a shared resource converts a local event into a global one. While there is headroom, one retry is one surplus request. Once the thread pool is full, that same retry takes the slot of a request already waiting, pushes that request into timeout, and produces one more retry. Behavior that is entirely safe on one node becomes a storm across a thousand — not because its nature changed, but because the slack ran out.

V. “Human Error” Is a Stopping Place, Not an Answer #

Concluding that an incident was caused by “human error” is a stop rule¹ — a convenient place to close the file, usually chosen because it costs less than admitting an architectural defect.

Two concepts explain why the label is nearly always wrong:

• Local rationality. Sidney Dekker and Richard Cook make the argument: no engineer comes to work intending to break the system. From where they stood, with what the dashboard showed them at that moment, their action was reasonable. It only looks foolish in hindsight, with information nobody had at the time. An investigation built on “they should have known” is penalizing people for lacking something the system never gave them.
• Graceful extensibility². David Woods’s term for a system’s capacity to stretch past every scenario it was designed against. In large incidents, the humans operating the system are usually not the cause of the failure. They are the source of recovery: the person who invents a load-shedding move no runbook describes, the person who decides to turn off one feature to save the rest. Automation cannot do that, because automation only handles scenarios someone thought of in advance.

VI. Replacing RCA with STAMP and CAST #

Traditional Root Cause Analysis rests on the domino image: find the first one that fell, stand it back up, done. In non-linear systems that image is wrong at the premise — the dominoes are not in a line, they are in a mesh, and several fall at once.

Nancy Leveson’s STAMP replaces the event chain with a control structure: the system is viewed as a set of safety constraints plus the mechanisms responsible for holding them. An accident happens not because a component failed, but because a safety constraint stopped being enforced by anything.

CAST is the matching investigation process. What separates it from RCA: it does not ask “what broke” or “who erred,” it audits the whole control structure — the technical part (Kubernetes, circuit breakers, alerts) and the organizational part (CI/CD gating, on-call policy, alert thresholds)³.

Three questions replace the root-cause question:

1. Which safety constraint was violated? (For example: “never allow more than 30% of a service’s pods onto a single node.”)
2. Which mechanism was supposed to hold that constraint, and why did it not hold?
3. Which feedback loop was broken — what signal should the operator have seen and did not?

These produce a different repair list. RCA produces “fix the bug in service X.” CAST produces “we have no mechanism that detects this constraint being violated, and that is the work item.”

VII. Simulating the Shape of Risk #

The simulation below is not for prediction — it is for seeing. It shows one thing: tighten a complex system and the risk curve does not creep up, it snaps up.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

import numpy as np
import matplotlib.pyplot as plt

def model_system_risk(complexity, coupling):
    # Heuristic: risk accumulates as the product of both variables, converging to 1
    return 1 - np.exp(-(complexity * coupling))

complexity_range = np.linspace(0.1, 5, 100)
coupling_levels = [0.5, 1.5, 3.0]  # loose (queues), medium, tight (sync calls)
colors = ['#D4D4D4', '#FB923C', '#F97316']

plt.figure(figsize=(10, 5))
for level, color in zip(coupling_levels, colors):
    risk = [model_system_risk(c, level) for c in complexity_range]
    plt.plot(complexity_range, risk, label=f'Coupling τ = {level}', color=color)

plt.title('Shape of Systemic Collapse Risk')
plt.xlabel('Interactive Complexity (χ)')
plt.ylabel('P(Accident)')
plt.legend()
plt.show()

Reading the chart: at tight coupling ($\tau = 3$), risk passes 90% while complexity is still at $\chi = 0.8$. At loose coupling ($\tau = 0.5$), reaching that same threshold takes $\chi = 4.6$. Same system, only the way components wait on each other changed, and the safe complexity budget is nearly six times wider.

VIII. Four Things to Do This Quarter #

1. Create temporal slack. Convert synchronous dependencies into asynchronous event streams wherever the business logic permits. This is the most direct way to pull $\tau$ down, and per the equation in Section III, the cheapest.
2. Invest in observability, not just telemetry. The right question is not “do we collect this metric” but “at 2 a.m., can the on-call engineer see which constraint is being violated?” This is the work of closing the above-the-line/below-the-line gap from Section I.
3. Design so the system can stretch. At the limit, the architecture must leave room for a human to act — manual breakers, feature kill switches, load-shed valves — instead of locking into a stuck state.
4. Protect constraints, not just components. Write the system’s safety constraints down as specific sentences, then build detection for their violation. That list should include organizational constraints, not only technical ones.

IX. Conclusion #

Normal Accidents Theory and STAMP offer a modest but practical view: we do not fully control a complex system, we only set and hold the constraints that govern it. Accepting that systemic risk is a property of the structure itself, we stop chasing the root-cause illusion and spend the effort on what actually helps — making the system survive things nobody thought of.

X. References & Further Reading #

• Perrow, C. (1984). Normal Accidents: Living with High-Risk Technologies.
• Cook, R. I. (1998). How Complex Systems Fail.
• Leveson, N. (2012). Engineering a Safer World: Systems Thinking Applied to Safety.
• Dekker, S. (2006). The Field Guide to Understanding ‘Human Error’.
• Woods, D. D. (2018). The Theory of Graceful Extensibility: Basic Rules that Govern Adaptive Systems.
• SNAFUcatchers Consortium (2017). STELLA: Report from the SNAFUcatchers Workshop on Coping with Complexity.