Paper 06: Phase Transitions — Why Overloaded Systems Freeze Instead of Slowing Down

I. Abstract #

Intuition says: 10% more load, roughly 10% slower. Real systems do not behave that way. Under overload they change state abruptly, the way water turns to ice, and leave three measurable traces. One: useful completed work does not decline gradually; it drops vertically. Two: once jammed, the system gets less done than it did just before, and stays there even after load falls. Three: before the drop, latency swings harder while its average stays calm, so the drop can be seen coming. Two tools explain this: the physics of phase transitions and queueing theory. Practical consequences: set capacity where the latency curve starts to bend rather than at the highest number ever measured. Put alerts on the swings rather than only on the average; and once the system is stuck, cut load deeply instead of trimming it bit by bit.

II. What “More Load, Proportionally Slower” Misses #

Some bugs take three days to find and ten minutes to fix. The system keeps running, nothing turns red, and users just wait 5 seconds per tap. I open with one because it points straight at where our intuition about load fails.

A service runs smoothly for months with a 10-connection pool¹. A service is software on a server that does one job, such as payments. A connection pool is a set of database connections opened in advance and shared, like a bank branch with exactly 10 counters: customer 11 waits, however many chairs are empty.

Then demand climbs to 200 concurrent requests. A request is one message from an app to a server, like one tap on “Place order”. The 190 extra requests compute nothing, they just wait for a counter, so CPU stays moderate, memory flat, and no alarm fires. Meanwhile p95 jumps to 5 seconds and stays.

Latency is the time from tap to result. p95 is the latency 95% of requests beat: 5 taps in 100 wait longer. By every resource gauge this is a healthy service, with one small issue: users wait 5 seconds. The bottleneck is a hard limit of 10 jobs at once, which CPU and memory never show.

The error is linear intuition: the system’s response changes evenly with load, the number of requests arriving per second. Ask me how much room a server at 80% CPU has, and my first reflex is still “20%”. By that logic, 10% more load means about 10% more latency, and cutting load ends the problem. Every alert threshold on mean latency or error rate quietly assumes the same.

Anyone who has ridden through rush hour in Ho Chi Minh City, on roads packed with motorbikes, knows better. Traffic thickens and keeps moving; then one vehicle brakes hard and the whole stretch stops, with no accident or red light ahead. Inflow has already eased, yet the jam stays, because fully stopped vehicles take time to get rolling. I use this road as the running example, because traffic and requests jam in very similar ways.

The second observation is much larger. In October 1986, the link between Lawrence Berkeley Lab and UC Berkeley, about 400 yards apart across three routers, fell from 32 kbps to about 40 bps (Jacobson & Karels, 1988). A router forwards data between networks; kbps is thousands of bits per second, the same family as the Mbps on a home internet plan. Input load rose by percentages, delivered data fell nearly a thousandfold, and no cable was cut. The network stayed there until the way data was sent was fixed.

The two markers travel together: output jumps in a step instead of sliding, and the bad state outlives its cause. Physics calls that pair an abrupt phase transition. A phase transition is a system switching to a different state entirely once an outside condition crosses a threshold, like water turning to ice below 0°C. In software, it is a service going from “slow” to “frozen”, a jump linear thinking has no room for. Why the system jumps, and why it does not jump back, is Section III’s job.

III. Two Tools: Phase Transitions and Queueing Theory #

Two Ways to Change State #

• Continuous. The system degrades gradually, like a road filling up in the late afternoon. There is no moment to point at, and there is time to react.
• Discontinuous. Like water freezing: above 0°C, water; below, ice. The system can sit in the wrong state for a while: pure water can stay liquid below 0°C until a speck of dust falls in and it all freezes. And going back is harder than going forward.

On the road, that is traffic dense enough that one hard brake stops the whole stretch; once stopped, removing a few vehicles does not restart it.

Physics pictures this as a marble on bumpy ground². Each dip is a state the system can rest in. With one valley, the marble slides gradually as conditions change: a continuous change. With two valleys and a hill between, the marble must jump across: a discontinuous change, and the hill is why the system stays stuck in the wrong valley.

Hold on to this picture. “It crashed and never came back” means the system fell into the second valley, and a small cut in load does not push it back over the hill. For a service, that is requests back at normal levels while users still wait.

Why the Last 20% of Capacity Costs the Most #

The second tool is queueing theory: the math of waiting in line, from bank counters to servers. Its best-known result, from Kleinrock (1975), is that latency rises by dividing by the capacity still free.

Suppose a lunch restaurant takes orders through a food-delivery app, and its kitchen cooks 60 meals an hour. At 30 orders an hour the kitchen is half busy, and by the simplest queueing formula an order spends 2 minutes in the kitchen on average, waiting plus cooking. At 57 an hour it is 95% busy, and that becomes 20 minutes. Real kitchens cook in batches, so their numbers differ, but the curve still shoots up near full.

That busy ratio is utilization: incoming work divided by the work the system can do. The free part is headroom, the slack that absorbs bursts.

On paper, a kitchen at 95% is very efficient. To the person waiting for lunch, it is 20 minutes for a 1-minute meal.

A server handling 1,000 requests per second has the same shape: mean latency is 2 milliseconds (ms) at 50% utilization, 5 ms at 80%, 20 ms at 95%. The step from 80% to 95% is half the step from 50% to 80%, yet adds 15 ms instead of 3. Around 80%, each extra 1% costs about six times what it did around 50%, so “20% headroom left” is not 20% safety left.

This model is still gentle: every job finishes, just later. Real systems do not get that deal: queues have limits, users have limited patience, and the more work fights over resources, the more effort goes into coordination. Those three turn the smooth curve into a drop in useful work. In short, the queueing curve describes the approach to the transition; the two valleys describe what happens at it.

A Limit to State Up Front #

A strict mathematical phase transition exists only in systems with infinitely many parts³. A 200-machine cluster, or Roblox’s 18,000, is finite: its curve is a very steep bend, smoother than theory, not a true jump. So I use “phase transition” as an approximate model. It is tested through three predictions: two states coexisting at one load, less work done after jamming, and an early warning signal. Section IV locates each one in a real system.

IV. From Physics to Your System: The Mapping #

For the mapping, I need two separate measures. Throughput is the number of requests a server finishes per second, including ones nobody is waiting for. Goodput counts only requests answered in time and still useful. Healthy, the two are nearly equal; jammed, they split apart, and this paper is about that gap.

Physics adds three concepts classical queueing theory lacks: two stable states at one load, a small trigger that tips the system from one to the other, and capacity drop. Capacity drop means a jammed system clears less work than it did just before jamming. On the road, vehicles leave a jam more sparsely than they flowed before it, because a stopped vehicle needs extra time to get rolling.

The fourth concept is critical slowing down: the closer a system is to a phase transition, the slower it recovers from each jolt. It is what lets you see the drop coming.

Why the Cliff Is Built Into the Structure #

The argument needs two things every system has — a waiting-time limit and a shared resource — and takes four steps.

1. The queue gets longer. As arrivals approach capacity, queue and latency climb the curve from Section III. Nothing is broken; the system is only slowing down.

2. Timeouts turn everything around. A timeout is the longest a caller will wait before giving up. Past it, the user is gone but the server does not know. It still pulls expired requests off the queue, pays the full processing cost, and writes results into closed connections. I call that dead work: work for requests nobody is waiting for.

In the hypothetical restaurant, a customer waits 40 minutes and cancels, but nobody tells the kitchen, so it cooks, boxes, and bins the meal. Each such meal pushes real orders back, and more customers cancel. The kitchen still turns out 60 meals an hour; more and more of them just go straight into the bin. The example is off in one respect: real delivery apps usually notify the restaurant, while servers usually get no signal. The road cannot show this step either, since stuck drivers do not quit halfway.

This is where goodput splits from throughput: goodput is roughly accepted requests times the share answered before the caller left. Suppose a server processes 1,000 requests per second but only 30% still have someone waiting: throughput is 1,000, goodput is 300. A CPU dashboard cannot tell the two apart, since the server is 100% busy either way.

3. Capacity burns on coordination. A thread is a line of work running in parallel inside a server. A lock lets only one thread change a record at a time; the others wait their turn.

Picture 128 threads updating one record. One holds the lock for 1 ms while the other 127 keep checking whether it is their turn, burning CPU for nothing. Real capacity drops exactly when demand peaks, like adding cooks to a kitchen with one stove: past some number, dishes come out slower.

Gunther (2007) wrote this down as the Universal Scalability Law. It estimates throughput from the number of parallel threads, with a coordination cost that rises with the square of that number. Past a threshold, more threads mean less throughput: with the illustrative parameters in the block below, going from 32 to 128 threads cuts it by about 40%.

4. The loop feeds itself. Lower real capacity means higher real utilization, even with outside load flat. Busier means longer queues, more expired requests, more dead work and contention. The system slides into the second valley and stays, with nothing more needed to hold it.

None of this needs retries. A retry is a caller resending a request when no answer comes; it adds another amplifying loop, but that belongs to a later paper. The cliff exists without it. What remains: when does the system fall, and how do you get it back up?

V. When the System Tips Over, and Why Less Load Is Not Enough #

Three Limits That Create the Cliff #

• Buffer size. A buffer is memory holding waiting requests, and it cannot hold an endless line.
• Timeouts. Every request has an expiry date.
• Contention cost. Shared data makes coordination cost rise faster than the thread count.

Because of these limits, within a load range near the tipping point, one load can match two states. The tipping point is the load around which the system can fall from one state to the other. One is free flow, with high goodput carried over from lighter times; the other is jammed, with low goodput held down by dead work. Which one the system is in depends on its history, not only on current load.

The road works the same way: with the same number of vehicles, a stretch stays clear if traffic was flowing, and stays jammed if a jam was already there. These are the two valleys from Section III, and they explain a familiar incident-report line: “load returned to normal but the system never recovered.”

The Dense-but-Moving Zone #

Kerner, who built a traffic theory from German highway data, adds a refinement⁴: between “free flow” and “jammed” sits a third state, dense and stop-and-go yet still moving plenty of vehicles. Anyone who drives at rush hour knows it. I call it the dense-but-moving zone.

Servers have it too. A machine at 90% CPU does not die on the spot; it can run for hours in heavy contention, latency swinging widely, goodput holding.

This matters for monitoring. Monitoring is the system that watches a server’s numbers and alerts when one crosses a threshold. Treat every entry into the dense-but-moving zone as a crash, and you will wake the on-call engineer at night over a state that can last harmlessly for hours. The alarm belongs on the system leaving this zone for the jammed state, as the diagram shows.

Two Real Incidents #

NSFNET, October 1986. The second observation in Section II belongs to this episode: the first goodput cliff recorded across a whole network. Overloaded routers dropped data packets, senders resent them without slowing down, and more and more of the link carried packets that would be dropped again: dead work, moving down the wire.

TCP is the core protocol for reliable data transfer on the internet. Congestion avoidance is the mechanism Jacobson later added to it: when packets go missing, the sender slows down, which keeps the total traffic sent into the network below the tipping point.

Roblox, October 2021. This 73-hour outage reads like a demonstration of being stuck in the wrong valley. At its center was Consul, the system Roblox’s services use to find each other and read shared configuration⁵. The trigger was streaming, a newly enabled Consul feature. Under heavy read and write load, that feature piled contention onto a single point and pushed the storage layer beneath into an abnormal cleanup mode.

Write latency at p50 jumped from under 300 ms to 2 seconds, a jump rather than a drift, and the systems depending on Consul timed out one by one. p50 is the middle latency: half of all writes are faster.

The most instructive part is the rescue that failed: the engineers replaced the whole Consul cluster with 128-core machines on fast NVMe drives, and it stayed stuck. A core is one processing unit; 128 cores run 128 jobs in parallel. When I see a stuck cluster, my first reflex is also to go find bigger machines.

That reflex fails for the reason in step 3 of Section IV. What bound the system was contention rather than compute, so more cores only enlarged the part of the cost that rises with the square. The system recovered only after the engineers turned streaming off and let users back in from zero, in small steps, through DNS. DNS is the internet’s directory, deciding which server a user reaches.

A jammed state is exited by bringing load very low; the exit cannot be bought with hardware. The 2015 DynamoDB and January 2021 Slack incidents mentioned in the previous paper sit on the same diagram. Section VI tests the three predictions by simulation.

VI. Three Simulations for Three Claims #

Each simulation tests one claim; the full scripts are in the series assets folder.

Simulation A — the approach is a curve that shoots up. It computes the formula from Section III for a server handling 1,000 requests per second and lands on 2, 5 and 20 ms. How to read it: the horizontal axis is utilization; the vertical axis, labeled waiting time, is mean latency on a log scale, each major gridline 10 times the one below. The curve rises gently to 80%, then shoots up past 90%.

Simulation B — capacity drop without retries. This is the rush-hour road as code: a 500-cell ring road with vehicles following the Nagel–Schreckenberg rules. Each step, a vehicle speeds up if the road ahead is clear, slows before reaching the vehicle ahead, and sometimes brakes at random. Barlović added a “slow-to-start” rule: a fully stopped vehicle needs extra time to get rolling. Each density runs from two starts: vehicles spread evenly, and all vehicles packed into one jam.

There is no retry logic in this simulation, no timeout, and no server either. Just vehicles, and one rule: once stopped, slow to start.

Density is the share of cells holding a vehicle: 0.16 means 16 per 100 cells. Flow is vehicles passing a point per step: 0.79 means 79 per 100 steps. The even start gives free flow, holding up to density 0.16 with peak flow 0.79. The jammed start gives the jammed state, flow flat around 0.21, 73% less at the same density. Between densities 0.07 and 0.16 both coexist: the two valleys from Section III.

In that range, free flow is metastable. Metastable means a state that holds only until a large enough jolt arrives; one unlucky brake pushes it into the jam. The 73% belongs to this model, which shows that the gap exists; its size depends on the system. How to read it: density on the horizontal axis, flow on the vertical; orange is free flow, dashed black is the jam, and the gap between them at density 0.16 is the capacity drop.

No parameter was tuned for that result; the “quick to stop, slow to restart” rule is enough. In a server, that rule is a wiped cache being refilled, a drained connection pool reopening connections, memory full of stale data. A cache keeps frequently used data ready so the database is not asked every time; with the cache empty, every request falls through to the database.

Simulation C — the warning comes before the drop. A queue under slowly rising load, from 50% to 99% utilization over 300,000 steps. With critical slowing down, each jolt takes longer to fade as the transition nears; on the road, that is traffic turning stop-and-go before it halts. The simulation tracks queue length, which moves with latency, through two statistics:

• Variance measures how hard queue length swings around its average.
• Autocorrelation measures how much the current value resembles the previous one. Near 1 means the system is “sluggish”: one jolt has not faded before the next arrives.

Variance is 3.2 at 60% utilization, 11 at 75%, 56 at 90% and 707 at 98%; autocorrelation at those points is 0.87, 0.95, 0.985 and 0.999. The key number is at 75%: variance is already over three times its 60% level. Meanwhile, mean queue length has only crept from 1.2 to 2.6 requests, still hugging the bottom of the chart. The mean only looks worrying past about 95%, long after the other two have left the safe zone. That gap is time you are given, if monitoring looks in the right place.

Critical slowing down has served as an early warning in ecology and climate science (Scheffer et al., 2009) and is only starting to be tried on software. How to read the chart: three panels share a utilization axis, with mean queue length on top, variance on a log scale in the middle, and autocorrelation at the bottom. The bottom two climb steadily from 60%; the top one only jumps in the shaded area past 95%.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import numpy as np

rng = np.random.default_rng(7)
T = 300_000                          # number of simulation steps
rho = np.linspace(0.50, 0.99, T)     # load creeps up from 50% to 99%

# Discrete-time queue: each step, some requests arrive and some get served
arrivals = rng.poisson(rho)          # on average rho requests arrive per step
services = rng.poisson(1.0, size=T)  # on average 1 request is served per step
q = np.zeros(T, dtype=np.int64)
cur = 0
for t in range(T):
    cur = max(0, cur + arrivals[t] - services[t])
    q[t] = cur

def lag1_autocorr(w):
    w = w - w.mean()
    return (w[1:] * w[:-1]).sum() / (w * w).sum()

WIN = 10_000                         # rolling window: the last 10,000 steps
for target in (0.6, 0.75, 0.9, 0.98):
    i = int(np.argmin(abs(rho - target)))
    w = q[i - WIN : i].astype(float)
    print(f"rho={target}: mean={w.mean():.1f}  variance={w.var():.1f}  "
          f"autocorr={lag1_autocorr(w):.3f}")

VII. Three Things to Do, and What Each Costs #

A fair objection first. Capacity planning is working out in advance how many machines an expected load needs, and together with queueing theory it is a mature discipline. What does physics add? I see three things:

• A mechanism for the goodput shortfall after overload; a static queueing model has no parameter for capacity drop.
• A leading signal: variance and autocorrelation rise first, mean latency later.
• The line between discontinuous and continuous, which separates a system that degrades gradually and visibly from one that freezes without warning.

Where those three add nothing (services that keep no data between calls, generous timeouts, no shared data), classical capacity planning is enough, and the heavier vocabulary can be put away.

1. Set capacity at the knee, not at the load-test peak. A load test pumps fake traffic into a system to see how much it can take. Its highest number is the peak of the metastable state, the most fragile operating point the system has. I understand the pull of that number, since it looks best in a plan. Planning at “90% of peak”, however, just parks the whole cluster on the cliff edge.

   Anchor capacity at the knee instead. The knee is where the latency curve starts bending up and variance starts climbing; in these simulations, around 70–75% utilization. Everything above it is borrowed time, and the plan should say so. A question for your engineering team: was our capacity figure measured at the knee or at the load-test peak?

2. Alert on the swings, not only on the average. Add variance and autocorrelation of latency or queue length, over a short recent window, to your alerts, calibrated to each service’s own baseline. The dense-but-moving zone also swings hard while staying livable, so watch for autocorrelation heading toward 1 and variance trending up, not an absolute level.

   DAGOR, WeChat’s overload-control system, is the real-world evidence. It accepts or rejects requests right at the entrance based on mean waiting time in the queue. Its cutoff sits at 20 ms, right at the knee, while the timeout is 500 ms. Past the threshold, it immediately drops the lowest-priority 50% of requests at the first service that receives them. Under 10 times normal load at Lunar New Year across 3,000 small services, this kept success rates 1.5 to 2 times higher than running without overload control.

   A question for your engineering team: do our alerts watch how much latency swings, or only its average?

3. Once stuck, cut load deeply, then reopen in steps. In the jammed state, trimming 10–20% of load only holds the system where capacity drop already holds it: load falls, but so has processing capacity. To escape the jammed state, you must cut accepted requests far below normal capacity — by 60–80%, or to zero as Roblox did. Hold that cut until queues drain and resource contention clears, then reopen load in steps. For the restaurant, that is pausing orders in the app until every order someone still waits for is cooked.

   This needs a kill switch built in advance and a rehearsed runbook, not mid-incident improvisation. A kill switch cuts load in a single action. A runbook is a step-by-step guide for one type of incident. A question for your engineering team: if the system jams, who may flip the switch, and when did we last rehearse it?

VIII. References #

• Barlović, R., Santen, L., Schadschneider, A., & Schreckenberg, M. (1998). Metastable states in cellular automata for traffic flow.
• Goldenfeld, N. (1992). Lectures on Phase Transitions and the Renormalization Group. Addison-Wesley.
• Gunther, N. (2007). Guerrilla Capacity Planning. Springer.
• Jacobson, V., & Karels, M. (1988). Congestion Avoidance and Control.
• Kerner, B. S. (2004). The Physics of Traffic. Springer.
• Kleinrock, L. (1975). Queueing Systems, Volume 1: Theory. Wiley.
• Landau, L. D., & Lifshitz, E. M. (1980). Statistical Physics, Part 1 (3rd ed.). Pergamon Press.
• Nagel, K., & Schreckenberg, M. (1992). A cellular automaton model for freeway traffic.
• Roblox Engineering (2022). Roblox Return to Service 10/28–10/31 2021.
• Scheffer, M., Bascompte, J., Brock, W. A., et al. (2009). Early-warning signals for critical transitions.
• Stumpf, M. P. H., & Porter, M. A. (2012). Critical Truths About Power Laws. Science, 335(6069), 665–666.
• Yang, C. N., & Lee, T. D. (1952). Statistical Theory of Equations of State and Phase Transitions. I. Theory of Condensation. Physical Review, 87(3), 404–409.
• Zhou, H., et al. (2018). Overload Control for Scaling WeChat Microservices. ACM Symposium on Cloud Computing (SoCC).