↓Skip to main content
  1. Posts/

Paper 02: Entropy — Why a System Split Into Services Is Harder to Control

·2496 words·12 mins
The Illusion of Control: Complex Adaptive Systems in Distributed Architecture Distributed Systems Complexity Theory Resilience Engineering Software Architecture
Table of Contents

Splitting an application into many small services multiplies its states, makes slow requests routine, and can leave the system down after the cause is gone. This paper explains why, and which kinds of data operations still run safely without machines waiting on each other.

I. Abstract #

Intuition says: splitting an application into small parts just makes the code tidier. In reality, multiplying states bring three consequences: slow requests become routine, adding machines slows the system down, and retry storms keep it down after the cause is gone. Complex-systems theory and queueing math explain this. The way out sits in the data layer: operations that only add and never retract, the basis of the CALM theorem and CRDTs, let copies of the data converge without waiting on each other. Decrementing stock or debiting money still pays for coordination.

II. Why Splitting an Application Multiplies Its States #

It is 11:30 a.m. in Ho Chi Minh City. You open a food-delivery app and tap “Place order”. Suppose ten services answer behind that button: menu, price, promotions, delivery fee, driver, payment… Each runs fine almost all the time, yet the chain is often slow, and now and then it goes down in a way no single service caused.

A service is a piece of software on a server that does exactly one job. A request is one message from an app to a server, like one tap on “Place order”.

The old way packs everything into one program, called a monolith. Microservices split it into many services that talk over the network, so each team can update its part without waiting on others. That sounds like dividing up code, but it changes the nature of the whole system.

Inside a monolith, a call between two functions finishes or fails right away. Over the network, a call can be lost, arrive late, arrive twice, or succeed on the other side without this side knowing. Such a system is called a distributed system, and like the AI software in Paper 01, it is only right with some probability.

The word “entropy” in the title keeps its original sense. Entropy measures how many ways a system could currently be arranged1: the more ways, the harder it is to guess which one it is in.

A state is what a service is doing at a given moment, such as waiting for the bank, restarting, or running an old version. All combinations of states across the ten services form the state space2; each combination is one “arrangement” of the app.

A monolith’s parts start, stop, update and share memory together, so most combinations never happen. Split apart, each service restarts, updates and slows down on its own, and those combinations open up. Ten services with 100 states each give 10²⁰ combinations: about 230 times the seconds since the Big Bang, before counting the network.

Going deeper: why states multiply instead of adding
$$ \Omega = |C| \times \prod_{i=1}^{n} |S_i| $$

In words: the system’s combination count is the network’s state count times every service’s state count, since each service can sit in any of its own states.

  • $n$: number of services. $|S_i|$: number of states of service $i$.
  • $|C|$: number of network states, where packets can be lost, delayed, duplicated, or reordered.
  • $\Omega$: the maximum number of combinations, assuming every combination can occur.

Worked numbers: $n = 10$, $|S_i| = 100$, network ignored ($|C| = 1$): $\Omega = 100^{10} = 10^{20}$, about 230 times the age of the universe in seconds ($4.4 \times 10^{17}$). At a billion checks per second, that takes $10^{11}$ seconds, about 3,200 years.

When I hear “we tested all the cases”, my reflex is to nod, reassured. With 10²⁰ combinations at a billion cases per second, “all” takes about 3,200 years. And some behavior lives in no service at all.

A technical conceptual illustration of the Cartesian Product explosion. On the left, a few solid silver cubes representing a Monolith. On the right, those cubes shatter into a vast, expanding galaxy of interconnected glowing cinnabar red points, forming a complex 3D web that fades into a white infinite background. Modern anime vector art style, clean line art, flat colors, cel shading, minimalist professional technical concept, high contrast, pure off-white background.

III. Why Knowing Each Service Is Not Knowing the System #

Understand each gear of a mechanical watch and you understand the watch. Understand each species in an ecosystem and you still cannot predict the whole. Microservices are the second kind of complexity, because they have emergence3.

Emergence is behavior that appears only when the parts run together and belongs to none of them. At 11:30, no customer means to crash the app; each one just taps “Retry” once. The crash lives in tens of thousands of taps combined. Section V shows machines do the same: no service has a line of code that causes the crash; it arises when all of them resend at once.

So summing each service’s performance does not give the system’s; the gap is usually coordination cost. Neil Gunther drew that cost as the Universal Scalability Law, a curve estimating throughput as machines are added. Throughput is the number of requests completed per second.

More machines add parallel capacity, but they must wait for each other and keep their data in sync. The syncing cost rises with the number of machine pairs, roughly the square of the machine count. Past a threshold that cost wins, and more machines mean lower throughput4.

A 4-person meeting has 6 pairs who must understand each other; a 12-person meeting has 66. Anyone who has sat through the second knows it does not end three times faster. For the food app, adding machines to the payment service at lunchtime can mean fewer orders processed. And one kind of slowness is out of reach for more machines: multiplied probability.

IV. At Scale, Slow Requests Are Routine #

Latency is the time from tap to result. Dashboards tend to show p50: half of all requests are faster than that.

Users remember p99: 1 request in 100 is slower than that. That slowest group is called tail latency, and 1% sounds minor until you multiply.

Jeff Dean and Luiz André Barroso did the multiplication in “The Tail at Scale” (2013). Suppose a request waits for 100 services, each with an independent 1% chance of being slow. It is fast only if all 100 are: 0.99 multiplied by itself 100 times, about 37%. So about 63% of requests are slow. If the services all wait on one congested database, they are not independent and the number changes.

Each service reports a lovely number: 99% of requests fast. Combine 100 lovely numbers and customers wait almost two times in three. With the food app’s ten services, each also 1% slow, the figure is 9.6%, about one wait per ten taps on “Place order”. Nothing is broken; this is arithmetic.

The practical fix is a hedged request: when the first request runs late, send a duplicate to a replica. A replica is another running copy of the same service, on another machine.

p95 is the latency 95% of requests beat. A hedged request waits until the p95 mark, then sends the duplicate and takes whichever answer comes first, so only about 5% of requests get one and traffic rises about 5%. The slow request does not get faster, but the customer stops depending on exactly one path. This suits read-only requests; sending a payment twice is another matter.

Read the diagram below left to right: the request fans out to 100 services, waits for all of them, then branches fast or slow.

graph LR User((User)) --> GW[API gateway] GW -->|"parallel call to 100 services"| Fan[Parallel calls] Fan --> S1["Service 1
1% slow"] Fan --> S2["Service 2
1% slow"] Fan --> Sn["Service ...100
1% slow"] S1 --> Join{Wait for every response} S2 --> Join Sn --> Join Join -->|"63.4% of the time at least one is slow"| Slow["Slow request"] Join -->|"36.6%"| Fast["Fast request"] Slow -.->|"past the p95 mark, send a duplicate to a replica"| Hedge["Hedged request"] style Slow fill:#f96,stroke:#333,stroke-width:2px style Hedge fill:#f96,stroke:#333,stroke-width:2px

Question for your engineering team: how many services does the most important screen wait on, and does the dashboard track p99?

V. Why a System Stays Down After the Cause Is Gone #

The nastiest failure is the one that outlives its cause. It is called a metastable failure5: a self-sustaining loop holds the system in a collapsed state.

The usual culprit is the retry. A retry is the caller automatically resending a request that got no answer. When someone does not pick up, my first reflex is also to call right back. Multiply that by tens of thousands of phones at 11:30 and you have a retry storm: masses of machines retrying at once.

Suppose the app’s payment gateway slows down for two minutes and requests to it pass the timeout. A timeout is the longest a caller will wait before giving up. The app retries, customers tap again, and each retry is a new request landing on an overloaded service. Utilization is incoming work divided by the work a service can do, and retries push it near 100%.

Queueing theory is the math of waiting in line. Its Kingman formula gives the price: waiting time rises by dividing by the capacity still free. Call the wait at 50% utilization 1: at 90% it is 9, at 99% it is 99, eleven times worse for 9 more percentage points.

A restaurant kitchen on the app follows the same curve: suppose each order waits 10 minutes to be cooked when the kitchen is 90% busy; at 99% it waits about 110 minutes.

Going deeper: Kingman's formula
$$ E[W] \approx \frac{\rho}{1-\rho} \cdot \frac{c_a^2 + c_s^2}{2} \cdot \tau $$

In words: queue waiting time is a load factor, times how irregular the work is, times the time per job. It is an approximation, best when the system is already busy.

  • $\rho$: utilization, from 0 to 1. $\tau$: mean time to process one job.
  • $c_a$, $c_s$: standard deviation over mean, for the gaps between arrivals and for processing times; 1 when fully random.
  • $E[W]$: mean waiting time in the queue, not counting processing.

Worked numbers with $c_a = c_s = 1$: $E[W]$ is $\tau$ at $\rho = 0.5$, $9\tau$ at $0.9$, $99\tau$ at $0.99$. For the hypothetical kitchen, $\tau \approx 67$ seconds: $9\tau \approx 10$ minutes, $99\tau \approx 110$ minutes.

Little’s Law closes the loop: the number of requests in the system equals the arrival rate times how long each one stays. When the stay stretches, the queue lengthens with it and fills memory and the thread pool. A thread pool is a fixed-size group of workers, like the counters at a bank branch.

New requests time out, the app retries, and the loop feeds itself. The gateway is fast again, yet the queue is full of requests whose senders have left. Removing the cause is not enough; load has to be cut.

Countermeasures come in two layers. The first fixes how retries happen:

  • Exponential backoff: widen the wait between retries, say 1, 2, 4, 8 seconds.
  • Jitter: add a random amount to each wait. Without it, thousands of phones that failed in the same second retry in the same second, in waves called a thundering herd.

The second layer cuts the call path:

  • Circuit breaker: a switch in the code that, after a few failures in a row, cuts the call path for tens of seconds and fails immediately. It works like a home circuit breaker, except it tries letting calls through again on its own. That quiet gap lets the service behind it drain its queue, which retries never allow.

VI. The Way Out: Operations That Need No Waiting #

If machines waiting on each other is what collapses under load, what holds up? The answer sits in the data layer.

According to the CALM theorem (as Hellerstein and Alvaro summarize it), an operation that only adds information and never retracts it runs correctly on every replica without them agreeing first. Such an operation is called monotonic.

  • Monotonic: adding dishes to a group order from several colleagues’ phones; incrementing a view counter; recording “order paid”. Any arrival order gives the same result.
  • Non-monotonic: decrementing stock when one portion is left. The order decides who gets it; without agreeing first, one portion of rice can be sold to two people.

At this point I badly want the same trick for the last discounted item in a flash sale. CALM flatly says no: “in stock” is a fact that can be retracted the moment someone else buys first.

A CRDT (Conflict-free Replicated Data Type) applies this idea. It is a data type whose replicas merge on their own thanks to three properties. The order of changes does not matter, any grouping gives the same result, and receiving a change twice is harmless.

Once the network is back and the replicas have each other’s changes, they converge on the same state. Suppose the group order is built as a CRDT: two phones without signal can still add dishes, and when signal returns the two carts merge. Some editors where several people type at once work the same way.

Decrementing stock or debiting money has no shortcut. There you accept eventual consistency: services disagree for a while, then converge. The usual approach is a Saga: split the process into steps, each with a compensating step, such as returning the portion to stock if payment fails.

A Saga usually pairs with the Outbox pattern: write the data change and the event that notifies other services in one transaction. A transaction is a group of database operations that all succeed or all fail, so data never changes without its event being recorded.

Question for your engineering team: which operations in the product only add data, and which are like the last portion, needing agreement first?

Engineers do not fight entropy. Engineers choose where to pay for it.

VII. References and Further Reading #

  1. Strictly, entropy is proportional to the logarithm of the number of arrangements (Boltzmann’s formula), so when arrangements multiply, entropy adds. ↩︎

  2. State space explosion: the number of states rises exponentially with the number of components, past the point where any verification tool can check them one by one. ↩︎

  3. Emergence: behavior that appears only at the whole-system level and cannot be reduced to, or predicted from, the behavior of any single component. ↩︎

  4. Retrograde scalability: the region of the Universal Scalability Law curve where adding resources (nodes, threads, processes) lowers total throughput, because coordination cost outgrows the benefit of running in parallel. ↩︎

  5. Metastable failure: a system held in a collapsed state by a self-sustaining feedback loop, usually retries, so it does not recover even though the root cause is gone. Escaping usually requires cutting load from outside. ↩︎