Paper 01: The Illusion of Control — System Design in the Era of AI

I. The Limits of Traditional Programming #

Traditional software runs on one promise: same input, same output, every time. Put an LLM in the middle of a business flow and that promise breaks — and it breaks in a way that can be described mathematically, not randomly. This paper takes that break apart: why hallucination and runaway loops are structural consequences of how a model generates text, not bugs patchable with one more if. Then three layers of containment that restore control without requiring the model to be correct: blocking malformed tokens at generation time, converting “it seems fine” into a committed error rate, and wiring a circuit breaker to signals read from inside the network.

For decades, software engineering ran on a simple promise: write the logic, get the result. Same input, same output, every time. A test that is green today is green tomorrow unless somebody edits the code.

Then we wired Large Language Models (LLMs) into core features, and the promise broke. The same question asked twice can return two different answers. The work stopped being about managing if-else branches and started being about orchestrating things that are only right with some probability.

Keep the old control-first mindset and the cracks turn into wide-scale failures the moment the system meets an input nobody tested against.

II. Where the Difference Lives: FSMs vs. Absorbing Markov Chains #

The shift has a mathematical root, and it is simpler than its name.

Software 1.0 behaves like a Finite State Machine (FSM). An FSM is a machine with a finite set of states plus a table saying: in this state, on this event, jump to that state. A vending machine is an FSM — waiting for coins, receives the full amount, moves to the dispense state. No third state cuts in. Written probabilistically, every transition is either $P = 1$ (always happens) or $P = 0$ (never happens). Nothing in between, which is exactly why it is debuggable.

An LLM system behaves like an Absorbing Markov Chain¹. A Markov chain is a walk where the next step depends only on where you are standing, not on the path that got you there. “Absorbing” means at least one square has no exit: step onto it and you stay forever. For a text generator, that exit-less square is the <EOS> token. Every generation step is a weighted draw across the entire vocabulary, and the process stops only when the draw lands on <EOS>.

The key difference: an FSM guarantees structurally that it will reach its destination. A Markov chain only guarantees that it tends to.

Push AI into a system without accounting for this and you get Degenerate Loops² — the model trapped in a closed cycle, never drawing <EOS>. In practice it looks like this: the model starts repeating “Sorry for the inconvenience. Sorry for the inconvenience.” until it hits the max_tokens ceiling. With a 4096-token ceiling, one failure of this shape burns its entire token budget and holds a serving slot for tens of seconds.

Seen this way, hallucination is not a content defect. It is a process with no stopping mechanism.

And no, you cannot patch the exit with conventional code. Add one more guard per edge case and the codebase swells past the point anyone will touch it — while the edge-case set of a probabilistic model never closes.

III. Familiar Ground for Anyone Who Has Run Distributed Systems #

Orchestrating an LLM is oddly close to operating microservices: both force decisions without ever having complete, certain state.

In distributed systems, the CAP theorem states that constraint. Stripped of notation: when the network between two halves of a system is cut, you get one of two things — everyone reads the same data (consistency) or the system still answers (availability). No configuration gives you both, because the other half is not talking.

AI systems face the same shape of constraint. When the model lacks the facts, it gets one of two things: refuse to answer (keep correctness, lose availability) or produce a fluent invention (keep availability, lose correctness). The default in every commercial LLM today is the second. The architect’s job is changing that default.

Two familiar distributed-systems techniques carry over directly:

• RAG as a data anchor. RAG (Retrieval-Augmented Generation) means looking up real documents first, then placing that text into the prompt so the model answers from it. The role is the same as reading from the system of record instead of a cache: the answer is pinned to something verifiable, and there is a citation path back to it.
• Byzantine Fault Tolerance for multi-agent systems. BFT is the classic problem of a group that must agree while some members answer nonsense or lie. In a multi-agent system, an agent inventing a figure is a Byzantine node. The handling is identical: ask three independent agents the same question, take the answer at least two agree on, and log the three-way disagreement as a signal for human review. The cost is real — three times the tokens and three times the latency per question — so reserve it for expensive decisions.

IV. Blocking Malformed Output with Constrained Decoding #

The first containment layer wraps a strict state machine around the probabilistic chain, trapping the uncertainty before it leaks out.

To see the mechanism, you need how generation works. At each step the model scores the entire vocabulary — roughly 100,000 to 200,000 tokens depending on the model — then draws one token according to those scores. Nothing in that procedure requires it to respect JSON syntax.

Constrained Decoding inserts itself between the two steps: after scoring, before drawing, it forces the score of every invalid token to zero.

What counts as valid comes from a state machine compiled ahead of time from your JSON Schema or regex, stored as a prefix (radix) tree so lookups cost almost nothing. Take a schema allowing only {"status": "ok"} or {"status": "error"}: the instant the model has produced {"status": ", the tree has two live branches left, ok and error. The other 200,000 tokens are pinned to probability zero. The model cannot emit malformed output, however much its weights want to.

What makes this worth the wiring: the guarantee does not depend on the model “understanding” the schema. It is a hard constraint at the decode layer, so the syntax-error rate goes to exactly 0% and all the retry-on-parse-failure code disappears. What it does not do: make the content true. {"status": "ok"} can still be a well-formed lie — which is the job of the second layer.

V. Committing to an Error Rate with Conformal Risk Control #

The next problem is a model far more confident than it has earned.

The cause is in the training. RLHF — tuning a model against human ratings — rewards answers that sound decisive, coherent, and pleasant. Raters rarely reward “I am not sure.” After a few million rounds of that, the model has learned that a confident register always pays, including when the content is wrong.

The measure of that gap is Expected Calibration Error (ECE). Reading it takes no math: sort every answer into 10 buckets by the confidence the model declared, then compare within each bucket. If the “90% confident” bucket is right 90% of the time, the model is well calibrated and ECE sits near 0. If that bucket is right 30% of the time, the model is lying about itself.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

import numpy as np

def calculate_sequence_ece(confidences: np.ndarray, accuracies: np.ndarray, M: int = 10) -> float:
    """
    Measure the gap between declared confidence and actual accuracy.
    Sort answers into M buckets by confidence, then take the weighted mean
    difference between each bucket's average confidence and average accuracy.
    """
    bin_boundaries = np.linspace(0, 1, M + 1)
    ece = 0.0

    for i in range(M):
        mask = (confidences > bin_boundaries[i]) & (confidences <= bin_boundaries[i+1])
        if np.any(mask):
            acc_bin = np.mean(accuracies[mask])
            conf_bin = np.mean(confidences[mask])
            ece += np.abs(conf_bin - acc_bin) * np.mean(mask)

    return ece

# A model pushed toward fluent, empty answers
sim_conf = np.array([0.99, 0.98, 0.99, 0.97]) # Declared: near certainty
sim_acc = np.array([1, 0, 0, 0])              # Reality: 1 correct out of 4

print(f"Sequence-Level ECE: {calculate_sequence_ece(sim_conf, sim_acc):.4f}")
# Output: 0.7325

Reading 0.7325 in words: the model declared an average certainty of 98.25% and was right 25% of the time. A 73-percentage-point gap. Every alert and every automatic threshold built on model “confidence” is standing on that number.

Conformal Risk Control reframes the problem. Instead of making the model less confident, it takes a calibration set with known answers, runs the model over it, and finds a threshold: below this score, the system declines to answer and routes to a human. The threshold is chosen so the error rate on new data stays under the level you set — say 5% — and that is a statistical guarantee you can demonstrate, not a feeling.

The price is explicit: every declined answer becomes human work. Moving the ceiling from 10% to 5% typically raises the hand-off rate substantially. This is a dial trading risk against operating cost, not a button labeled “make it correct.”

VI. A Circuit Breaker Inside the Model with Mechanistic Interpretability #

Measuring risk after the answer exists is measuring late. The third layer goes inward.

Inside a neural network, each processing step is a vector of a few thousand dimensions, and no single dimension corresponds to a readable concept — every concept is smeared across many dimensions at once. A Sparse Autoencoder is a secondary network trained to unpack that dense vector into tens of thousands of “switches,” under a constraint that nearly all switches must be off at any moment. That constraint forces each remaining lit switch to carry a specific meaning, and the meaning is read off by observing which inputs light it up.

With readable switches, a real-time circuit breaker becomes possible: when the switch cluster characteristic of fabrication lights up, the system cuts generation at that token — before the broken answer reaches a user. The same logic as the circuit breaker you already put in front of a flaky service, except it sits inside the model.

The maturity of this technique deserves a plain statement: it is new, expensive, and today mostly runs where you own the model weights.³ For teams on a closed API, the two layers above are what ships this quarter.

VII. Architectural Summary #

The Software 1.0 role of designing a static system is giving way to engineering that supervises a probabilistic flow. Three layers, in the order worth building them:

1. Constrained Decoding — block every malformed token with a JSON Schema or regex, at the decode layer.
2. Conformal Risk Control — trade subjective judgment for a committed error ceiling, with a refusal path below threshold.
3. Internal Circuit Breaker — trap risk signals inside the network with a Sparse Autoencoder, before broken output exists.

VIII. References #

• Karpathy, A. (2017). Software 2.0.
• Shannon, C. E. (1948). A Mathematical Theory of Communication.
• Outlines/SGLang (2025). Constrained Decoding Frameworks.
• Angelopoulos et al. (2024). Conformal Risk Control.
• Bricken et al. (2024). Towards Monosemanticity.