Paper 01: The Illusion of Control — System Design in the Era of AI

I. When the Same Input Stops Giving the Same Output #

Traditional software promises same input, same output. A language model in a business flow breaks that promise: if-else code follows a fixed table, while the model writes text piece by piece by probability, like a Markov chain. Made-up answers and endless loops follow from how text is generated, not from bugs one more if can patch. Three containment layers restore control without a perfect model. They block malformed text during generation, commit to an error rate as a number, and cut generation on signals from inside the model.

Ask ChatGPT the same question twice and you may well get two different answers. To anyone who builds software, that means a decades-old promise has expired: same input, same output.

The promise comes from if-else code, which spells out every branch in advance: “if this, do that.” When the test suite goes red on a Monday morning, the first question is always “who changed what over the weekend?” Fair question: with code like that, if nobody changes anything, nothing changes.

An LLM, short for Large Language Model, is the kind of AI behind chatbots like ChatGPT: it takes text and continues it. Put one in a core feature and engineering shifts to orchestrating parts that are right only with some probability.

My running example is hypothetical: a food-delivery app in Ho Chi Minh City adds an LLM chatbot to answer questions about orders. Built with the old “total control” mindset, it can pass every test and still fail at scale on the first question nobody tried. The reason is how the two kinds of software differ.

II. If-Else Code Follows a Table; an LLM Follows Odds #

Karpathy (2017) calls hand-written code Software 1.0 and models learned from data Software 2.0.

Software 1.0 behaves like a Finite State Machine (FSM). An FSM is a machine with a finite set of states and a table: in this state, on this event, go there. A vending machine is one: waiting for money, it receives the full 10,000 dong and moves to dispensing a can. Each transition either always happens or never happens. Every path is in the table, so a fault can be traced through it.

An LLM system behaves like an Absorbing Markov Chain.¹ A token is the unit of text a model reads and writes, usually a word or part of one. At each step, the LLM looks at all the text so far, gives each token in its vocabulary a chance of being picked, and draws one. Vocabularies hold tens to hundreds of thousands of tokens, and generation stops when the draw hits the end token, written <EOS>.

A Markov chain is a walk where the next step depends only on where you stand, as in a dice board game, where the next square depends on the current one and the roll. “Absorbing” means at least one square has no exit, like the finish. For an LLM, the finish is <EOS> and “where you stand” is all the text so far, so the board is too big to list; there the analogy bends. Whether an FSM finishes can be read from its table; for an LLM, it is only a probability.

Ignore this and you get Degenerate Loops:² the model enters a cycle where drawing <EOS> is so unlikely that the length ceiling comes first. That ceiling, max_tokens, is the most tokens one answer may use. In the delivery app, the chatbot writes “Sorry for the inconvenience. Sorry for the inconvenience.” until it hits a 4,096-token ceiling.

Each such failure burns the answer’s whole token budget and ties up a server slot for tens of seconds. It is probably the longest apology the customer has ever received, and since providers bill per token, every word of it is paid for.

A hallucination is when a model writes something false that sounds plausible. It shares a root with the loop: generation picks tokens by likelihood, and no step checks truth or guarantees a stop.

An edge case is a rare situation nobody foresaw at design time. Patching with if-else adds a branch per edge case until nobody dares touch the code. With a probabilistic model, the list of edge cases never closes. Software has met this shape of problem before.

III. Distributed Systems Met This Problem First #

A distributed system runs on many networked machines, like a delivery app with one machine for orders and another for payments. Running one also means deciding without ever having complete, certain information.

The CAP theorem names that constraint: when the network between two groups of machines is cut, the system keeps only one of two properties. Either consistency: every place reads the same latest data. Or availability: every place asked still answers. No setting keeps both, because the machines on the other side cannot be reached.

An LLM faces a choice of the same shape, though as an analogy, not a theorem. Lacking the facts, the model either refuses, keeping correctness but losing availability, or writes a fluent sentence, keeping availability but losing correctness. Commercial LLMs usually lean toward the second; the system designer must change that default.

In a demo, “always has an answer” is a feature. In real life, it is the chatbot telling a customer “your order arrives in 5 minutes” when no driver has accepted it. Two distributed-systems techniques help change the default.

RAG as a data anchor. RAG, short for Retrieval-Augmented Generation, means fetching real documents first and putting them in the prompt so the model answers from them. A prompt is the text sent to the model. For the delivery chatbot, that means reading the order status from the database first.

What a model learned in training is an old snapshot, so RAG is like reading the system of record instead of a stale copy: answers rest on something verifiable and traceable. RAG reduces hallucination but does not remove it, since the model can still misread the documents.

BFT-style voting for multi-agent systems. An agent is a program that uses an LLM to do a chain of tasks by itself, like looking up an order and computing a refund. Byzantine Fault Tolerance (BFT) is the problem of machines agreeing on a decision while some may answer nonsense; an agent inventing a figure is exactly such a machine.

The handling is similar: ask three independent agents the same question, and take the answer at least two agree on. When all three disagree, the system logs the split for human review.³ The price is triple the token cost, plus a longer wait if the calls run in sequence. So it is only worth using for high-stakes decisions. The three containment layers that follow sit right around, and inside, the model.

IV. Layer 1: Block Malformed Output While It Is Generated #

Suppose the chatbot must send the refund system a piece of JSON after each conversation. JSON is a format programs use to exchange data through an API; the receiver reads only the exact shape it expects.

At each step, the model gives every token in the vocabulary a raw score, called a logit. Logits are then converted to a probability distribution: each token’s percentage chance, summing to exactly 100%. Nothing in this procedure forces valid JSON.

Constrained Decoding steps in between the two. It pushes the logit of every invalid token to negative infinity, bringing their chances to 0%, while the valid tokens’ shares are rescaled in proportion. Validity comes from an FSM with a precomputed table per state, so each step is only slightly slower.

That FSM is built from a JSON Schema or a regex. A JSON Schema describes which fields and values valid JSON must have. A regex is a text pattern, such as “exactly 10 digits”.

Suppose the schema allows only {"status": "ok"} or {"status": "error"}, and each word is one token. After {"status": ", if the model scores “fine” at 67%, “ok” at 24% and “error” at 9%, Constrained Decoding sets “fine” to 0% and rescales to “ok” 73%, “error” 27%. However high a wrong token scores, the output stays well formed.

As a hard constraint, it needs no “understanding” of the schema, so syntax errors drop to zero and most code that re-asks the model after unreadable JSON can go. One exception to remember is an answer cut off at max_tokens before it closes its brackets.

I like this layer because so little in AI work is guaranteed outright rather than “usually”. The catch: what is guaranteed outright here is the brackets, and {"status": "ok"} for a failed refund is still valid JSON. Whether it is true is the second layer’s job.

V. Layer 2: Commit to an Error Rate as a Number #

Confidence is a 0–100% number attached to an answer, derived from the written tokens’ probabilities or reported by the model when asked. It often runs far above real accuracy.

One cause is RLHF. RLHF tunes a model against human ratings, and raters tend to reward decisive, pleasant answers, rarely “I am not sure”. Weights are the billions of numbers inside a model that training adjusts; over many rounds of rating, they get pushed toward a confident tone, even when the content is wrong.

How well confidence matches accuracy is called calibration, and Expected Calibration Error (ECE) measures the gap. Sort answers into 10 buckets by confidence and compare within each. If the “90% confident” bucket is right 90% of the time, ECE is near 0. If it is right 30% of the time, the confidence number is misleading, like guessing a multiple-choice answer and shading the bubble extra dark to be sure.

Suppose the chatbot answers four questions at 97–99% confidence and gets one right. Average confidence is 98.25%, accuracy 25%, a 73-point gap: ECE is 0.7325. Every alert and automatic threshold built on model confidence stands on that gap.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

import numpy as np

def calculate_sequence_ece(confidences: np.ndarray, accuracies: np.ndarray, M: int = 10) -> float:
    """
    Measure the gap between declared confidence and actual accuracy.
    Sort answers into M buckets by confidence, then take the weighted mean
    difference between each bucket's average confidence and average accuracy.
    """
    bin_boundaries = np.linspace(0, 1, M + 1)
    ece = 0.0

    for i in range(M):
        mask = (confidences > bin_boundaries[i]) & (confidences <= bin_boundaries[i+1])
        if np.any(mask):
            acc_bin = np.mean(accuracies[mask])
            conf_bin = np.mean(confidences[mask])
            ece += np.abs(conf_bin - acc_bin) * np.mean(mask)

    return ece

# A model pushed toward fluent, empty answers
sim_conf = np.array([0.99, 0.98, 0.99, 0.97]) # Declared: near certainty
sim_acc = np.array([1, 0, 0, 0])              # Reality: 1 correct out of 4

print(f"Sequence-Level ECE: {calculate_sequence_ece(sim_conf, sim_acc):.4f}")
# Output: 0.7325

Conformal Risk Control reframes the problem. A calibration set is a batch of questions with known correct answers. Instead of making the model less confident, the method runs it over that set to find a confidence threshold. Answers below the threshold go to a human instead of the customer.

You set an error ceiling in advance, say 5%. The algorithm picks the threshold so that out of 100 new questions, at most 5 get a wrong answer. Hard questions handed off to a person do not count as errors. That is a mathematical guarantee rather than a gut feeling, provided new questions match the calibration set.⁴ A new menu or new kinds of questions means rerunning calibration.

The cost: every declined answer becomes human work, and moving the ceiling from 10% to 5% typically raises the hand-off rate noticeably. It is a dial trading risk for operating cost, not a “make it correct” button. In the delivery app, it sets how many support staff work the lunch shift.

VI. Layer 3: Cut Generation on Signals From Inside the Model #

Measuring risk after the answer is written is measuring late. The third layer builds on Mechanistic Interpretability. This research field opens a model up to see how it computes.

Inside an LLM is a neural network: many layers of calculations in sequence, using learned weights. At each step, the internal state is a list of thousands of numbers. Almost no single number maps to a readable concept, because each concept is spread across many numbers.

A Sparse Autoencoder is a secondary network that splits that list into tens of thousands of “switches”, with nearly all switches forced off at any moment. As a result, most switches carry a meaning specific enough for people to name, by watching which text lights each one up (Bricken et al., 2023).

A circuit breaker is a familiar distributed-systems mechanism: when one part keeps failing, others stop calling it for a while, like a household breaker tripping. If a switch cluster tied to hallucination can be found, a circuit breaker can sit inside the model. Whenever that cluster lights up, the system cuts generation at that token and hands off to a person, before the broken answer reaches the customer.

The technique is new, costly, and today runs mostly where you have full access to the model weights.⁵ For teams calling a provider’s API, the two layers above are what can ship now. Ranked by how good it sounds in a meeting, this layer comes first. Ranked by its odds of serving real customers this quarter, it comes last.

VII. The Three Layers, in the Order to Build Them #

The table lists the three layers in the order worth building them. That order also happens to run from cheap to expensive, which is rare in architecture.

Questions for your engineering team: which of these layers does our AI feature have, what is its error ceiling, and when was calibration last rerun?

VIII. References #

• Karpathy, A. (2017). Software 2.0.
• Shannon, C. E. (1948). A Mathematical Theory of Communication.
• Lamport, L., Shostak, R., & Pease, M. (1982). The Byzantine Generals Problem. ACM Transactions on Programming Languages and Systems, 4(3), 382–401.
• Outlines/SGLang (2025). Constrained Decoding Frameworks.
• Willard, B. T., & Louf, R. (2023). Efficient Guided Generation for Large Language Models.
• Angelopoulos et al. (2024). Conformal Risk Control.
• Bricken et al. (2023). Towards Monosemanticity.