Paper 01: The Illusion of Control: System Design in the Era of AI

I. The Limits of Traditional Programming #

For decades, software engineering ran on a simple promise: write the logic, get the result. Same input, same output, every time.

Then we wired Large Language Models (LLMs) into core features — and the promise broke. The job stopped being about managing if-else statements and started being about orchestrating probability distributions. Keep the old control-first mindset, and the cracks turn into cascading failures the moment the system meets unfamiliar data.

II. Fundamental Differences: FSMs vs. Absorbing Markov Chains #

Underneath the surface, the shift is mathematical:

• Classical Software 1.0 operates as a Finite State Machine. The transition from state A to state B is always absolute ( $P=1$ or $P=0$).
• Modern LLM Systems function as Absorbing Markov Chains¹. The text generation process only halts when it converges on a single destination: the <EOS> (End of Sequence) token.

Push AI into a system without that absorbing state in mind, and you get Degenerate Loops² — the model stuck in a closed cycle, never reaching <EOS>. Hallucination, in this framing, isn’t a content error. It’s a system that can’t terminate.

And no, you can’t patch your way out with traditional code. Try to catch every edge case with another loop and the codebase swells until it’s unmaintainable.

III. Structural Parallels with the CAP Theorem #

Orchestrating LLMs feels strangely familiar to anyone who’s run Microservices: both ask you to make decisions without ever having 100% accurate state data.

In distributed computing, the CAP Theorem defines the physical constraints of databases. With AI, demanding an algorithm to always generate 100% accurate output is impossible. Instead, accuracy relies on probability:

• RAG and Conformal Prediction: Just as a database locks tables to ensure consistency, AI utilizes data retrieved via RAG to anchor itself to factual information.
• Byzantine Fault Tolerance: In a Multi-agent network, AI agents producing hallucinations can be viewed as malicious entities known as Byzantine Nodes. Cross-consensus mechanisms allow the system to determine the most reliable answer based on the total number of perfectly accurate votes.

IV. Controlling Output with Constrained Decoding #

The strongest architectural solution available today wraps a strict state machine around the probability chain — caging the uncertainty before it can leak out.

This mechanism is known as Constrained Decoding. Simply put, it compiles strict data rules like JSON or Regex into Radix Trees. During every loop where the AI generates the next word, this mechanism explicitly forces the probability of any rule-violating vocabulary to exactly $0\%$.

Pruning invalid vocabulary early prevents the system from wasting resources evaluating incorrect sequences, ensuring the output meets strict JSON format requirements without negatively impacting performance.

V. Tackling Overconfidence with Conformal Risk Control #

Another major challenge is the Calibration Paradox. Training AI via RLHF to sound fluent can inadvertently encourage the model to confidently output incorrect information.

Instead of relying on general metrics like Expected Calibration Error, engineers are adopting the Conformal Risk Control standard. This framework enforces a mathematically proven boundary on acceptable risk percentages. For example, ensuring that a RAG feature’s semantic error rate never exceeds 5%.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import numpy as np

def calculate_sequence_ece(confidences: np.ndarray, accuracies: np.ndarray, M: int = 10) -> float:
    """
    Calculates ECE for sequence-level calibration, measuring the
    alignment between predicted confidence and empirical accuracy.
    """
    bin_boundaries = np.linspace(0, 1, M + 1)
    ece = 0.0

    for i in range(M):
        mask = (confidences > bin_boundaries[i]) & (confidences <= bin_boundaries[i+1])
        if np.any(mask):
            acc_bin = np.mean(accuracies[mask])
            conf_bin = np.mean(confidences[mask])
            ece += np.abs(conf_bin - acc_bin) * np.mean(mask)

    return ece

# Simulation of an AI trained to sound fluent but lacking substance
sim_conf = np.array([0.99, 0.98, 0.99, 0.97]) # Overconfident
sim_acc = np.array([1, 0, 0, 0])              # Reality: Mostly incorrect

print(f"Sequence-Level ECE: {calculate_sequence_ece(sim_conf, sim_acc):.4f}")
# Output ECE error: 0.7325

VI. Internal Network Risk Monitoring with Mechanistic Interpretability #

Measuring risk after the fact isn’t enough. The next move is proactive: Mechanistic Interpretability. Drop Sparse Autoencoders into the network as internal filtering grids, and you can dissect its “thought process” into independently readable feature groups — opening the black box from the inside.

This structure acts as an internal Circuit Breaker running in real-time. The API tier will proactively kill a text generation request the moment it detects an internal activation pattern associated with hallucination data, decisively stopping the flawed text generation long before that incorrect vocabulary is ever produced.

VII. Architectural Summary #

The role of the static system designer from Software 1.0 is rapidly giving way to the orchestration engineer of uncertainty:

1. Constrained Decoding: Eradicate invalid outputs instantly using JSON Schema Regex boundaries.
2. Conformal Risk Control: Stop relying on blind vibe-checks. Adopt this framework for mathematically proven accuracy guarantees.
3. Internal Circuit Breakers: Catch systemic risks early from within the core neurons using Sparse Autoencoders before the end-user ever notices.

VIII. References #

• Karpathy, A. (2017). Software 2.0.
• Shannon, C. E. (1948). A Mathematical Theory of Communication.
• Outlines/SGLang (2025). Constrained Decoding Frameworks.
• Angelopoulos et al. (2024). Conformal Risk Control.
• Bricken et al. (2024). Towards Monosemanticity.