The Risk Ruler: guardrails that continue to work even when the model fails

This article is about how to draw the layer that decides when an agent can act alone and when it needs to stop and wait for a human, without turning the entire system into an approval queue that no one can maintain.

Why model self-confidence is not a reliable guardrail

The most common mistake in this design is using a single confidence threshold as a scaling criterion: if the model's confidence falls below a number, it scales to a human, otherwise it continues alone. The problem is that the confidence declared by the model itself should not be interpreted as a calibrated probability of success. Recent research in mechanistic interpretability shows that language models encode real calibration and verbalized confidence in practically orthogonal directions within the network [1], which means that the correct information about uncertainty exists internally, but the text that the model generates to express this confidence often does not reflect it [2]. In practice, this appears as an incorrect response presented with language as confident as a correct response.

The most robust defense against this is not to adjust the threshold, but not to rely on a single signal. A second model, smaller and dedicated to checking the work of the first, helps, but it is not guaranteed independent validation: two models can share the same error, especially when they come from similar training data or reasoning patterns. The design that works combines signals of different nature, not just another model evaluating the first: deterministic rule, schema validation, permission verification, evidence retrieved from a reliable source, consistency test between executions, the evaluating model as one of these signals, and human review reserved for the truly critical case.

How action decides the level of control

Not every action deserves the same level of control. A read query does not have the same risk as an irreversible write, canceling an order, approving a payment, publishing content, or changing a production configuration. Mapping all actions that the agent can execute, and classifying each one before deciding which control standard to apply, is the first step that most projects skip, going straight to "let's put a human reviewing everything" or to the opposite, "the agent already makes few mistakes, let it run alone".

Four patterns cover most real cases. Auto-run for low-risk and reversible actions, without any intervention. Audit for medium-risk actions, the agent executes and the action is recorded for later review, not blocking. Conditional validation for high-risk but reversible and controllable actions, where release depends on a composition of factors, reversibility, impact, quality of retrieved evidence, result of deterministic validations, and model uncertainty, not just an isolated confidence number. Gate for critical or irreversible actions, mandatory human approval before any execution, without exception.

The structured checkpoint

When an action requires approval, the format of the approval request matters as much as the decision to request it. The standard that works in production follows a fixed sequence: the agent prepares the action with a complete audit trail, a human reviews the objective justification and the evidence that supports it, the human authorizes or denies, and the agent executes with the attribution of that authorization recorded along with the result. The human does not need, and should not need, to read an internal chain of reasoning from the model to decide. They need auditable evidence: input data, triggered rule, tool called, validation result, and origin of the information used.

This is quite different from a generic approval request of the type "can I continue?". Structured approval exposes the intention of the action, the origin of the data used to decide, the permission chain involved, the expected impact radius, and the reversal plan if something goes wrong. An approver who only sees "confirm action?" tends to approve out of habit after the tenth time. An approver who sees the complete context has what they need to actually judge.

Authorization is not the end of the story. Even after approval, an action should run under minimal privilege: explicit allowlist of permitted tools and operations, scope limited by the identity that authorized, value and volume limit per execution, idempotence so that re-execution does not duplicate the effect, sandbox when the domain allows, and temporary or scoped credential instead of a permanent and broad access key. Separating the moment of preparing the action from the moment of executing it is what makes this layer possible, because only after separating it does a clear point exist where all these limits can be applied before the side effect occurs.

Loop detection and circuit breaker are two different guardrails

An agent that tries the same step repeatedly, without converging, almost always means that the plan is wrong, not that it just needs one more attempt. This is not just a cost problem, it is a guardrail problem: an agent stuck in a loop can start trying increasingly aggressive variations of the same action until one of them "works", which is exactly the behavior you do not want near a risky action.

It is worth separating two mechanisms that often appear mixed. The loop detector, or step budget, controls the agent's own behavior: repetition of the same tool, repetition of the same state, lack of measurable progress, maximum number of steps, abnormal plan growth. The circuit breaker, on the other hand, protects the call between components: consecutive failures above a limit, timeout of a dependent service, high error rate, recovery period before new attempts. One acts at the level of orchestration and agent execution, the other at the level of infrastructure that the agent depends on. Both are guardrails, but they solve failures of different nature, and a mature system needs both, not one covering the role of the other.

Where most go wrong: excess approval gate

The opposite error to a weak guardrail is a guardrail that locks everything. If every action goes through human approval, the agent does not save any effort, it becomes an assistant that requires more supervision than the manual work it would replace. An approval queue that takes hours to respond is not responsive, and becomes the reason why no one trusts the system for anything urgent.

The practice that avoids both extremes is to start with conservative gates, review the queue weekly looking for approver fatigue and route deviation, and only loosen a specific gate after accumulating a relevant volume of clean executions in that action. Expanding autonomy is a decision based on accumulated evidence, not on general confidence that "the agent is already good".

A real example of decision

"

Real case

In a payment gateway that I architected, the AI Risk Agent never had the authority to block or approve a transaction alone, it always returned a recommendation for the deterministic engine to evaluate. This was a guardrail in itself: the boundary between analysis and decision was designed in the response contract, not in a prompt instruction.

The resilience layer included Shadow Mode, executing the AI's recommendation in parallel to the real flow, without influencing the result, until accumulating operational evidence about agreement, stability, and error rate in relation to the deterministic engine. Only after this shadow period did any signal from the AI actually weigh in the decision flow, and even then as input, not as a final decision.

A circuit breaker completed this layer, interrupting calls to a component that started to fail repeatedly, instead of letting the failure propagate through the rest of the system. Guardrail, in this design, was not a feature added later, it was part of the contract between components from the beginning.

The real cost of the choice

Handoff latency. Every time an action scales to human approval, the response time stops being measured in seconds and starts being measured in minutes or hours, depending on the availability of the approver. This needs to be in the product design, not a surprise discovered after the first customer complains about delay.

Approver fatigue. A human who approves the same category of action repeatedly, without almost ever denying, tends to stop reading the context and start approving out of habit. This is not a character flaw, it is a well-documented automation bias in human factors research and also discussed in the legal debate on human supervision in AI systems [3][4][5], the expected result of putting a human in the way of things that rarely go wrong.

False sense of security. A single confidence threshold, without mult signal validation behind it, gives the feeling of being protected without delivering actual protection, because the signal itself that the threshold is based on is the signal that most fails exactly in the cases that matter most.

Queue without SLA. An approval queue without a defined response time is not a guardrail, it is a bottleneck waiting to happen. If the response expectation is not dimensioned according to the real availability of approvers, the queue grows until someone decides to ignore it.

The question that almost no one asks

Before designing this layer, I answer five questions about the system I am building.

Is every action that the agent can execute mapped and classified by risk level, or only the obvious ones received this care?

Does the scaling criterion depend on a single signal of confidence from the model itself, or is there mult signal validation behind it?

Is there a technical limit of repeated attempts in the same step before the system stops and scales, instead of letting the agent insist indefinitely?

Does the approval request arrive at the human with sufficient context to judge, or is it a generic "confirm?" that trains the approver to click without reading?

Is there a defined SLA for how long an action can wait in the queue before becoming a problem in itself?

Each weak response here is a guardrail that exists on paper, but not in production.

How this evolves in practice

Guardrail in prompt
"Be cautious", without real enforcement
    │
    ▼
Single confidence threshold
Better than nothing, but poorly calibrated
    │
    ▼
Risk classification by action
Auto-run, audit, conditional validation, gate
    │
    ▼
Mult signal validation
Rules, schemas, permissions,
evidence and evaluating model
    │
    ▼
Mature production
Loop detector and step budget
Circuit breaker for dependencies
Approval queue with SLA
Explicit boundary between recommendation and decision

The riskiest jump in this ladder is skipping from guardrail in prompt directly to mature production without going through risk classification by action. It is this jump that produces both the system that locks in approval and the system that lets an irreversible action pass without any review.

My recommendation as an architect

Guardrail is not a layer that you add after the agent is already working. It is part of the contract between the agent and the rest of the system from the first action it has permission to execute alone. Start by mapping all possible actions and classifying the risk of each one, before writing any scaling logic.

The simplest criterion I use to know if a guardrail is real: it continues to work even when the model, by mistake or by malicious design of an external prompt, decides to ignore its own caution instruction. If the answer is no, what you have is a well-written hope, not an architecture.

A too-fast agent, approving everything alone, and a too-slow agent, waiting for approval for everything, fail for the same reason: no one designed the risk ruler between the two extremes.

Technical references

[1] "Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMs". arXiv, 2026. https://arxiv.org/pdf/2604.01457

[2] "Large Language Models Are Overconfident in Their Own Responses". arXiv, 2026. https://arxiv.org/pdf/2606.03437

[3] Center for Security and Emerging Technology, Georgetown University. "AI Safety and Automation Bias", issue brief. November 2024. https://cset.georgetown.edu/wp-content/uploads/CSET-AI-Safety-and-Automation-Bias.pdf

[4] "Check the box! How to deal with automation bias in AI