Chio/Docs
LOGIN · JOIN

Fail-Closed Semantics

See how a guard that errors, times out, or loses its state turns into a signed refusal before dispatch.

Concept chio@5f63e6e

On this page

A guard that cannot decide refuses the call. In the kernel's guard loop only an Allow moves a call forward, so a deny, an error, a timeout that exhausts its retries, and an unsupported verdict all end in the same signed refusal, before anything is dispatched.

The running example is a session-velocity guard that reads a journal lock. When the lock is poisoned the guard returns an error, and the agent receives an ordinary denial whose receipt names the guard and the failure.

The guard loopChoose what the guard returns.
callforbidden-pathg₁session-velocityg₂egress-allowlistg₃dnot dispatchedErr(KernelError)guard "session-velocity" error(fail-closed): session-velocityjournal error: poisoned lockrKρreceipt · denyevidence: 1 guardg₁forbidden-pathg₂session-velocityg₃egress-allowlistErr(KernelError)guard "session-velocity" error(fail-closed): session-velocityjournal error: poisoned lockdnot dispatchedrKρdeny

An error is refused exactly like a deny

A guard that cannot decide returns Err, and the loop maps it to the same GuardDenied refusal. The message carries the tag (fail-closed) and the guard's own error text, so a receipt tells a broken guard from a refusing one. Failure classes.

One call through three guards, with the middle guard's result chosen; the guard names are illustrative. Output checks after dispatch, the advisory pipeline, and the emergency stop that refuses before any guard runs are left out.

On the Error path the middle guard's Err stops at the loop, the tool is never invoked, and the receipt's reason carries (fail-closed) followed by the guard's own error text.

Only Allow moves a call past a guard

Each guard returns a GuardDecision (a Verdict plus evidence) or an Err. The kernel's pre-dispatch loop turns any deny, any approval verdict, and any error into KernelError::GuardDenied:

crates/kernel/chio-kernel/src/kernel/dispatch.rs387-436rust
fn evaluate_guards_sequential(
    guards: &[Arc<dyn Guard>],
    ctx: &GuardContext,
) -> Result<Vec<chio_core::receipt::metadata::GuardEvidence>, GuardRunError> {
    let mut evidence = Vec::new();
    for guard in guards {
        match guard.evaluate(ctx) {
            Ok(decision) => {
                evidence.extend(decision.evidence);
                match decision.verdict {
                    Verdict::Allow => {
                        debug!(guard = guard.name(), "guard passed");
                    }
                    Verdict::Deny => {
                        return Err(GuardRunError::new(
                            KernelError::GuardDenied(format!(
                                "guard \"{}\" denied the request",
                                guard.name()
                            )),
                            evidence,
                        ));
                    }
                    Verdict::PendingApproval => {
                        // The `Guard` trait does not carry the HITL approval flow; that runs via
                        // `ApprovalGuard::evaluate`. A `Guard` returning `PendingApproval` is an
                        // unsupported state, so fail closed.
                        return Err(GuardRunError::new(
                            KernelError::GuardDenied(format!(
                                "guard \"{}\" returned an unsupported approval verdict",
                                guard.name()
                            )),
                            evidence,
                        ));
                    }
                }
            }
            Err(e) => {
                // Fail closed: guard errors are treated as denials.
                return Err(GuardRunError::new(
                    KernelError::GuardDenied(format!(
                        "guard \"{}\" error (fail-closed): {e}",
                        guard.name()
                    )),
                    evidence,
                ));
            }
        }
    }
    Ok(evidence)
}

An Err and an Ok(Deny) both stop the loop, and the receipt tells them apart by message: the error path adds (fail-closed) and the guard's error text. The synchronous guard catalog runs in this one mode, so a failure always resolves to a refusal. Approvals run on their own path, bound to the exact request, so the loop refuses a Guard that returns PendingApproval.

Read more in Pipelines & Composition.

Every failure class reaches the same refusal

Most failures surface from the guard as KernelError::Internal, which the loop maps to GuardDenied. Adapters for external and WASM guards refuse on their own terms first.

FailureRaised asVerdictAlso happens
Mutex poisonedKernelError::InternalDenyLater calls deny until the guard restarts or its state clears
Parse error on inputKernelError::InternalDenyError text can enter receipt evidence; keep raw input out of it
Session journal unavailableKernelError::InternalDenyTracing event names the guard
Any other internal errorKernelErrorDenyTagged (fail-closed); receipt persisted
WASM fuel exhaustedWasmtime abort, host errorDenyPer-call deadline metric increments; instance reaped
WASM trapModule trap, host errorDenyModule marked unhealthy, quarantined until a reload
Network timeoutretry_with_jitter exhaustedDenyFailure recorded on the circuit breaker
Permanent error (4xx, malformed request)ExternalGuardError::PermanentDenyNo retry
Circuit breaker openCircuitOpenVerdict::DenyDeny by defaultNo external call
Rate limiter emptyRateLimitedVerdict::DenyDeny by defaultNo external call, no breaker increment
Ambiguous inputAuthor conventionDenyReceipt records the guard's reason
Regex fails to compileConstruction errorNone: start-up failsThe guard never reaches the pipeline
Panic in evaluateAbort, or JoinErrorAbort, or DenyRelease builds abort the process
Receipt persistence unavailableDurable evidence missingRefuse, or retain an unknown outcomeDepends on the failure stage

An external guard retries through retry_with_jitter until RetryConfig.max_retries, and the final failure becomes Verdict::Deny. ExternalGuardError::Permanent always skips the retry loop, so a misconfigured provider credential denies at once instead of spending the retry budget. Cumulative failures can open the breaker.

CircuitOpenVerdict::Deny and RateLimitedVerdict::Deny are the adapter defaults. An operator can set either to Allow per adapter for a guard whose output feeds a review queue; keep both at Deny when the external guard is the last check on a capability. External Guard Adapters specifies both settings.

Receipt persistence fails according to its stage. Before dispatch, the kernel refuses an action that requires durable admission. After an effect may have occurred, it retains an unknown outcome and reconciles it, since a persistence error cannot undo the effect. See Unknown Outcomes.

Reason classes on the deny counter

WASM guard denials carry a bounded reason_class label on the chio_guard_deny_total{guard_id, reason_class} counter. chio-wasm-guards produces it and the kernel /metrics endpoint renders it. The label takes one of nine values (policy, pii, secret, prompt_injection, oversize, fuel, trap, malformed, other), assigned by substring-matching the guard's free-form deny reason, with any unrecognized reason folded into other.

Denials from native guards, circuit breakers, poisoned mutexes, session journals, and the receipt store are counted without a reason_class. Read their reason from the receipt.

Misconfiguration stops the kernel at load, before the first call

A configuration error fails start-up, so it never becomes a storm of denials at traffic time. BrowserAutomationGuard, MemoryGovernanceGuard, and ContentReviewGuard each return an InvalidPattern error when a configured pattern fails to compile, and the guard never builds.

Put regex compilation, JSON-schema validation, and configuration parsing in a guard's new() for the same effect. The kernel applies this rule to its own signing: a process-wide CryptoFloor from chio-policy is loaded once at start and threaded through the receipt signer, the capability validator, and the compliance-certificate issuer.

crates/guards/chio-policy/src/crypto_floor.rs45-54rust
pub enum CryptoFloor {
    /// Accept classical-only envelopes (no post-quantum key required). Default.
    AllowClassical,
    /// Accept either classical-only or hybrid classical-plus-ML-DSA-65
    /// envelopes. Requires a PQ key to be provisioned.
    AllowHybrid,
    /// Reject classical-only envelopes; require hybrid signing on every
    /// signed artifact. Requires a PQ key to be provisioned.
    PqRequired,
}

The variants are strictly ordered, AllowClassical < AllowHybrid < PqRequired, and serialize as allow_classical, allow_hybrid, and pq_required. CryptoFloor::validate_with_pq_key runs at policy load: selecting AllowHybrid or PqRequired without a provisioned ML-DSA-65 key returns CryptoFloorLoadError::HybridFloorRequiresPqKey, and the kernel refuses to start before its first signing call.

A sibling WeightsCardRequired enum (Disabled, Required, RequiredWithPin) applies the same load-time requirement to model provenance. Set above Disabled, it makes a signed weights or model-card binding mandatory on every provider bind and rejects a bind that arrives without one.

A panic aborts the process instead of returning a verdict

Guards run inline in evaluate_guards_sequential, and release builds compile with panic = "abort". A panic inside a guard ends the kernel process, so that call reaches no verdict. The workspace sets clippy::unwrap_used = "deny" and clippy::expect_used = "deny" because an unhandled Option or Result on a guard or receipt path is an availability fault. Handle every one.

With panic unwinding enabled, a guard offloaded onto tokio::task::spawn_blocking, under a configured guard budget or always_offload_guards, surfaces a caught unwind as a JoinError. The kernel maps it to KernelError::Internal("guard task join failed: ..."), and that request fails closed. The workspace's aborting release profile still terminates the process when the panic occurs in a blocking task.

Rust's panic documentation explains the abort boundary, and Tokio's JoinError contract describes caught task failures.

Advisory guards are non-blocking only while they succeed

The AdvisoryPipeline records signals and lets a successful request proceed. Two cases still deny it:

  • Promotion. When a PromotionRule matches a signal at or above its min_severity, the advisory pipeline returns Verdict::Deny for that call and marks the signal promoted = true.
  • Failure. When an AdvisoryGuard::evaluate returns Err, the advisory pipeline propagates it, and the wrapping GuardPipeline maps it to GuardDenied. Advisory evaluation is non-blocking on success and fails closed like every other guard.

Read more in Advisory Guards.

The emergency stop refuses before any guard runs, across restarts

ChioKernel::emergency_stop(reason) engages a stop that every evaluate_tool_call* path checks first. While it holds, each call receives a signed deny receipt carrying EMERGENCY_STOP_DENY_REASON ("kernel emergency stop active") before capability validation or the guard pipeline runs. emergency_resume() disengages it, and is_emergency_stopped() reports the current state.

The stop is durable. It appends a signed, hash-chained record to the admission store, so a kernel that restarts comes up stopped and still refuses: readiness reports ready_stopped while orchestrators and health probes see a live process. Operators reach it with chio emergency-stop, chio emergency-resume, and chio emergency-status from the CLI, the process-host socket, and the mounted HTTP route.

HushSpec carries a second, independent switch. activate_panic(), deactivate_panic(), and is_panic_active() in chio-policy set a global flag that each policy evaluate() call consults, beside the built-in chio:panic ruleset, one of the seven embedded rulesets a policy can extends. Both switches deny requests, at different layers, and each engages on its own.

Read more in Emergency Stop.

A poisoned journal lock becomes a signed deny

The session-velocity guard reads a journal before it compares a tool's call count with its limit. When the journal lock is poisoned, the guard returns an internal error, and the kernel's guard loop turns it into a refusal:

src/session_velocity.rsrust
impl Guard for SessionVelocityGuard {
    fn name(&self) -> &str {
        "session-velocity"
    }

    fn evaluate(&self, ctx: &GuardContext) -> Result<GuardDecision, KernelError> {
        let counts = self.journal
            .tool_counts()
            .map_err(|e| KernelError::Internal(
                format!("session-velocity journal error: {e}"),
            ))?;

        match counts.get(&ctx.request.tool_name) {
            Some(n) if *n >= self.limit => Ok(GuardDecision::deny(vec![])),
            _ => Ok(GuardDecision::allow()),
        }
    }
}
Kernel error
KernelError::GuardDenied(
    "guard \"session-velocity\" error (fail-closed): \
     session-velocity journal error: poisoned lock"
)

The kernel signs a receipt with verdict Deny and a reason field that contains that message. The agent receives a denial, and the audit log records which guard failed and why.

A spike of fail-closed denials points at a guard failure, such as a regex bug or journal corruption, or at a run of rejected agent requests. The receipt message identifies the guard and the reason, which tells the two apart.

Read more in Failure & Recovery, the operator runbook for fail-closed events.

Next: compose guards into a pipeline