What Survives a Crash

Every admitted request has a durable identity

AdmissionOperationV1 binds a request to an operation, its participants, its current state, and any dispatch commitment. Versions and coordinator leases let recovery confirm that it is acting on the operation it actually read, not on a copy another coordinator has since advanced.

That identity is what makes the interrupted compute call recoverable. A retry resolves the existing operation instead of minting a new one. A request issued only because the caller never saw a response could repeat the work or spend a second allocation; the operation record is how the kernel avoids both. The captured compute example shows admission and charging on that path, without an injected crash.

if to == AdmissionOperationState::CompensatedBeforeDispatch {
    return from.is_pre_dispatch() && predispatch_state_enabled(requirements, from);
}
let ready_source = if requirements.approval {
    AdmissionOperationState::ApprovalReserved
} else if requirements.budget_capture {
    AdmissionOperationState::BudgetAuthorized
} else {
    AdmissionOperationState::Prepared
};
match to {
    // …
    AdmissionOperationState::ReadyToDispatch => {
        from == ready_source
            || (kind == AdmissionOperationKind::ToolDispatch
                && requirements.budget_capture
                && requirements.approval
                && from == AdmissionOperationState::BudgetAuthorized)
    }
    // …
    AdmissionOperationState::DispatchCommitted => {
        from == if requirements.budget_capture {
            AdmissionOperationState::CapturePending
        } else {
            AdmissionOperationState::ReadyToDispatch
        }
    }
    // …
}

The states before the dispatch line are the ones recovery may compensate: each enabled pre-dispatch state has a legal move to CompensatedBeforeDispatch. The states after it describe what the kernel knows about an effect that may have happened. Which states exist depends on the call’s requirements, so the transitions are a rule the kernel evaluates; a single fixed chain would misrepresent them.

pub enum AdmissionOperationState {
    Prepared,
    BrokerAttemptRegistered,
    ApprovalRequired,
    BudgetAuthorized,
    ApprovalReserved,
    ReadyToDispatch,
    CapturePending,
    DispatchCommitted,
    Finalizing,
    Completed,
    CompensatedBeforeDispatch,
    NotAcceptedAfterDispatchCommit,
    OutcomeUnknownAfterDispatch,
    /// The delivered output did not match a grant's committed output
    /// digest. A signed Deny is persisted; the open hold is released and
    /// zero is captured.
    DeniedAfterDelivery,
    MutationReady,
    MutationSubmitted,
    EconomicMutationApplied,
    EconomicMutationNotApplied,
}

pub(super) fn is_legal_transition(
    kind: AdmissionOperationKind,
    requirements: AdmissionParticipantRequirements,
    from: AdmissionOperationState,
    to: AdmissionOperationState,
) -> bool {
    if kind == AdmissionOperationKind::GovernedEconomicMutation {
        return matches!(
            (from, to),
            (
                AdmissionOperationState::Prepared,
                AdmissionOperationState::MutationReady
            ) | (
                AdmissionOperationState::MutationReady,
                AdmissionOperationState::MutationSubmitted
            ) | (
                AdmissionOperationState::Prepared
                    | AdmissionOperationState::MutationReady
                    | AdmissionOperationState::MutationSubmitted,
                AdmissionOperationState::EconomicMutationNotApplied
            ) | (
                AdmissionOperationState::MutationSubmitted,
                AdmissionOperationState::EconomicMutationApplied
            )
        );
    }
    if !kind.uses_dispatch() {
        return false;
    }
    if to == AdmissionOperationState::CompensatedBeforeDispatch {
        return from.is_pre_dispatch() && predispatch_state_enabled(requirements, from);
    }
    let ready_source = if requirements.approval {
        AdmissionOperationState::ApprovalReserved
    } else if requirements.budget_capture {
        AdmissionOperationState::BudgetAuthorized
    } else {
        AdmissionOperationState::Prepared
    };
    match to {
        AdmissionOperationState::BrokerAttemptRegistered => {
            requirements.broker_attempt && from == AdmissionOperationState::Prepared
        }
        AdmissionOperationState::BudgetAuthorized => {
            requirements.budget_capture
                && ((requirements.approval && from == AdmissionOperationState::ApprovalRequired)
                    || from
                        == if requirements.broker_attempt {
                            AdmissionOperationState::BrokerAttemptRegistered
                        } else {
                            AdmissionOperationState::Prepared
                        })
        }
        AdmissionOperationState::ApprovalRequired => {
            requirements.budget_capture
                && requirements.approval
                && from
                    == if requirements.broker_attempt {
                        AdmissionOperationState::BrokerAttemptRegistered
                    } else {
                        AdmissionOperationState::Prepared
                    }
        }
        AdmissionOperationState::ApprovalReserved => {
            requirements.approval
                && from
                    == if requirements.budget_capture {
                        AdmissionOperationState::BudgetAuthorized
                    } else {
                        AdmissionOperationState::Prepared
                    }
        }
        AdmissionOperationState::ReadyToDispatch => {
            from == ready_source
                || (kind == AdmissionOperationKind::ToolDispatch
                    && requirements.budget_capture
                    && requirements.approval
                    && from == AdmissionOperationState::BudgetAuthorized)
        }
        AdmissionOperationState::CapturePending => {
            requirements.budget_capture && from == AdmissionOperationState::ReadyToDispatch
        }
        AdmissionOperationState::DispatchCommitted => {
            from == if requirements.budget_capture {
                AdmissionOperationState::CapturePending
            } else {
                AdmissionOperationState::ReadyToDispatch
            }
        }
        AdmissionOperationState::Finalizing
        | AdmissionOperationState::NotAcceptedAfterDispatchCommit
        | AdmissionOperationState::OutcomeUnknownAfterDispatch => {
            from == AdmissionOperationState::DispatchCommitted
                || (to == AdmissionOperationState::OutcomeUnknownAfterDispatch
                    && from == AdmissionOperationState::Finalizing)
        }
        AdmissionOperationState::Completed => from == AdmissionOperationState::Finalizing,
        // A delivery-digest mismatch is decided during finalization, so
        // the only legal predecessor is Finalizing.
        AdmissionOperationState::DeniedAfterDelivery => from == AdmissionOperationState::Finalizing,
        _ => false,
    }
}

pub(super) fn predispatch_state_enabled(
    requirements: AdmissionParticipantRequirements,
    state: AdmissionOperationState,
) -> bool {
    match state {
        AdmissionOperationState::Prepared | AdmissionOperationState::ReadyToDispatch => true,
        AdmissionOperationState::BrokerAttemptRegistered => requirements.broker_attempt,
        AdmissionOperationState::ApprovalRequired => {
            requirements.budget_capture && requirements.approval
        }
        AdmissionOperationState::BudgetAuthorized | AdmissionOperationState::CapturePending => {
            requirements.budget_capture
        }
        AdmissionOperationState::ApprovalReserved => requirements.approval,
        _ => false,
    }
}

Dispatch decides what recovery may do

Two kinds of interruption reach the same records by different routes. When a future is dropped or a panic unwinds, Rust cleanup runs in the live process: the kernel’s drop guard acts at once. A hard process stop, an abort, or a power loss runs no cleanup at all; recovery then depends on what was persisted and on startup reconciliation.

Before dispatch, compensate each held resource

The live drop guard attempts each applicable cleanup independently. A failure in one step does not skip the others.

When every step succeeds, the durable operation can be terminalized as compensated_before_dispatch without writing a cancellation receipt. When a step fails, the guard attempts a signed fault receipt carrying pre_dispatch_cleanup_failed, the failing step, and the affected hold or reservation identifiers.

After dispatch, keep what was held and record the uncertainty

Once dispatch begins, releasing a single-use reservation could authorize the same effect a second time. The guard instead flushes buffered child receipts that were already signed, retains the runtime-admission reservations, and attempts to record the parent cancellation with metadata describing the ambiguous dispatch.

When the operation has a dispatch commitment but no durable tool outcome, terminalization records outcome_unknown_after_dispatch. It refuses to overwrite a durable outcome that already exists. The state gives an operator a concrete recovery target without inventing success or compensation; Unknown outcomes treats the same retained uncertainty as a security boundary.

const POST_ADMISSION_DROP_REASON: &str = "tool evaluation future dropped after admission";
const POST_DISPATCH_CREDENTIAL_COMMIT_FAILURE_REASON: &str =
    "dispatch credential commit failed after tool execution";
const POST_DISPATCH_URL_ELICITATION_REASON: &str =
    "tool server returned URL elicitation after dispatch; outcome is unknown";
const PRE_DISPATCH_CLEANUP_FAULT_REASON: &str =
    "tool evaluation future dropped before dispatch with cleanup fault";

After a hard stop, reconcile what was persisted

Startup reconciliation reads recoverable operations from the configured durable store. It attempts compensation for applicable pre-dispatch states, resolves dispatch commitments without a durable outcome as unknown, and completes persisted finalization work through its recovery path.

Handled compensation and terminalization failures are deferred, so the sweep can attempt the other operations before it returns a failure. Store access errors and malformed recovery data can stop the sweep earlier. A failed operation stays unresolved; it never becomes a successful recovery. Failure and recovery covers the runtime mechanisms.

One interrupted dispatch survives a restart

The restart case is part of the self-test that ships with every release. A paid tool runs once, then an injected failure prevents its outcome from being recorded. The suite closes the authority, reopens the same database with a fresh kernel, and runs startup reconciliation against the installed binaries, using real persisted records and restart logic. Run it on the host that will hold your durable store:

chio security selftest restart --evidence ./evidence

suite            result
restart          pass   every required case passed once; inventory complete
1 suite passed; evidence in ./evidence

A missing, renamed, or failed case fails the suite. The case this section follows is sqlite_restart_terminalizes_an_unrecorded_dispatch_without_moving_funds, and its log in ./evidence shows the same transition the SQLite integration suite checks.

Only the operation state changes across the restart. It records the unresolved outcome:

Reconciliation handles one recoverable operation, and a second sweep handles zero. The unknown-outcome record stays addressable by its operation identity and terminal replay key. Because the caller’s failure did not establish whether the tool ran, recovery preserves the uncertainty and makes no further payment transition.

Install the stores before admitting work

kernel.set_durable_admission_store(
    operations.clone(), outcomes, fence.clone(),
)?;
kernel.set_budget_store_handle(budget.clone());
kernel.set_payment_adapter(Box::new(payment_adapter));
// Register the same application tool and restore its required context.
let recovered = kernel.reconcile_durable_admission_startup()?;

Here operations, outcomes, budget, and the mutation fence come from the reopened SQLite authority. The complete test also provisions private directories, kernel configuration, the signed grant, and the payment adapter; this excerpt alone does not make a host recovery-capable.

Investigate by operation, not by request

let operation = operations.load_by_operation_id(&operation_id)?;
let journal = operations.load_payment_journal(
    operation_id.as_str(), &fence,
)?;

Resolve the operation, its tool outcome, its payment journal, and its retained evidence together. If the outcome is unknown, query the tool or payment system through the integration’s supported reconciliation path; a client timeout is not a reason to mint a fresh authorization. If cleanup failed, investigate the named hold or reservation. If finalization is pending, let the configured coordinator complete it.

When the outcome is already durable

The neighboring tests sqlite_restart_completes_a_committed_capture_without_request_replay and sqlite_restart_completes_a_committed_release_without_request_replay exercise persisted finalization after an interrupted completion path; run the whole file to cover them. A durable outcome is never overwritten by an invented cancellation or a second tool invocation.

Retained state is not serving readiness

A separate test killed the pricing service process after a successful write. Opening its SQLite database in a separate process recovered the committed rows, and the retained HTTP receipts still passed signature verification. The restarted HTTP listener did not answer within the readiness check. The captured observations establish retained data and evidence, not a recovered serving path.

After a hard stop, check both layers: inspect the persisted operation and records, then exercise a governed request before admitting new traffic. A bound port is not enough. Keep an unready instance out of service while its recovery and authority state are investigated. The in-flight outcome and payment results above come from the separate SQLite suite; they are not inferred from this completed write.

A consumed nonce must outlive a restart

Durable operations stop the kernel from repeating its own work. Execution nonces stop a caller from presenting the same dispatch authorization twice. A nonce binds one dispatch to the subject, capability, server, tool, request ID, and parameter hash. Its signature establishes that it is authentic; consuming it through the nonce store establishes whether it has already been used.

    /// Reserve a nonce while telling the store when the nonce stops
    /// being cryptographically valid. Durable implementations (SQLite,
    /// remote KV stores) MUST retain the consumed marker until at least
    /// `nonce_expires_at` so replay protection covers the nonce's full
    /// validity window.
    // …
    fn reserve_until(&self, nonce_id: &str, _nonce_expires_at: i64) -> Result<bool, KernelError> {
// …
pub fn consume_execution_nonce(
    nonce_store: &dyn ExecutionNonceStore,
    nonce_id: &str,
    nonce_expires_at: i64,
) -> Result<(), ExecutionNonceError> {
    match nonce_store.reserve_until(nonce_id, nonce_expires_at) {
        Ok(true) => Ok(()),
        Ok(false) => Err(ExecutionNonceError::Replayed),
        Err(error) => Err(ExecutionNonceError::Store(error.to_string())),
    }
}

The second half of that check is only as durable as the store behind it. A store that keeps the consumed marker in memory forgets it on restart, and a replay inside the nonce’s signed window is accepted again. A persistent store keeps the marker through the signed expiry, so the same replay is refused.

The feature requires an installed ExecutionNonceConfig. Setting require_nonce makes execution-bound dispatch require a fresh nonce; with the default of false, a presented nonce is checked but callers may omit one. One-time authorization does not establish exactly-once effects inside an external tool: the executor and its own persistence still decide that.

What the recovery record establishes

Destructor cleanup is best effort. A failed receipt append is logged as an audit_fault and may leave no cancellation receipt. Admission storage, outcome storage, receipt storage, and recovery must all be configured for the failures a deployment needs to survive.

Follow the operation through recovery

An ongoing application needs to retain its progress as well as the state of individual operations. The kernel’s admission records preserve operation state; a configured process host retains the actor’s identity, checkpoints, and named calls. To resume a worker’s planning state, tool keys, and messages after an operating-system restart, use Agent Processes and its runnable handoff.

Data restrictions survive the same restart. With information flow configured, the flow store retains the labels each read left behind for the authenticated principal, its delegation lineage, and the session. A reopened host evaluates the next call against them. The persistent taint example reopens a host after a restricted read and shows the restriction still in force.