Emergency Stop

One authenticated request stops a kernel scope

One authenticated operator is enough to stop. The request names a reason and a scope: the kernel, one tenant, or one recovery scope.

curl -sS -X POST "$CHIO_URL/emergency-stop" \
  -H "X-Admin-Token: $CHIO_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"reason": "fixer read PR #421 and touched /secrets"}'
chio emergency-status

result               stop_durable
scope                kernel
epoch                1

readiness            ready_stopped
stopped              true
reason               fixer read PR #421 and touched /secrets
host_latch           none
withheld_operations  0

You should see stop_durable (the output is illustrative): the record is committed and anchored. Each transition is a signed chio.stop-epoch.v1 record in a hash-chained table, where every record names the digest of the one before it. The record keeps a salted SHA-256 commitment to the reason; the reason text lives in an operator-readable note table.

The routes POST /emergency-stop, POST /emergency-resume, and GET /emergency-status are mounted on the Trust Control Plane, the API sidecar, and the hosted MCP admin service, and each compares its X-Admin-Token in constant time. The process-host socket lives in the host’s runtime directory and checks the peer’s owning uid plus the control credential. chio emergency-stop, chio emergency-resume, and chio emergency-status drive either one, and with the host down the CLI takes the serving-owner lock and appends the stop itself.

pub struct EmergencyAdmin {
    kernel: Arc<ChioKernel>,
    expected_admin_token: String,
}

The kernel refuses at entry and again inside the crossing

Every kernel attached to one admission store reads the same stop heads, so a stop reaches every hosted session over that store, including the sidecar’s per-request HTTP authority. The first check is that early read at each entry site, and a process host makes it before committing a call slot, so a stopped call spends no call budget.

The second check decides. It runs inside the writer transaction that would let an effect cross custody, and each kind of crossing has a recorded disposition:

A settling crossing passes only for effects ordered before the stop record’s commit, so a new rail authorization is refused while a capture of an existing hold completes. The checks follow the fail-closed convention, and the overload shed in Backpressure runs right after the early stop check on the same fast path.

Work already in motion stays in the incident

A stop never recalls an effect that crossed before it and never dispatches one again. When a tool admitted before the stop returns during it, the kernel withholds the output and keeps the bytes in durable custody. The caller receives OutputWithheld with retry: AfterResume: an output_withheld error on JSON-RPC, status withheld from the process ABI. After resume, a replay with the same request ID returns the released output.

A non-durable side-effecting call has no custody to resume from. Its result carries retry: Never and effect_executed: true; treat that call as executed with its output lost. A parked operation waiting for approval stays parked. An approval that arrives during the stop is refused at the edge without being consumed, and you resubmit it after resume.

Governed active response, whose every effect narrows authority, runs during a stop when the stop allows containment. For a forensic freeze, stop with containment refused; a restrict transition tightens an existing stop in place. Reconcile the original operations under their existing identities before you decide what may continue.

Read more in Unknown outcomes.

A stopped kernel restarts stopped and still refuses

On boot the kernel loads and verifies the stop heads before the reconciliation sweep runs. The sweep retains operations whose output the stop withholds, and the host reports ready_stopped. Health probes see a live, serving process, and every refusal and withholding is in force from the first request.

1. A failed or unknown commit. A stop-intent latch is fsynced before the transaction. The route returns stop_not_durable or stop_outcome_unknown, and the next boot still comes up stopped.
2. A broken chain. A digest mismatch or epoch gap leaves the kernel not_ready with stop_chain_unverified. The kernel scope is latched stopped, and the stop and status routes still answer.
3. A host that is down. The CLI appends the stop under the serving-owner lock, and the next boot loads it.

A kernel with no durable admission store, as in ephemeral development profiles, keeps its stop in process and reports durability: process_local. Work profiles and process hosts run on durable admission and get the durable stop.

Read more in Protect durable state.

A host latch lasts for one kernel process

Trusted host code can stop its own kernels without an operator. When the native broker’s authority RPC service fails, it calls kernel.emergency_stop("broker authority service failed") and stops serving. The latch applies to every kernel in that process and to every check inside a crossing, and it forbids containment because the authority service itself has failed.

A host latch is held in memory, and a restart clears it, along with the reason and the Unix time in seconds the kernel recorded. A fresh process either re-establishes the failed component or fails readiness. Host code that set the latch clears it in process with emergency_resume(), so no operator resume is involved. The status route reports it as host_latch, and its deny receipts name the latch in place of an epoch. is_emergency_stopped, emergency_stop_reason, and emergency_stopped_since read it.

use chio_kernel::{
    ChioKernel, KernelError, SecurityInvocationContext, ToolCallRequest, ToolCallResponse,
};

/// This embedding already owns the authenticated request and durable state.
/// The stop is process-local; this function deliberately does not resume it.
pub async fn stop_and_observe(
    kernel: &ChioKernel,
    request: &ToolCallRequest,
    trusted_context: &SecurityInvocationContext,
) -> Result<ToolCallResponse, KernelError> {
    kernel.emergency_stop("fixer read PR #421 and touched /secrets")?;
    assert!(kernel.is_emergency_stopped());
    assert!(kernel.emergency_stopped_since().is_some());
    kernel
        .evaluate_tool_call_with_security_context(request, trusted_context)
        .await
}

The prepared example resumes the latch and keeps a revoked grant refused

The containment exercise builds the Execution application’s persistent kernel with an authenticated process context and its repository connector. It keeps one kernel alive through a useful read, a latched stop with its incident reason, a refused read, and a resume; it checks that the latch is engaged before the probe. It then latches again, revokes the grant, resumes once more, and finally leaves the latch engaged for operator review before it exits.

The read helper forms each request through the host’s ProcessRuntime, invokes under the known-outcome-only contract, and retains the original receipt. Use the prepared Execution example commands from the example directory, with fresh disposable state:

containment demo .containment repository > containment.json
jq '{stop, calls: [.calls[] | {phase, decision}],
     resource_reads, capability_revoked, emergency_stopped}' containment.json
containment inspect .containment repository

{
  "calls": [
    {
      "phase": "before-stop",
      "verdict": "allow"
    },
    {
      "phase": "during-stop",
      "verdict": "deny"
    },
    {
      "phase": "after-resume",
      "verdict": "allow"
    },
    {
      "phase": "after-withdrawal",
      "verdict": "deny"
    }
  ],
  "resource_reads": 2,
  "capability_revoked": true,
  "emergency_stopped": true
}
{
  "capability_revoked": true,
  "emergency_stopped": false,
  "resource_reads": 2
}

You should see two allowed reads and two refusals. The latched call adds no read, and neither does the call under the revoked grant, so the connector journal .containment/resource-reads.jsonl holds exactly two. Reopened in a new process, the latch is clear, the revocation is still recorded, and both reads remain. The exercise owns its own disposable kernel and resumes automatically to show the API; an incident runbook resumes only after operator review.

A stopped kernel signs each refusal with the epoch it observed

A stop refusal is temporary. It writes no deny tombstone, and whether the same request ID proceeds after resume depends on where the call stopped:

Every stop deny receipt carries chio_runtime.stop with the scope, decided_by (the early check or the crossing check), retryable_after_resume, and observed: the epoch that refused, the host latch, or a pending intent. An auditor joins each refused request to the transition behind it. A prepared operation refused at dispatch commit is compensated as PreDispatchNoEffect, so its request ID now names that terminal result.

Read more in Replay protection, which covers the replayed-result recheck at caller output release.

Resume needs separate authority and restores nothing that was withdrawn

chio emergency-resume carries the epoch it was decided against, so a delayed resume never clears a newer stop. It needs a principal outside every stopper of the incident, an offline-verified chio.stop-control-quorum.v1 artifact, or a single-operator cooldown of at least 300 seconds set in signed deployment configuration.

The resume record is signed inside its own transaction, within a two-second timeout; if signing fails, the scope stays stopped and the route returns ResumeRefused with EvidenceUnavailable. When authority time is unavailable, a quorum carrying a signed time attestation resumes by break-glass. A tenant resume never clears a kernel stop.

Resuming releases withheld output and runs the full validation pipeline again. A stop revokes nothing, so a revoked grant stays revoked and a capability that expired during the stop stays expired. Pair the stop with revocation when authority has to end for good.

Next: end the authority behind the incident