Failure & Recovery
Bring a degraded guard fleet back: roll back a faulty module, block a withdrawn build, and restore traffic after a provider outage.
On this page
The end state is a guard fleet on known-good modules, with withdrawn builds refused by digest and every external check answering again. While a blocking guard is degraded, the calls it covers are refused before dispatch, so recovery restores service without reopening a gap.
Each procedure runs on the node that hosts the guard. The reload engine, the digest blocklist, and the incident record are per-node files and state, and an external provider's circuit breaker lives in the adapter inside the kernel process.
StartThe watchdog restores the module it holds
reload_with_watchdog keeps epoch 0 alive. 5 error-class verdicts inside 60 seconds trip it: it restores epoch 0 without compiling, counts rolled_back, then writes the incident directory. Call C acquires epoch 0 again. Guard Supply Chain.
1. Identify the control that failed
Start from what you observe: a rejected or quarantined module, an open circuit breaker, an unavailable provider, or a configuration error. A reload that rolled back leaves three signals, and each one names the guard:
metric chio_guard_reload_total{guard_id="refund-ceiling", outcome="rolled_back"} 1
event chio.guard.reload.rolled_back
files $XDG_STATE_HOME/chio/incidents/2026-10-11T09:14:03Z-refund-ceiling-7/
incident.json
last_5_eval_traces.ndjsonThe reload counter chio_guard_reload_total labels each outcome as applied, canary_failed, or rolled_back. Failures that trace to an external provider (timeouts, permanent errors, an open breaker, an empty token bucket) come from the adapter on External Guard Adapters, which specifies its breaker, retry, and cache once. Fail-Closed Semantics holds the matrix of each failure and the verdict it produces; classify against it, then take the step below.
2. Replace or roll back a faulty WASM module
A WASM guard that traps or reaches its fuel ceiling repeatedly is quarantined: the bundle store marks it unhealthy, and pipeline evaluations short-circuit to Verdict::Deny. Recover with a corrected bundle pushed through the hot-reload path, which validates the replacement before an atomic swap.
- Canary. The canary replays a frozen corpus of exactly
CANARY_FIXTURE_COUNT(32) fixtures against the new module. Verdicts are compared as serialized bytes. A corpus of any other size is rejected, a failing fixture leaves the prior epoch untouched, and on a pass the canary backend is dropped and a fresh one built from the same bytes is published. - Publish. The swap is one atomic epoch change. In-flight evaluations keep the module snapshot they started with, and new calls use the new epoch.
- Watch. A rollback watchdog attaches to each accepted reload. It trips after 5 consecutive error-class verdicts inside a 60-second window: traps, fuel exhaustion, serialization failures, and other fail-closed backend errors.
On a trip the watchdog restores the prior module, emits chio.guard.reload.rolled_back, and writes ${XDG_STATE_HOME}/chio/incidents/<utc-iso8601>-<guard_id>-<reload_seq>/ holding incident.json and last_5_eval_traces.ndjson. The traces are redacted summaries; request payloads are never persisted. You should see the guard back on the epoch it served before the reload, and the incident directory from step 1.
Read more in Guard Supply Chain.
3. Block a withdrawn build by digest
The local blocklist at ${XDG_STATE_HOME}/chio/guards/blocklist.json (or ~/.local/state when that variable is unset or empty) pins a known-bad build out of rotation. Engine::reload refuses a replacement whose sha256:<digest> is listed and returns E_GUARD_DIGEST_BLOCKLISTED; chio guard pull runs the same check against the pinned OCI manifest digest before it contacts a registry or writes cache.
The digest is the one sha256sum reports for the module. These runs stage it as quarantined-guard.wasm, with XDG_STATE_HOME pointing at a state directory inside the project:
$ sha256sum quarantined-guard.wasmd83c00ceb39d975f51f62eef13b17bbb759c1a7f19c8c29100fc62a493f5e83d quarantined-guard.wasm
The file is one JSON object with a digests array. Write the entry into it directly, which also works on a node with no CLI installed:
$ mkdir -p "$XDG_STATE_HOME/chio/guards"
$ printf '{"digests":["sha256:%s"]}\n' \
"$(sha256sum quarantined-guard.wasm | cut -d' ' -f1)" \
> "$XDG_STATE_HOME/chio/guards/blocklist.json"
$ cat "$XDG_STATE_HOME/chio/guards/blocklist.json"{"digests":["sha256:d83c00ceb39d975f51f62eef13b17bbb759c1a7f19c8c29100fc62a493f5e83d"]}You should see the digest inside digests. From then on, a reload of that module stops before any backend is built. Blocking the same release at pull time takes a second entry, its OCI manifest digest, because the two checks hash different bytes.
Clear an entry, and what each outcome prints
chio guard blocklist remove reports which of two things happened and prints the file it operated on, so a runbook logs the resolved path. Removing a listed digest:
$ chio guard blocklist remove "sha256:$(sha256sum quarantined-guard.wasm | cut -d' ' -f1)"removed guard digest from blocklist: sha256:d83c00ceb39d975f51f62eef13b17bbb759c1a7f19c8c29100fc62a493f5e83d blocklist: ~/chio/state/chio/guards/blocklist.json
Removing a digest that was never listed exits 0 and says so, so the command is safe to run unconditionally:
$ chio guard blocklist remove "sha256:$(sha256sum quarantined-guard.wasm | cut -d' ' -f1)"guard digest was not blocklisted: sha256:d83c00ceb39d975f51f62eef13b17bbb759c1a7f19c8c29100fc62a493f5e83d blocklist: ~/chio/state/chio/guards/blocklist.json
A state directory the process cannot write is a hard failure, so an edit that did nothing never reads as success. This run removes the write bit from the directory; a read-only mount reports the same failure under a different errno:
$ XDG_STATE_HOME=./locked-state chio guard blocklist remove "sha256:$(sha256sum quarantined-guard.wasm | cut -d' ' -f1)"error[CHIO-CLI-IO]: cannot write the guard blocklist: ./locked-state/chio/guards is not writable (os error 13) help: make ./locked-state/chio/guards writable, or set XDG_STATE_HOME to a writable directory code: urn:chio:error:cli:io docs: https://chio.computer/docs/reference/errors#error-cli-io
The bracketed code, CHIO-CLI-IO, names the failure class, and the help line names the directory to fix. The command exits 1, so a runbook that branches on this failure branches on the exit status.
Read more in Guard Supply Chain, which covers the two digest domains.
4. Let the breaker close once the provider recovers
An open breaker returns its configured verdict, Deny by default, without calling the provider. It moves between states on its own timers: after the reset timeout it admits a bounded set of trial calls, and consecutive successes close it.
Restore the provider, its credentials, or the network path to it, and the breaker follows; External Guards lists each provider's credentials and endpoints. Hosts that embed chio-guards as a library can also call CircuitBreaker::reset, a public method that clears the failure window and returns the breaker to Closed for operator intervention. You should see calls the guard covers admitted again once the trial calls succeed.
5. Tune the adapter for a provider that keeps failing
When a provider is reachable but unreliable, tune its adapter so the breaker and cache absorb the failures. The adapter block nests under the provider in the policy:
# The adapter block nests under a provider, not a standalone file.
# Path here: guards.threat_intel.safe_browsing.adapter
guards:
threat_intel:
safe_browsing:
api_key: "sb-provider-key" # required provider credential
adapter:
# Cache aggressively because the provider is unreliable.
cache_ttl_seconds: 120 # default 60
# Provider QPS is 50; leave 20% headroom.
rate_per_second: 40 # default 20
rate_burst: 40 # default 20
# Trip the breaker after five failures in the rolling window.
circuit_failure_threshold: 5 # default 5 (unchanged)
# Retry transient failures up to twice (three total attempts).
retry_max_retries: 2 # default 3The block deserializes as ExternalAdapterPolicyConfig, which is deny_unknown_fields and has exactly five keys, so any other key fails policy validation. Four move off their defaults here. cache_ttl_seconds goes from 60 to 120 to absorb more of the provider's failures in cache. rate_per_second and rate_burst go from 20 to 40 to track the provider's stated 50 QPS with headroom. retry_max_retries drops from 3 to 2 so retries never amplify a degraded service, and circuit_failure_threshold stays at 5.
Every other breaker, retry, and cache setting comes from the compiled adapter defaults, including the backoff strategy, which is fixed to Exponential. The default table is on External Guard Adapters.
A restart empties session journals, and receipts keep the record
A SessionJournal is in-process state: a Mutex-guarded, capacity-bounded ring built through new, from_memory_budget, or with_caps. It is the fast index session-aware guards read during a session, and it lives only as long as the kernel process.
A restarted kernel starts every session journal empty, and session-aware guards rebuild their view from new traffic. The journal has no persistence and no replay from receipts, so there is nothing to repair after a restart. Signed receipts remain the durable record of every decision made before it.
Read more in Session-Aware Guards.