Failure & Recovery · Chio Docs

This is the runbook rather than the contract. Circuit-breaker, retry, and cache semantics are specified once, on External Guard Adapters, and the failure-mode matrix with the verdict each failure produces lives on Fail-Closed Semantics. What follows is the recovery step for each one.

Which failure am I looking at

Classify the incident against the failure-mode matrix on Fail-Closed Semantics, then come back here for the recovery step. Failures that trace to an external provider, meaning timeouts, permanent errors, an open breaker, or an empty token bucket, come from the adapter documented on External Guard Adapters.

Recovery patterns

Hot-reload a WASM guard

A WASM guard that hits its fuel ceiling or traps repeatedly is quarantined: the bundle store marks it unhealthy and pipeline evaluations short-circuit to Verdict::Deny. Recovery is a corrected bundle pushed through the hot-reload path, which validates the replacement before an atomic swap.

Before a reload is accepted, a canary harness replays a frozen corpus of exactly CANARY_FIXTURE_COUNT (32) fixtures against the new module; a corpus of any other size is rejected outright, and a failing fixture leaves the prior epoch untouched. Publish is an atomic epoch swap: in-flight evaluations keep their original module snapshot while new calls use the new epoch. The reload metric chio_guard_reload_total labels by outcome: applied, canary_failed, rolled_back.

A post-publish rollback watchdog attaches to every accepted reload and trips after 5 consecutive error-class verdicts inside a 60 second window: traps, fuel exhaustion, serialization failures, and other fail-closed backend errors. On trip it restores the prior module, emits chio.guard.reload.rolled_back, and writes an incident directory under ${XDG_STATE_HOME}/chio/incidents/<utc-iso8601>-<guard_id>-<reload_seq>/ holding incident.json and last_5_eval_traces.ndjson (redacted trace summaries only; request payloads are never persisted).

Digest blocklist

A known-bad guard build can be pinned out of rotation by digest. The local blocklist lives at ${XDG_STATE_HOME}/chio/guards/blocklist.json, falling back to ~/.local/state when that variable is unset or empty. Engine::reload refuses a replacement module whose sha256:<digest> is listed and returns E_GUARD_DIGEST_BLOCKLISTED, and chio guard pull runs the same check against the pinned OCI manifest digest before it contacts a registry or writes cache.

The digest is whatever sha256sum reports for the module being pinned out. The runs below stage one as quarantined-guard.wasm and point XDG_STATE_HOME at a state directory inside the project:

guard-blocklist · digesttranscript

$ sha256sum quarantined-guard.wasm

d83c00ceb39d975f51f62eef13b17bbb759c1a7f19c8c29100fc62a493f5e83d  quarantined-guard.wasm

exit 0

There is no chio guard blocklist add. The file is one JSON object with a digests array, so an entry goes in by writing it directly, which is also how a node with no CLI gets one:

guard-blocklist · pintranscript

$ mkdir -p "$XDG_STATE_HOME/chio/guards"
$ printf '{"digests":["sha256:%s"]}\n' \
    "$(sha256sum quarantined-guard.wasm | cut -d' ' -f1)" \
    > "$XDG_STATE_HOME/chio/guards/blocklist.json"
$ cat "$XDG_STATE_HOME/chio/guards/blocklist.json"

{"digests":["sha256:d83c00ceb39d975f51f62eef13b17bbb759c1a7f19c8c29100fc62a493f5e83d"]}

exit 0

Clearing an entry is the one subcommand. chio guard blocklist remove reports which of the two things happened and prints the file it operated on, so a recovery runbook logs the resolved path instead of guessing at XDG_STATE_HOME. Removing a digest that was listed:

guard-blocklist · removetranscript

$ chio guard blocklist remove "sha256:$(sha256sum quarantined-guard.wasm | cut -d' ' -f1)"

removed guard digest from blocklist: sha256:d83c00ceb39d975f51f62eef13b17bbb759c1a7f19c8c29100fc62a493f5e83d
blocklist: ~/chio/state/chio/guards/blocklist.json

exit 0

Removing one that was never listed is not an error, so the command is safe to run unconditionally from a runbook. It exits 0 and says so:

guard-blocklist · remove-absenttranscript

$ chio guard blocklist remove "sha256:$(sha256sum quarantined-guard.wasm | cut -d' ' -f1)"

guard digest was not blocklisted: sha256:d83c00ceb39d975f51f62eef13b17bbb759c1a7f19c8c29100fc62a493f5e83d
blocklist: ~/chio/state/chio/guards/blocklist.json

exit 0

A state directory the process cannot write is a hard failure rather than a silent skip, which keeps a blocklist edit from appearing to succeed where it did nothing. The run below points XDG_STATE_HOME at a directory with its write bit removed; a read-only mount reports the same failure under a different errno:

guard-blocklist · remove-unwritabletranscript

$ XDG_STATE_HOME=./locked-state chio guard blocklist remove "sha256:$(sha256sum quarantined-guard.wasm | cut -d' ' -f1)"

error [urn:chio:error:cli:other]: failed to update guard blocklist: failed to create guard blocklist path ./locked-state/chio/guards: Permission denied (os error 13)
context: {"domain":"cli","severity":"error","stability":"deprecated","string_code":"CHIO-CLI-OTHER"}
suggested fix: Preserve the original message and migrate the call site to a specific registry code when touched.

exit 1

The bracketed identifier is a registry URN and the string_code beside it, CHIO-CLI-OTHER, is its stable spelling. The registry summarizes that pair as a uncategorized compatibility error and rates its stability deprecated, so a runbook branching on this failure branches on the exit status rather than on the code.

Circuit-breaker recovery

The breaker transitions on its own timers; the state machine and its thresholds are specified on External Guard Adapters. What matters operationally is that the running service exposes no reset API. No admin endpoint and no CLI subcommand forces the breaker Closed, so deployment recovery follows provider recovery rather than a manual override. Callers that embed chio-guards as a library have one escape hatch: CircuitBreaker::reset is a public method that clears the failure window and returns the breaker to Closed for operator intervention.

Session journal loss

A SessionJournal is in-process state: a Mutex-guarded, capacity-bounded ring built through new, from_memory_budget, or with_caps. It has no persistence and no replay-from-receipts path, so it does not survive a kernel restart: a restarted kernel starts every session journal empty, and session-aware guards rebuild their view from new traffic. Signed receipts remain the durable audit record; the journal is the fast in-memory index guards read during a session, not a store to recover.

Worked example: tuned for a flaky provider

policy.yamlyaml

# The adapter block nests under a provider, not a standalone file.
# Path here: guards.threat_intel.safe_browsing.adapter
guards:
  threat_intel:
    safe_browsing:
      api_key: "sb-provider-key"        # required provider credential
      adapter:
        # Cache aggressively because the provider is unreliable.
        cache_ttl_seconds: 120          # default 60

        # Provider QPS is 50; leave 20% headroom.
        rate_per_second: 40             # default 20
        rate_burst: 40                  # default 20

        # Trip the breaker after five failures in the rolling window.
        circuit_failure_threshold: 5    # default 5 (unchanged)

        # Retry transient failures up to twice (three total attempts).
        retry_max_retries: 2            # default 3

The adapter block deserializes as ExternalAdapterPolicyConfig, which is deny_unknown_fields and exposes exactly five knobs: keys outside that set fail policy validation. Four move off their defaults here: cache_ttl_seconds 60 to 120 to absorb more of the provider's flakiness in cache; rate_per_second and rate_burst both 20 to 40 to track the provider's stated 50 QPS with headroom; and retry_max_retries 3 to 2 so a degraded service is not amplified. circuit_failure_threshold stays at its default 5.

Those five keys are the whole policy surface. Everything else the breaker, retry loop, and cache use comes from the compiled adapter defaults and has no policy-YAML mapping, the backoff strategy included: it is fixed to Exponential. The default table is on External Guard Adapters.

Next steps

Fail-Closed Semantics · the failure-mode matrix and the invariant behind these defaults
External Guard Adapters · the circuit breaker, retry, and cache contract
Custom WASM Guards · where a guard digest comes from, and the manifest that binds it
Node Observability · which signals to watch for early breaker telemetry
External Guards · operator contract for the six bundled providers

PreviousExternal Guard Adapters NextGuard Supply Chain