Health & Readiness

Two questions, two routes

A platform probe asks two different questions and they have different correct answers. "Is this process alive?" is answered by GET /chio/live, which returns 200 while the process runs and consults nothing. "Can this process still serve honestly?" is answered by GET /chio/health, which reads runtime dependencies and returns 503 when they are broken.

Collapsing the two is a real failure mode, in both directions. A dependency-aware liveness probe recycles a container that is serving correctly through a transient blip. A process-only readiness probe keeps an instance in rotation after its receipt database stopped accepting writes, so it accepts traffic it can only deny. Both routes are fixed paths on the sidecar and are not configurable; the deployment detail lives in Sidecar HTTP Service.

The health flag

The routes report. The flags decide. HealthFlag, from crates/core/chio-supervisor, a leaf crate that depends on no other Chio crate, is the primitive a supervised worker carries, and it is what the kernel’s pre-dispatch gates read before admitting a mediated call. Readiness reads a dependency probe of its own, and does not read this flag at all; that gap is covered further down. The flag is a cloneable handle over shared atomics, so level and counter reads are lock-free and cheap enough to run on the pre-dispatch path; only the human-readable reason string sits behind a mutex.

HealthFlag::snapshot() renders those fields as a serializable HealthSnapshot: camelCase field names, snake_case level values such as "degraded", optional fields #[serde(default)]. It is an operator payload, not a signed artifact. Nothing in the tree serializes the whole snapshot yet: the only reader is the SQLite receipt store, which lifts level and restart_total out of it into writerLevel and writerRestartTotal on its own health report.

Three levels, one downgrade

The rule that makes the flag worth reading is that it never heals itself. A completed unit of work resets the consecutive-failure counter and stamps liveness, and deliberately does not lower the level:

/// Record a completed unit of work. Resets the consecutive-failure counter and
/// stamps liveness, but never lowers a tripped level.
pub fn record_ok(&self, now_ms: u64) {
    self.0.consecutive_failures.store(0, Ordering::SeqCst);
    self.0.last_ok_unix_ms.store(now_ms, Ordering::SeqCst);
}

A supervisor that restarted its worker but lost work in the gap must still report the gap. Raising is a compare-and-swap loop, so a concurrent escalation to Failed can never be lost behind a lower-severity raise, and clear is the only path back to Healthy. The unit suite pins each edge (record_ok_resets_consecutive_failures_without_lowering_level, raise_is_monotonic_and_ignores_lower_severity, clear_is_the_only_downgrade_and_preserves_restart_total), and a property test over random operation sequences asserts the level is non-decreasing between clears while restart_total equals the number of recorded failures. A nightly loom model covers one worker recording a failure against one concurrent reader and asserts the reader never observes a torn or invalid level. That is the whole model: it is a two-thread interleaving check, not a proof about the supervisor loop.

tcb_critical: denying at the door

tcb_critical is the difference between a flag that informs an operator and a flag that changes verdicts. It is a single predicate:

/// True when the surface must fail closed: TCB-critical and not
/// [`HealthLevel::Healthy`]. Read on the pre-dispatch path.
#[must_use]
pub fn is_serving_closed(&self) -> bool {
    self.0.tcb_critical && !matches!(self.level(), HealthLevel::Healthy)
}

There is no grace band. The instant the level leaves Healthy, a TCB-critical flag reports serving-closed, and the callers that read it deny before dispatch rather than after. Two flags do this today.

The lock-poison flag. The kernel holds one HealthFlag::new(true) for trusted state. When a lock guarding multi-field state is found poisoned, meaning a panic unwound while it was held, record_tcb_lock_poison records a failure with trip_after = 0, so the flag is degraded and serving-closed on the first occurrence. The pre-dispatch gate ensure_tcb_locks_healthy then returns a fail-closed deny for every subsequent evaluation. Every in-tree call site is a budget-registry lock, where continuing on half-mutated state would be silent monetary accounting corruption.

The receipt commit writer. The SQLite receipt store supervises its writer thread with tcb_critical: true and trip_after: 1, so any writer fault is immediately visible. ensure_receipt_persistence_ready consults it before dispatch. This ordering is the whole point: the kernel persists receipts after the tool executes, so a writer discovered dead afterwards means a side effect already happened with no durable evidence. Denying at the door means the tool never runs.

Restart budget and backoff

SupervisorConfig carries the whole policy. Every field is public, and SupervisorConfig::new(name, tcb_critical) supplies a conventional default. Read the convention column as the crate’s documented starting point and the receipt commit writer column as what a running node uses: the writer, the one supervised worker in the tree, builds the struct field by field rather than calling ::new.

The delay for the count-th consecutive restart is min(max_backoff, base_backoff * 2^(count - 1)), computed with checked_shl and saturating multiplication, so a large count saturates at the ceiling instead of overflowing.

Two details matter for reading the counters. First, a worker whose iteration completes returns SupervisedOutcome::Continue, which resets the consecutive counter; isolated faults separated by healthy work therefore never accumulate into a trip. Second, the workspace release profile sets panic = "unwind", so the catch_unwind restart path is live in release builds and not only under test profiles.

The crate also ships an async supervisor, SupervisedTask, behind an optional async cargo feature. No workspace crate enables that feature, so it is compiled into nothing that ships. Where it is enabled it runs each iteration in its own child task, because a panic across an .await point is not something catch_unwind captures cleanly.

What readiness actually checks

On the sidecar, readiness is one method on ProxyState:

pub(crate) async fn readiness_status(&self) -> SidecarStatus {
    if let Some(store) = &self.receipt_store {
        let store = store.lock().await;
        if !store.is_reachable() {
            return SidecarStatus::Unhealthy;
        }
    }
    SidecarStatus::Healthy
}

is_reachable is not a connection ping. A bare SELECT 1 still answers over dropped tables, a read-only mount, or a full disk, which would hold an instance in rotation while every append fails after an already-allowed upstream call. Instead it inserts into http_receipts and tool_receipts under the reserved primary key __chio_readiness_probe__ inside a transaction that is always rolled back, so all three fail readiness and no probe row is ever persisted. The test reachability_probe_touches_the_write_path_and_persists_nothing drops http_receipts out of band and asserts the store stops being reachable.

The handler maps that status to a code: Healthy to 200, anything else to 503, alongside the version and the embedded kernel’s receipt_backend and revocation_backend. Liveness returns 200 with those two backend fields left empty, because a process-only answer does not inspect storage.

Both routes against a sidecar started with chio api protect and a durable receipt store. The difference is visible in the payload, not the status line:

$ curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:9090/chio/health
{"status":"healthy","version":"0.1.0","receipt_backend":"durable","revocation_backend":"durable"}
HTTP 200

$ curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:9090/chio/live
{"status":"healthy","version":"0.1.0","receipt_backend":"","revocation_backend":""}
HTTP 200

Writer liveness verdicts

A dead worker is the easy case. The harder one is a worker that is alive and making no progress, which the supervisor flag alone cannot see because the thread never exited. The store classifies that case from its own in-memory writer counters, and the kernel gates on the verdict:

An opt-in watchdog polls this every receipt_writer_poll_ms (default 1,000) and publishes the verdict the gate reads. If no watchdog is running the published verdict stays Unknown, which is permissive, so the gate falls back to sampling the store inline on the pre-dispatch path. A host that never started the watchdog still fails closed on a wedged writer; it just pays the sample per evaluation.

chio receipt health reads the same counters out of a store on disk, which is how an operator sees the verdict without a running edge. On a store that has committed two entries and never checkpointed:

$ chio --receipt-db ./chio.db receipt health
status: healthy
committed_entry_seq: 2
checkpoint_seq: none
checkpointed_entry_seq: 0
writer_level: healthy
writer_restart_total: 0
uncheckpointed_range: 1..=2
writer_accepted_total: 0
writer_committed_total: 0
writer_failed_total: 0
writer_timed_out_total: 0
writer_timed_out_inflight: 0
writer_saturated_total: 0
writer_inflight: 0
writer_last_commit_unix_ms: none
writer_last_error: none
db_size_bytes: none
retention_watermark_entry_seq: none

The writer counters all read zero here: this invocation opened the file and did not own the commit writer. The --json form separates the two readings that the human form collapses, carrying "writerLevel": "healthy" beside "writerLiveness": "unknown" under schema chio.cli.receipt.health.v1, which is the permissive Unknown row of the table above.

Guarantees and limits

Three more boundaries are easy to read past. Readiness consults the sidecar receipt log and nothing else: capability-authority reachability and policy load state are not part of the verdict, so a node can report ready with a dependency the probe does not model. SidecarStatus::Degraded exists in the type and maps to 503 in the handler, but no code path returns it, so readiness is binary in practice. And chio receipt health always opens the database read-only from outside the serving process. It rejects --control-url outright, and it cannot see the serving process’s in-memory supervisor, so it prints writer_level: healthy and serializes writerLiveness: "unknown" whatever that process is actually doing, alongside zeroed writer counters. Those are structural defaults, not observations, and the human renderer does not print the liveness field at all, so read it from --json if you need it. Read writer health from the serving node.

Claim: an unready node leaves rotation

Next Steps

• Cluster Overview · per-subsystem degradation once more than one node answers under the same authority
• Sidecar HTTP Service · the two routes in deployment context, the response body, and probe wiring
• Node Observability · what to watch beyond the probe, and why a silent exporter looks like a quiet one
• Failure & Recovery · what the guard runtime does when a dependency is the thing that broke