Health & Readiness

Readiness probe

The receipt check is not a connection ping. A bare SELECT 1 still answers over dropped tables, a read-only mount, or a full disk, which would hold an instance in rotation while every append fails after an allowed upstream call. The probe inserts into http_receipts and tool_receipts under the reserved key __chio_readiness_probe__ inside a transaction that is always rolled back. reachability_probe_touches_the_write_path_and_persists_nothing drops http_receipts out of band and asserts readiness fails.

The handler maps Healthy to 200 and Degraded or Unhealthy to 503, alongside the version and the embedded kernel’s receipt_backend and revocation_backend. A sidecar with no receipt_db has no receipt store to fail, so its receipt check always passes.

Rotation contract

A node whose receipt store can no longer accept a write, whose commit writer has tripped, or whose authority or policy is unavailable returns 503 from /chio/health, and a platform with a readiness probe pulls it from routing instead of sending it traffic it can only refuse. The contract covers one sidecar process. readiness_status and sidecar_health_handler implement it; the readiness probes in the Cloud Run and Azure reference manifests and the ECS container healthCheck act on it.

Writer conditions

writer_serving_closed() is true when the supervised thread’s flag has left Healthy, when the store has no writer handle, or when the writer’s verified head is poisoned. The head starts poisoned: ReceiptCommitWriterHealth::default sets it, and the actor clears it only after it seeds a head, so a store still attaching cannot admit a governed action before its first append could reject. A store-wide append fault poisons it again, and every append is rejected until an operator reseeds. The thread flag alone cannot see this, because the thread is alive.

The kernel holds one HealthFlag::new(true) for trusted state. record_tcb_lock_poison records a failure with trip_after = 0 and increments chio_lock_poison_total. The receipt store supervises its writer with tcb_critical: true and trip_after: 1.

Liveness classification

Dead is classified when the writer’s last recorded error contains unavailable; the store writes sqlite receipt commit actor is unavailable when it finds the channel disconnected at enqueue. Wedged covers a caller-timed-out command still owned by the actor, or a backlog with no progress for receipt_writer_stall_ms (default 10,000). The stall clock anchors to the later of the last commit and the current backlog’s start, so an idle writer is not judged wedged the instant work arrives; with neither anchor set it is never stalled. The watchdog polls every receipt_writer_poll_ms (default 1,000).

Health flag fields

HealthFlag lives in crates/core/chio-supervisor, a leaf crate with no other Chio dependency. It is a cloneable handle over shared atomics, so level and counter reads are lock-free on the pre-dispatch path; only the reason string sits behind a mutex.

HealthFlag::snapshot() renders these as a serializable HealthSnapshot: camelCase field names, snake_case level values such as "degraded", optional fields #[serde(default)]. It is an operator payload, not a signed record. The SQLite receipt store lifts level and restart_total into writerLevel and writerRestartTotal on its health report, and the SIEM exporter’s ExporterManager::health() embeds the whole snapshot.

Level transitions

/// Record a completed unit of work. Resets the consecutive-failure counter and
/// stamps liveness, but never lowers a tripped level.
pub fn record_ok(&self, now_ms: u64) {
    self.0.consecutive_failures.store(0, Ordering::SeqCst);
    self.0.last_ok_unix_ms.store(now_ms, Ordering::SeqCst);
}

Raising is a compare-and-swap loop, so a concurrent escalation to Failed is never lost behind a lower raise, and clear is the only path back to Healthy. The unit suite pins each edge (record_ok_resets_consecutive_failures_without_lowering_level, raise_is_monotonic_and_ignores_lower_severity, clear_is_the_only_downgrade_and_preserves_restart_total), and a property test over random operation sequences in tests/health_properties.rs asserts the level never decreases between clears while restart_total equals the number of recorded failures. A loom model in tests/loom_health.rs, run nightly behind --cfg loom, checks one worker recording a failure against one concurrent reader for a torn or invalid level; it is a two-thread interleaving check of the flag.

Supervisor configuration

SupervisorConfig carries the policy, every field public. SupervisorConfig::new(name, tcb_critical) supplies the convention below; the receipt commit writer builds the struct field by field.

The delay for the count-th consecutive restart is min(max_backoff, base_backoff * 2^(count - 1)), computed with checked_shl and saturating multiplication. An iteration that completes returns SupervisedOutcome::Continue and resets the consecutive counter, so isolated faults separated by healthy work never accumulate into a trip. The release profile sets panic = "unwind", so the catch_unwind restart path is live in release builds.

The async supervisor, SupervisedTask behind the async feature, runs each iteration in its own child task, because catch_unwind does not capture a panic across an .await cleanly. Trust-control’s cluster sync loop and the SIEM exporter run under it, the first reporting peer staleness on trust-control’s /health.

Receipt health command

chio receipt health samples the database through a separate read-only connection and rejects --control-url. It cannot see the serving process’s supervisor, so it prints writer_level: healthy with zeroed writer counters and serializes writerLiveness: "unknown" whatever that process is doing. The --json form, schema chio.cli.receipt.health.v1, carries writerLevel beside writerLiveness; the human form omits the liveness field.