Cluster Recovery

Repair is a peer-by-peer decision

There is no cluster-wide repair operation in Chio. No node can tell another to resync, no node can evict a peer, and nothing coordinates recovery across the fleet. What exists is one flag per peer entry per node, force_snapshot, and the next sync round’s decision to honour it by pulling a full snapshot instead of a delta page. Every recovery path on this page ends at that one decision.

The status is shipped and drilled. Eleven cluster tests in crates/products/chio-cli/tests/trust_cluster.rs drive real chio trust serve processes on loopback; the six drills tabled below are the ones this page relies on. Three of those are named checks in scripts/qualify-trust-control.sh (quorum-heal, stale-leader-fencing, replay-and-failover), the seven-check trust-control gate the operations runbook lists under Configuration Checks Before Promotion. Three of the eleven are #[ignore]d, including both callers of the quorum-loss proving scenario; the sections below say where that matters. The ceiling all of it is proving against is the one the runbook states: local or leader-local single-writer truth with deterministic leader selection and eventual repair, explicitly not consensus-backed HA.

What one node keeps about a peer it must repair

Recovery state is per-observer. Node A’s opinion of node C lives in A’s PeerSyncState for C and is never sent anywhere. Seven of its fields carry the repair, and GET /v1/internal/cluster/status reports every one of them under peers.

The cluster block on GET /health carries the same picture in aggregate, behind the bearer service token rather than a cluster-peer digest: peerCount, healthyPeers, unhealthyPeers, unknownPeers, partitionedPeers, lastErrorCount, selfUrl, leaderUrl, hasQuorum, quorumSize, reachableNodes, role, and electionTerm. It reports counts and never the offending error strings, so start an incident there and move to the internal status route for lastError, the per-peer cursors, and the snapshot counters.

Quorum loss stops writes and nothing else

A node below quorum computes no leader and fails every cluster-aware mutating request with 503. It keeps serving reads from its own disk, unchecked: every read handler validates the service token and queries the local store, and none of them consults the consensus view at all. Two drills assert the refusing half against live processes. trust_control_cluster_requires_quorum_and_heals_after_partition isolates one of three nodes and asserts leaderUrl: null, hasQuorum: false, reachableNodes: 1, role: candidate, and 503 on POST /v1/budgets/increment; the multi-region drill repeats the 503, on a tool receipt, once.

Treat that as the recovery contract, not a bug. A minority node is a correct reader of stale state and a refusing writer, and nothing marks its read responses as degraded, so a client that cannot tolerate staleness must consult hasQuorum itself. Recovery is restoring reachability: quorum returns on its own once enough peers answer a status probe, and no operator action promotes anything, because there is no promotion step to take.

Injecting a partition, and healing it

POST /v1/internal/cluster/partition is a production route, registered unconditionally in the router and authenticated like every other internal cluster route: a chio.cluster.peer.v1 digest over the shared service token from a node id in this node’s own --peer-url allowlist, covered in Cluster Peer Auth. There is no chio subcommand for it. The body carries the complete blocked set, so an empty blockedPeerUrls array unblocks everything; a URL that fails normalize_cluster_url is a 400, an unclustered node answers 404 cluster replication is not configured, and the node’s own URL is filtered out of the set it applies.

for (peer_url, peer_state) in &mut guard.peers {
    let was_partitioned = peer_state.partitioned;
    peer_state.partitioned = blocked.contains(peer_url);
    if peer_state.partitioned {
        peer_state.last_error =
            Some("cluster peer intentionally partitioned".to_string());
        peer_state.force_snapshot = true;
    } else if was_partitioned {
        peer_state.health = PeerHealth::Unknown;
        peer_state.last_error = None;
        peer_state.force_snapshot = true;
        peer_state.delta_records_since_snapshot = 0;
    }
}

The heal branch is the interesting half. Releasing a peer does not resume its cursor. It resets health to Unknown, clears the error, zeroes the delta counter, and flags a snapshot, so the peer re-enters as if it had never been seen. That is deliberate: a cursor formed before a block cannot be trusted to be contiguous with what happened during it, and the snapshot path is the only repair that does not need it to be. Quorum therefore does not return the instant you unblock: Unknown is not reachable, so the peer rejoins the candidate set only after the next round’s status probe succeeds.

It rejoins one step before it converges, though, and that gap is the part to hold onto. sync_peer calls update_peer_reachable as soon as the probe answers, ahead of the snapshot fetch, and compute_cluster_consensus_locked reads only health, partitioned, and last_contact_at, never force_snapshot or any cursor. A node flagged for a full resync is a full candidate, and if it sorts lowest it is leader again before a byte of repair has landed. Quorum, leadership, and read-serving resume on reachability alone; only budget witnessing waits, because budget_write_quorum_commit_view is the one view that reads the flag.

The force-resync decision

One predicate decides whether the next visit to a peer pulls a full snapshot.

pub(crate) fn peer_should_force_snapshot(state: &TrustServiceState, peer_url: &str) -> bool {
    with_peer_state(state, peer_url, |peer| {
        peer.force_snapshot
            || peer.delta_records_since_snapshot >= CLUSTER_SNAPSHOT_RECORD_THRESHOLD
    })
    .unwrap_or(false)
}

The flag clears in exactly two places, and both are on the success path. apply_cluster_snapshot clears it while resetting every cursor to the snapshot heads and clearing the peer’s cached budget acks in the same update, and update_peer_success clears it at the end of a round that reached finalize_peer_sync_round. That function returns early when the peer was demoted or is still pending a snapshot, so a failed repair cannot clear its own flag.

What a failed repair also cannot do is announce itself. The fetch is client.cluster_snapshot()? inside sync_peer: on error it propagates without calling update_peer_failure or update_peer_sync_error, and sync_cluster_once discards the result. A peer wedged on a snapshot it can never fetch therefore keeps reporting health: healthy, keeps counting toward quorum, keeps whatever stale lastError it already had, and shows the problem only as forceSnapshot: true against a frozen snapshotAppliedCount. Alert on that pair, not on health.

A late joiner is a peer nobody has seen yet

Joining is not a protocol. A node that starts late is just a peer whose entry on every other node still holds the default force_snapshot: true, and whose own peer entries hold it too. Its first successful visit to each peer pulls a full snapshot in both directions of interest. The precondition is configuration, not discovery: the late node must already be in every running node’s --peer-url list, because the peer map is built once at startup and the allowlist that authenticates internal calls is the same list.

trust_control_cluster_late_joiner_catches_up_from_snapshot_and_compacts runs it. Two of three nodes come up, converge on a leader with reachableNodes: 2 against a quorum of 2, and take ten tool receipts. The third node then starts and the test waits for it to report all ten, hold quorum, and show a peer entry with snapshotAppliedCount at least 1 and a non-null lastSnapshotAt. Ten more receipts follow, and the second assertion is the compaction half: the joiner reaches all twenty, crosses the eight-record threshold, applies a second snapshot, and settles with forceSnapshot: false. Both counter assertions are peers.iter().any(...), so they say the joiner snapshotted some peer, not which. That is adequate here because the joiner is fresh. It is not adequate in the quorum-heal drill, where the healed node has been running since bring-up and already holds a snapshot against both peers, so its snapshotAppliedCount >= 1 is satisfied before the partition is ever injected and the budget row is the only thing proving the heal.

What a snapshot carries matters for what the joiner may then do. The budget half replays mutation events, not only their projections, so lifecycle continuity survives the join: trust_control_cluster_snapshot_replays_holds_and_mutation_events opens the late node’s budget SQLite store directly and finds cap-snapshot-hold-1:authorize, :release, and :capture-invocation in order, then posts a reconcile through that node and finds the hold settled in its store at exposure 0 and realized spend 45. The reconcile is forwarded, not handled locally: the assertion is assert_leader_visible_metadata, so what the drill proves is that a joiner can carry a pre-join hold to the leader and take the result back, not that a joiner may settle one by itself. This drill is in the release gate as replay-and-failover.

The authority half is narrower. apply_snapshot merges the cluster’s public key and trusted-key history and never touches seed_hex, under a comment that says as much: cluster snapshots replicate verification history, not signing custody. The consequence is sharper than a missing capability. Once the replicated public key differs from the one the local seed derives, read_current_keypair refuses outright with local signing seed public key ... does not match replicated authority public key ..., so the joiner cannot sign at all locally and every issuance has to reach the leader. That is why /v1/capabilities/issue forwards. The rest is Authority & Rotation.

Two rejections a returning node must survive

A node that was leader, died, and came back is the dangerous case, because it can hold a term the rest of the cluster has moved past. Two independent checks stand in front of an authority mutation, and they fire at different depths.

The header check only engages when a request arrives with cluster peer headers; a plain bearer request falls through to service auth and carries no term. The fence check needs a cluster lease and --authority-db: unclustered it returns Ok(None) and does nothing, and without an authority database the SQLite step is skipped entirely, leaving only the in-memory lease check. That is why a clustered node is refused at startup when it is given --authority-seed-file instead. Both checks are described from the write path’s side under the term fence.

The restart half is what makes the drill more than a failover test. A returning node reloads its persisted fence row only if it still matches the authority’s current generation and rotation stamp, so its term starts where it left off, behind the cluster. It catches up through the same snapshot path everything else uses:

let conflicting_same_term_self_leader = snapshot_term == guard.election_term
    && guard
        .last_leader_url
        .as_deref()
        .is_some_and(|leader| leader == guard.self_url)
    && snapshot_leader
        .as_deref()
        .is_some_and(|leader| leader != guard.self_url);
if conflicting_same_term_self_leader {
    let now = unix_timestamp_now();
    guard.election_term = guard.election_term.saturating_add(1);
    guard.last_leader_url = Some(guard.self_url.clone());
    guard.term_started_at = Some(now);
    guard.lease_expires_at = Some(now.saturating_add(guard.lease_ttl_ms / 1000));
    return;
}

Read that branch carefully, because it does not resolve a disagreement, it escalates one. When a snapshot arrives at the same term naming a different leader while this node believes it is the leader, the node keeps itself as leader and bumps its own term. Otherwise the branch below adopts the snapshot’s higher term, leader, and lease outright. Either way seed_cluster_fence has already run, writing the snapshot’s term to disk when it is higher than the persisted one, so a later restart starts current. The escalation is a local tiebreak in favour of the incumbent, not agreement: election_term is a per-node counter and nothing reconciles two nodes’ views of it.

trust_control_cluster_rejects_stale_authority_term_after_failover_and_restart walks the whole path on three processes: record the term, kill the leader, wait for the survivors to converge on a higher one, restart the dead node, let the cluster reconverge. It then synthesizes the attack rather than waiting for the restarted node to mount it, posting a cluster-peer-authenticated POST /v1/authority at the current leader under another node’s id carrying the pre-failover term. It asserts 409 and, more usefully, that the authority generation is unchanged afterwards. It then posts the same mutation cleanly to a follower and asserts the generation advances by exactly one with handledBy naming the leader. The rejection and the still-working write are one test on purpose: fencing that also breaks the healthy path is not a fence. The 409 it observes comes from the header check, the shallower of the two; the persisted fence is not isolated by any drill here.

The drills

Every drill here spawns its nodes at a 2000ms --cluster-sync-interval-ms, a 5s lease and a 5s staleness window, so their timings say nothing about the 500ms CLI default. Every drill also opens with skip_when_loopback_bind_denied and returns green without running when the environment refuses a 127.0.0.1:0 bind. A passing trust-control gate is therefore evidence the drills did not fail, not evidence they ran; check the per-check logs under target/release-qualification/trust-control/logs/.

Read the multi-region report before you cite it. It cuts and heals the same minority twenty times and measures from heal until every node agrees on the leader and can see the receipt written during the cut. Its own notes call those "local simulated-region qualification numbers, not hosted WAN latencies". Of the four entries under consistencyChecks, only leaderUrl is measured. healedClusterRestoresQuorum restates a wait the loop performs every cycle, minorityWritesFailClosed restates a 503 assertion that runs on the first cycle only and against a tool receipt rather than a budget write, and splitBrainObserved: false restates nothing, because nothing in the loop watches for two leaders at once.

Guarantees and limits

Next Steps

• Replication & Convergence · the pull round every repair runs inside, its page guards, and what costs a peer its witness standing
• Backup & Restore · the node-local half: snapshots of one process’s stores, and the checkpoint chain a restored receipt log rejoins
• Leader & Failover · how the leader is picked, where a write goes, and the term fence from the write path’s side
• Authority & Rotation · the one keypair a snapshot replicates by public half only, and what promotion does not confer
• Budgets Across Nodes · quorum-commit witnessing, and why a resyncing peer witnesses nothing
• Health & Readiness · what one node reports about itself before any of this applies