Cluster Recovery

A recovered cluster has every peer back in its intended local view, a fresh snapshot applied on the affected peer, the records the next workflow needs readable there, and fresh budget acknowledgments before spending resumes.

A restarted, late, or partitioned trust-control node repairs itself by one snapshot-and-delta pull per source peer, covering authority, budget history, revocations, receipts, and lineage. Health shows a peer is reachable, snapshot progress shows a repair step finished, and the records show what the workflow can use. Local store restores belong to Backup & Restore.

A healed node routes again before its data catches up

Each node computes its leader from the peers it can reach. A partitioned node and a stopped leader both change that view, and both come back in the same order: routing first, then repair.

Quorum is peers.div_ceil(2) + 1, so three nodes need two. Below it, a node names no leader, clears its lease, and answers a covered write with 503; once it heals, it selects the majority’s leader at a new local term and catches up from a snapshot. A lease lasts three sync intervals, clamped between 500 ms and 5 s, and a stopped leader’s last stamp lapses while the new term is already serving.

1. Record each node’s view before changing anything

Set CHIO_CONTROL_URL to one node’s control-service URL and read its health. The liveness route takes no credential. The output is the isolated node of a three-node cluster.

curl --fail-with-body --silent --show-error "$CHIO_CONTROL_URL/health" \
  | jq '{ok, clustered, cluster: .cluster
        | {role, hasQuorum, quorumSize, reachableNodes, leaderUrl, electionTerm}}'

{
  "ok": true,
  "clustered": true,
  "cluster": {
    "role": "candidate",
    "hasQuorum": false,
    "quorumSize": 2,
    "reachableNodes": 1,
    "leaderUrl": null,
    "electionTerm": 2
  }
}

You should see the node’s own selection: role, hasQuorum, leaderUrl, electionTerm, and the peer counts. Repeat on every node and compare; equal quorum counts can hide different leaders and different data.

For the affected peer, read GET /v1/internal/cluster/status with cluster peer authentication; a bearer token alone does not authenticate it. Record the source URL with its counters, so a later increase is attributed to the same peer. The status fields are in the reference below.

2. Restore the intended peer view

A node below quorum reports role: candidate and refuses leader-forwarded handlers with 503, while replica repair keeps changing its stores and local budget reads keep returning existing data. A successful read and hasQuorum: true both leave freshness open.

• Configuration. Advertised and configured peer URLs, service-token agreement, and reachability.
• Authority. A receiving node with an authority database needs its peer to serve the authority snapshot; a failed authority merge ends the visit before the delta lanes, and reconnecting leaves a configuration mismatch in place.
• Intentional partitions. The blocked-peer set each node holds, cleared below.

The authenticated POST /v1/internal/cluster/partition route replaces the receiving node’s whole blocked-peer set, and an empty array clears it. It uses cluster peer authentication and has no CLI subcommand.

{
  "blockedPeerUrls": []
}

A released peer becomes unknown, its error and delta counter clear, and a full snapshot stays required; the next successful status probe makes it reachable, and repair follows. A malformed URL returns 400, and an unclustered node returns 404. Restored quorum can change the selected leader before the snapshot applies, so treat the new role as a routing observation.

A block is local. Blocking C on A changes A’s pull scheduler, leader candidates, and budget witnesses, while C can still pull from A and A still serves C’s internal requests. Independent blocks can produce intersecting views with different leaders, drawn on Leader & Failover, and the route exercises those views apart from any packet loss or latency fault.

3. Confirm the snapshot applied and the records are there

A new peer entry starts with a required snapshot. The same path runs after intentional isolation, selected contact or protocol failures, an unpageable delta window, a revocation-stream recovery condition, or once eight delta records accumulate. A transient delta error alone leaves the peer in place.

You should see the affected peer’s snapshotAppliedCount rise above the baseline from step 1. A count that was already positive can come from startup, so the quorum-heal drill proves catch-up with the newly replicated budget row. One record shows that record arrived, and the rest of the store needs its own reads.

4. Check authority and spending before resuming work

For a budget operation, confirm the writer’s same-origin acknowledgment requirement and the eligible witness set in Budgets Across Nodes. Reachability and an applied snapshot leave the write’s acknowledgment frontier to be met.

1. Forwarded authority term. A peer-authenticated mutation carries x-chio-cluster-auth-term; a term that differs from the receiver’s current lease term returns 409 before anything changes. Plain bearer requests carry no forwarded term.
2. Local authority fence. With an authority database, the lease term and leader are compared with the persisted fence: an expired lease returns 503, and conflicting or stale fenced state returns 409.

On restart, a persisted fence comes back only when its authority generation and rotation timestamp still match, and a compatible newer snapshot can seed the local term. When a same-term snapshot names another leader while this node holds the lease, the node increments its own term and keeps itself; that rule resolves its own conflict, and the coordination contract describes the leader-local guarantee it belongs to.

5. Keep a record of what recovered

Keep the before-and-after local views, the affected peer’s snapshot counter change, and the exact records that validated the workflow. For a budget write, keep its operation identity and witness metadata; for an authority mutation, the resulting generation and the node that handled it. That record replaces a healthy process, a pre-existing counter, or one successful request as evidence of agreement.

A preconfigured late node catches up from its peers

A node can start after its peers and catch up from their snapshots when the peer lists and authentication already admit it. Each node reads its peer list at startup, so adding a new member means updating and restarting every node whose membership changes.

Budget snapshots carry the mutation history that rebuilds existing holds; the late-joiner drill reconciles a pre-join hold through the joiner, which forwards the reconcile to the selected leader. Authority snapshots carry public trust state and retained key history, while the signing seed stays with each node’s custody. A node whose seed derives a key other than the replicated current authority key refuses to sign, even as leader; restore its custody through Authority & Rotation.

Two symptoms stop a recovery

Healthy peer, snapshot still pending

When forceSnapshot stays true and snapshotAppliedCount stops rising while the peer reads healthy, diagnose the snapshot path from those two fields. Check the source snapshot’s availability, its response size, transport failures, and store or import errors.

A full snapshot is assembled into one response and read under a 64 MiB cap, so a larger one fails every retry unchanged. The 15-second HTTP timeout bounds the transfer, separately from decode and local import. Application runs store by store (authority, revocations, receipts and lineage, budgets, fence state) and updates cursors and counters last, so a failed visit can leave the stores applied before the failure changed; restore local stores through Backup & Restore.

A recovered leader refuses to sign

The node’s local seed derives a key other than the replicated current authority key, so local signing refuses. Leader selection supplies no key; restore the intended custody through the deployment’s authority procedure, then repeat step 4.

Recovery repairs leader-local state among the authenticated peers of one operator’s trust domain. It provides no consensus across nodes and no Byzantine fault tolerance, and it never recovers a signing key.

Recovery reference

chio release verify --evidence qualification

qualification  bound to v0.2.0
  trust-control drills: node-identity               pass
  trust-control drills: quorum-heal                 pass
  trust-control drills: stale-leader-fencing        pass
  trust-control drills: denied-event-replication    pass
  trust-control drills: replay-and-failover         pass
  trust-control drills: late-joiner-catch-up        pass
  trust-control drills: repeat-run                  pass
verified

Next: watch the peers converge