Leader & Failover

The lowest-sorted node a quorum can see leads

A node counts itself and every peer that is healthy, unpartitioned, and contacted within the lease TTL. Once that set is a majority of the configured cluster, the lowest-sorted URL in it is the leader.

The sync loop supplies the peer observations, and status reads and forwarding checks recompute the leader from them. A node reports one of three roles: leader when the pick is its own URL, follower when it is a peer’s, and candidate when there is no quorum and so no leader. A candidate runs no campaign; it names a node with no selected leader. An unclustered node reports the literal standalone, and the internal status route substitutes it when no consensus view exists.

let now = unix_timestamp_now();
let lease_ttl_secs = Duration::from_millis(cluster.lease_ttl_ms).as_secs().max(1);
let quorum_size = cluster.peers.len().div_ceil(2) + 1;
let mut candidates = vec![cluster.self_url.clone()];
for (peer_url, peer_state) in &cluster.peers {
    let contact_is_fresh = peer_state
        .last_contact_at
        .is_some_and(|last_contact_at| now <= last_contact_at.saturating_add(lease_ttl_secs));
    if peer_state.health.is_reachable() && !peer_state.partitioned && contact_is_fresh {
        candidates.push(peer_url.clone());
    }
}
candidates.sort();
let reachable_nodes = candidates.len();
let has_quorum = reachable_nodes >= quorum_size;
let leader_url = if has_quorum {
    candidates.first().cloned()
} else {
    None
};
if cluster.last_leader_url != leader_url {
    cluster.election_term = cluster.election_term.saturating_add(1);
    cluster.last_leader_url = leader_url.clone();
    cluster.term_started_at = leader_url.as_ref().map(|_| now);
}

Read more in Cluster Recovery, which drills each scenario.

Each node’s own view sets its quorum and its leader

Every input to the pick is local: the node’s configuration, its peer map, and its clock. Five consequences follow from the selection rule.

1. A node always counts itself. candidates starts with self_url and nothing tests local health, so reachable_nodes is never zero.
2. Quorum is a majority of this node’s configuration. Two peers make a quorum of two and four peers a quorum of three. In a two-node cluster, either node going quiet stops writes on both. Quorum comes from this node’s --peer-url list, so give every node the same peer set: an asymmetric configuration gives nodes different quorum sizes, and nothing detects the mismatch.
3. The sort is byte-wise. http://node:10001 sorts ahead of http://node:9001. Comparison trims whitespace and a trailing slash, so a host written as an address on one node and as a DNS name on another counts as two nodes.
4. Leadership returns to the lowest node. When the lowest-sorted node is reachable and fresh again, it leads again, so a flapping lowest node moves leadership on every flap.
5. The term is per node. election_term advances on any change of the computed leader, including a change to no leader, over that node’s own observation history. The authority path reads the leader’s term from the leader’s own status before forwarding.

Membership is set at startup. build_cluster_state builds the peer map once from the configured URLs, and adding a node means restarting every node that must accept it. A node whose --peer-url list resolves only to its own advertised URL runs unclustered.

Read more in Node state fields and Lease TTL.

A follower forwards a covered write and returns the leader’s answer

The receipt-write handler calls forward_post_to_leader before it touches its store. The helper returns either a response to send back or Ok(None), which means handle it here: the node is unclustered, is the leader, or became the leader during a retry. Otherwise the helper checks quorum, then the lease, then the leader URL, and forwards.

A forward makes at most two attempts, the second only when a recomputation names a different leader. update_peer_failure marks the failed leader Unhealthy in this node’s peer map first, which can drop it from the candidate set or the node below quorum.

The leader writes, then re-reads the record once through respond_after_leader_visible_write, and answers 500 if the record is missing. A visible write gets handledBy, leaderUrl, visibleAtLeader, and the clusterAuthority lease block. A follower returns the leader’s body verbatim, so on a forwarded write handledBy equals leaderUrl. Budget writes then wait for peers to acknowledge the event, as Budgets Across Nodes describes.

match forward_post_to_leader(&state, TOOL_RECEIPTS_PATH, &receipt).await {
    Ok(Some(response)) => return response,
    Ok(None) => {}
    Err(response) => return response,
}

Read more in Forwarding helpers and Refusal statuses.

The term fences stale authority writes across failover and restart

Lease windows can overlap a new term, so authority mutation carries the term itself. A forwarded authority write sends it in x-chio-cluster-auth-term, which the peer keyed digest covers.

validate_authority_mutation_auth runs first, before anything is forwarded or written, and refuses a term that does not match the lease with 409. It engages for requests that carry cluster peer headers; a plain bearer request goes to validate_service_auth and carries no term. Then enforce_authority_mutation_fence compares the term with a fence row persisted in the node’s authority database, and answers 409 in three cases:

1. Stale. The term is below the persisted term.
2. Already fenced. The term equals the persisted term under a different leader URL.
3. Another generation. The fence row belongs to a different authority generation; it is refused, never overwritten.

The fence lives in --authority-db. At startup a fence row is restored only when it matches the authority’s current generation and rotation timestamp, and discarded with a warning otherwise, so a rotated key cannot resurrect an old term. A restored leader URL is kept only when it is this node or a configured peer. Without an authority database, only the in-memory lease check applies.

Read more in Authority & Rotation, which covers the keypair the fence protects.

Two intersecting views can each hold quorum

A minority node computes no leader and refuses covered writes with 503. Majorities intersect, yet each node computes over its own candidate set: with A < B < C, A can see A and C while B sees B and C, and both select themselves with a quorum of two. The intersecting-views scenario in the figure draws this case, which follows from the selection code.

Two narrower checks bound what either side commits. The persisted term fence refuses a stale or already-fenced authority mutation, and a budget write fails closed unless the writing node plus eligible peer acknowledgments meet quorum. Neither is a split-brain safety proof. The guarantee class is leader-local: the fence covers authority mutation, other writes are single-writer by observation, and the bounded operational profile makes no realized-spend claim for clustered budgets.

Six streams replicate: budgets, tool receipts, child receipts, lineage, revocations, and the authority snapshot. Replica repair keeps running below quorum, and a handler outside the forwarding helpers writes where it lands; handle_publish_certification keeps its file-backed registry on the node that received the request.

Next: see how a lagging peer catches up