Leader & Failover

Not an election

The control plane’s own architecture record states the mechanism and its non-claim in the same breath:

1. `cluster::consensus::compute_cluster_consensus_locked` recomputes the
   leader on every status check: the lowest-sorted URL among peers that are
   reachable, unpartitioned, and inside the lease TTL. This is a deterministic
   pick over the locally observed peer set, not a voted election - there is no
   term voting, log replication, or election RPC.

One correction to that record, because the code is the authority: the candidate set is not "peers", it is self plus qualifying peers. A node is always in its own set and its own health is never tested. Otherwise read it as a description of the whole subsystem, not a summary of one. There is no election timer, no candidacy message, no vote request, and no append-entries stream anywhere in crates/platform/chio-control-plane/src/trust_control/cluster/. The function is reached only from view helpers (cluster_consensus_view, cluster_authority_lease_view, the combined snapshot the internal status route uses, and budget_write_quorum_commit_view), so the leader exists only as the answer to a question somebody just asked.

It takes &mut ClusterRuntimeState because computing the view writes the term, the last leader, and the lease expiry back into memory. A term advance is therefore a side effect of a read. In a running cluster that read is never scarce: each clustered node pulls every peer’s GET /v1/internal/cluster/status once per sync round, answering that poll makes the polled node recompute, and so does every GET /health and every write that checks whether it should forward. The sync interval is not a guaranteed cadence in either direction: it is the sleep between rounds, a round visits peers serially with each call bounded by the control HTTP timeout, and a parked budget write can kick the next round early.

The state one node keeps

ClusterRuntimeState is built once at startup by build_cluster_state, and only when at least one --peer-url resolves to something other than this node’s own advertised URL. Without that, the node is unclustered and every mechanism below is inert. Every URL passes normalize_cluster_config_url, which requires an http or https scheme, rejects userinfo, a query string, and a fragment, and rejects loopback or private-network addresses unless --allow-local-peer-urls is set. Configuring peers alongside --authority-seed-file is rejected outright: a cluster wants the SQLite authority behind --authority-db, because that is where the term fence is persisted. With --authority-db set, a fence row found at startup is restored only when it still matches the authority’s current generation and rotation timestamp, and otherwise discarded with a warning, so a rotated key cannot resurrect an old term. A restored leader URL is kept only if it is this node or a configured peer.

A peer joins the candidate set only when three independent predicates hold. All three are fed by the replication round, not by this function:

How the pick is computed

let now = unix_timestamp_now();
let lease_ttl_secs = Duration::from_millis(cluster.lease_ttl_ms).as_secs().max(1);
let quorum_size = cluster.peers.len().div_ceil(2) + 1;
let mut candidates = vec![cluster.self_url.clone()];
for (peer_url, peer_state) in &cluster.peers {
    let contact_is_fresh = peer_state
        .last_contact_at
        .is_some_and(|last_contact_at| now <= last_contact_at.saturating_add(lease_ttl_secs));
    if peer_state.health.is_reachable() && !peer_state.partitioned && contact_is_fresh {
        candidates.push(peer_url.clone());
    }
}
candidates.sort();
let reachable_nodes = candidates.len();
let has_quorum = reachable_nodes >= quorum_size;
let leader_url = if has_quorum {
    candidates.first().cloned()
} else {
    None
};
if cluster.last_leader_url != leader_url {
    cluster.election_term = cluster.election_term.saturating_add(1);
    cluster.last_leader_url = leader_url.clone();
    cluster.term_started_at = leader_url.as_ref().map(|_| now);
}

Four consequences follow directly from those lines. First, a node always counts itself: candidates starts with self_url and nothing tests local health, so reachable_nodes is never zero. Second, quorum_size is the majority of the configured cluster: two peers means a quorum of two, four peers means three. A two-node cluster has a quorum of two, so either node going quiet stops writes on both. It is computed from this node’s --peer-url list, so an asymmetric configuration gives nodes different quorum sizes and nothing detects the disagreement.

Third, candidates.sort() is a byte-wise sort of URL strings, not a numeric one, so http://node:10001 sorts ahead of http://node:9001. The only normalization applied to a peer URL is a trim and a trailing-slash strip, so a host written as an address on one node and as a DNS name on another is two different nodes to this function. Fourth, the term advances on any change of the computed leader, including a change to no leader at all: the unit test compute_cluster_consensus_tracks_role_quorum_and_election_terms walks a three-node state from no quorum, to leader at term 1, to lost quorum at term 2.

The lease is a local stamp, not a grant

Nothing issues the lease. The same computation that names a leader restamps lease_expires_at to now + lease_ttl_secs whenever quorum holds, and a lease is valid when quorum holds and that stamp has not passed. The TTL itself is fixed at startup from the sync interval:

pub(crate) fn authority_lease_ttl(sync_interval: Duration) -> Duration {
    let scaled = sync_interval
        .checked_mul(3)
        .unwrap_or_else(|| Duration::from_secs(5));
    scaled
        .max(Duration::from_millis(500))
        .min(Duration::from_secs(5))
}

The third column is not a rounding of the second. lease_ttl_secs truncates the millisecond TTL to whole seconds and floors the result at one, so a 1500ms lease tolerates one second of peer silence, not one and a half. Reported lease metadata stays in milliseconds. Tune the interval, read the window in seconds. The interval itself is floored at 50ms by the CLI before it reaches this function, and TrustServiceConfig::validate rejects a zero interval outright.

Where a write goes

Every cluster-aware mutating handler opens the same way, before it touches a store:

match forward_post_to_leader(&state, TOOL_RECEIPTS_PATH, &receipt).await {
    Ok(Some(response)) => return response,
    Ok(None) => {}
    Err(response) => return response,
}

Ok(None) means "handle it here", returned when the node is unclustered, is itself the leader, or becomes the leader partway through a retry. Otherwise the helper checks quorum, then lease validity, then the leader URL, and forwards. It makes at most two attempts, and the second only when a recomputation after the failure names a different leader; a failure against a leader the node still computes as leader is returned to the caller rather than retried into the same wall. What makes that recomputation move at all is the failure itself: update_peer_failure demotes the leader to Unhealthy in this node’s own peer map before the next computation runs, which can drop it from the candidate set or drop the node below quorum entirely.

The leader then writes locally and verifies before it answers. respond_after_leader_visible_write runs a caller-supplied closure that re-reads the record it just wrote, and returns 500 with a message such as tool receipt was not visible on the leader after write when the read comes back empty. It is one re-read, not a poll: a write that is not immediately readable on the node that performed it is reported as a failure rather than waited out. Only after that check does json_response_with_leader_visibility add the cluster metadata: handledBy, leaderUrl, visibleAtLeader, and the clusterAuthority lease block. Because the follower returns the leader’s body verbatim, handledBy equals leaderUrl on a forwarded write, which is exactly what the integration suite asserts in assert_write_visibility_metadata.

That metadata is not universal. Only responses built through that helper carry it, and two clustered write paths do not use it: /scim/v2/Users answers through scim_json_response, and /v1/capabilities/issue returns a plain IssueCapabilityResponse. Both forward correctly; neither reports handledBy, and neither runs a leader-visible re-read. Do not treat the absence of visibleAtLeader as evidence a write stayed local.

Four helpers, three different checks

Read the last column twice. Only forward_post_to_leader maps a status error back into a response and returns it; the other three go through the shared client, which turns any non-2xx into an error, marks the leader unhealthy, and reports 503. A leader that is up and deliberately rejecting a request looks like a dead leader on those three paths. The SCIM pair also skips lease validity and re-wraps the leader’s JSON as a fresh 201 or 200 with SCIM error shaping rather than passing the response through. Budget writes go the other way: after the leader-visible write they park on wait_for_budget_write_quorum_commit until a majority of peers have acknowledged the event, under a wall-clock bound derived from the sync interval and peer count. A timeout, or quorum disappearing mid-wait, fails the write closed with 503 after it is already durable on the leader. That is Budgets Across Nodes.

What a write returns when it cannot proceed

Those strings belong to forward_post_to_leader. The authority helper carries a parallel set with authority writes substituted for trust-control writes and failed to forward authority write to leader for the forwarding failure. Match on the status code, not on the sentence.

The one path that fences on term

Authority mutation is the exception to everything above. A forwarded authority write carries the term in x-chio-cluster-auth-term, which the peer keyed digest covers rather than merely accompanies. validate_authority_mutation_auth runs first, before anything is forwarded or written, and rejects a mismatch with 409. It engages only when a request arrives with cluster peer headers; a plain bearer request falls through to validate_service_auth and carries no term at all. Past that, enforce_authority_mutation_fence checks the term against a row persisted in the node’s own authority database: a term below the persisted one is stale, a term equal to the persisted one under a different leader URL is already fenced, and a fence row recorded against a different authority generation is refused rather than overwritten. All three map to 409. All three need --authority-db; with no authority database configured the fence step is a no-op and only the in-memory lease check applies.

trust_control_cluster_rejects_stale_authority_term_after_failover_and_restart exercises the whole path on three live processes: it kills the leader, waits for the survivors to converge on a higher term, restarts the killed node, replays an authority mutation carrying the pre-failover term, and asserts a 409 with an unchanged authority generation. It then posts the same mutation to a follower and asserts the generation advances by exactly one with handledBy naming the leader.

Guarantees and limits

Claim: a minority partition cannot accept writes

Next Steps

• Replication & Convergence · the pull round that produces every peer observation this page reads, the partition gate, and the snapshot backstop a late joiner heals through
• Authority & Rotation · the one keypair the term fence protects, and how a follower takes over
• Budgets Across Nodes · quorum-commit witnessing, and the overrun bound stated as arithmetic
• Health & Readiness · what one node can answer about itself before any of this applies