Coordination Contract

Each node routes covered writes to the leader it computes

A node sorts its own URL together with its healthy, unpartitioned, freshly contacted peers. Once that set is a majority of the configured cluster, the lowest-sorted URL is the leader.

Quorum is peers.div_ceil(2) + 1, so a three-node cluster needs two. A lease is three sync intervals, clamped between 500 ms and 5 s, and every computation that holds quorum restamps it. Any change of leader, including a change to no leader, advances that node’s local term.

A node below quorum names no leader and answers a covered write with 503 before its handler writes. Its reads still answer from its own stores, so a read can succeed while the next covered write is refused. After a partition heals, the node selects the majority’s leader again and catches up from a snapshot.

Selection names a route only: whether the leader’s stores are current and whether it holds the active signing seed are separate checks, covered in Replication & Convergence and Authority & Rotation.

Read more in Leader & Failover.

A node joins as a peer, as a client, or stands alone

Membership comes from configuration. What a process is started with decides whether it pulls replication, takes part in leader selection, and counts toward a budget witness.

• Trust-control peer. chio trust serve with --peer-url, --advertise-url, and an authority database. It keeps peer state, pulls replication data, selects a leader from its own view, and can witness a budget event.
• Kernel using remote stores. Started with --control-url and --control-token set to the admin service token, it makes authenticated client requests. It receives no peer deltas and contributes no budget witness.
• Standalone service. With no effective peer URLs, the service uses its local state and runs no cluster runtime.

Each peer URL is normalized, and a node removes its own URL from its peer set. Local or private peer addresses need --allow-local-peer-urls. Clustered authority lives in --authority-db; a peer configuration with --authority-seed-file is rejected at startup.

Read more in Trust Control Plane, which starts a node.

Replication shares state; each node keeps its own rules

Peers replicate the state an admission reads. Policy, guard code, and node configuration load on each node and stay there.

Sharing an authority identity says nothing about which rules each node loaded. Roll policy out with a coordinated policy rollout and check each deployment on its own. The guarantee classes come from CHIO_BOUNDED_OPERATIONAL_PROFILE.

Versioned holds under /v1/budgets/holds/ run on the joint budget-and-revocation authority that --session-db selects, and a peer-URL configuration rejects that flag. On a peer cluster the hold routes answer 503 and the exposure routes carry budgets. Choose the budget architecture before you add peers; Budgets Across Nodes compares the two.

A budget write commits when peers acknowledge its exact event

A clustered budget write shows every part of coordination in four steps.

1. Select. The receiving node computes its leader from its own reachable view, as above.
2. Gate and forward. A leader-forwarded handler refuses with 503 when quorum, the lease, or a leader is missing. The gate covers the handlers that call it.
3. Record and witness. The leader records the mutation, then waits until enough peers acknowledge the write’s origin through a contiguous event sequence.
4. Repair. The sync loop pulls budget history, receipts and lineage, and revocations from every peer.

An acknowledgment head is the last same-origin event before the first gap, so a peer counts only once it holds that event. A peer awaiting a forced snapshot never counts. Pulls take dense pages from the cursor’s successor, up to 64 pages and 200,000 records per peer visit, and revocations get a fresh allowance so budget churn cannot starve them.

On a witness timeout, a new exposure authorization attempts a compensating local reversal. Replayed or already-captured authorizations keep their admission, and capture, release, and reconcile can answer 503 with the local mutation still applied. Treat a failed response as unknown and read the budget write lifecycle before retrying. A successful witness check describes replication of that one event; the clustered guarantee stays advisory_posthoc.

Coordination health is read beside process health

A responding service can have no leader. GET /health reports ok: true on a node that has lost quorum, so read its cluster object too. This node is the partitioned one in the figure.

curl -s http://127.0.0.1:8940/health \
  | jq '{ok, clustered, cluster: .cluster
        | {role, hasQuorum, quorumSize, reachableNodes, leaderUrl, electionTerm}}'

{
  "ok": true,
  "clustered": true,
  "cluster": {
    "role": "candidate",
    "hasQuorum": false,
    "quorumSize": 2,
    "reachableNodes": 1,
    "leaderUrl": null,
    "electionTerm": 2
  }
}

Alert on hasQuorum, role, and leaderUrl, gated on clustered: a healthy standalone service also reports hasQuorum: false.

A reachable peer can still be ineligible to witness. When budget writes time out with every peer reachable, read forceSnapshot, snapshotAppliedCount, and the acknowledgment heads on the authenticated /v1/internal/cluster/status route.

Nodes can disagree for a while, inside one operator’s domain

Coordination keeps each node’s records and repairs replicated state under the contracts above. There is no single global transaction history: for a while, nodes can disagree about leadership, revocation visibility, or budget exposure, and receipt replication leaves each node’s checkpoints local.

Until a merge propagates, exposure admitted independently on each node is bounded by overrun <= max_cost_per_invocation × node_count. The bound is arithmetic over a worked scenario and covers admitted exposure; realized external spend is a separate quantity. Read its assumptions in Budgets Across Nodes before planning capacity with it.

A cluster serves one operator’s trust domain. Work between independently operated identities belongs to Swarms and Federation.

Next: watch a leader change hands