Cluster Overview
Run several trust-control nodes as one operator's control service, and know what they share, who accepts a write, and what a partition refuses.
On this page
A revocation written on one node protects nothing until every other node honors it. A cluster gives one operator that guarantee: its trust-control nodes select a leader from the peers each can reach, route covered writes to it, and repair one another's stores, so authority, revocations, receipts, and budget exposure converge across nodes that each keep their own database.
The lowest-sorted node a quorum can see leads
Each node counts itself and the peers it reached within the lease TTL, then selects the lowest-sorted URL once that set is a majority of the configured cluster. All three nodes see each other, so all three select A at term 1. Every computation that holds quorum restamps the node’s lease to three sync intervals ahead. Leader & Failover.
The guarantee class is leader-local: deterministic leader selection, single-writer local truth, and eventual repair. Kernels reach the shared state as clients of the control service; only the trust-control nodes are peers.
Each node routes covered writes to the leader it computes
A node counts itself and the peers it reached within the lease, and once that set is a majority of the configured cluster, the lowest-sorted URL leads.
Quorum is peers.div_ceil(2) + 1, so three nodes need two. A lease lasts three sync intervals, clamped between 500 ms and 5 s, and any change of leader advances that node's local term. A node below quorum names no leader and answers a covered write with 503 before its handler writes.
Because a lapsing lease can overlap a new term, a forwarded authority write carries its term in x-chio-cluster-auth-term. A receiver refuses a stale term with 409 and persists the fence (term, leader URL, authority generation), so a restart keeps the old term closed.
Read more in Coordination Contract and Leader & Failover.
Configuration decides whether a process is a peer or a client
A kernel started with --control-url and --control-token is a client of the control service: it receives no peer deltas and witnesses no budget event. A trust-control service started with --peer-url, --advertise-url, and an authority database is a peer that pulls replication and takes part in leader selection. With no effective peer URLs, the service stands alone.
Start with one durable control service and connect the runtimes that need its authority, revocation, budget, and receipt APIs. Add peer URLs when that service itself needs replication.
Read more in Trust Control Plane, which configures credentials, databases, and peers.
A responding node can still lack quorum
GET /health separates process health from cluster health. This node answers requests while partitioned from both peers.
curl -s http://127.0.0.1:8940/health \
| jq '{ok, clustered, cluster: .cluster
| {role, hasQuorum, quorumSize, reachableNodes, leaderUrl, electionTerm}}'{
"ok": true,
"clustered": true,
"cluster": {
"role": "candidate",
"hasQuorum": false,
"quorumSize": 2,
"reachableNodes": 1,
"leaderUrl": null,
"electionTerm": 2
}
}Covered writes wait for quorum and an eligible leader, so alert on hasQuorum, role, and leaderUrl when clustered is true; a standalone service also reports no quorum. A failed budget write needs its own reading, because some operations mutate local state before a witness timeout. The write lifecycle names when a failure can leave state applied.
Add peers when the control service needs replication
One node with local durable stores is a complete deployment, and several execution nodes can share one control service without becoming peers. The peer coordinator provides leader-local writes and eventual repair; it makes no consensus commit and serves no globally linearizable read. Clustered budget exposure is advisory_posthoc, with no realized-spend ceiling during a split brain. The joint session-database hold lifecycle is a separate configuration, and a peer-URL configuration rejects --session-db.
Replication carries public authority snapshots, never private signing keys, and policy files and guard modules roll out on their own. Independently controlled operators share no database: they start from Federation.
Every Cluster page, by what it keeps consistent
- Cluster Topology. Coordination Contract states which state nodes share, Trust Control Plane runs the service, Leader & Failover follows writes through a failover, and Replication & Convergence follows a returning peer as it catches up.
- Membership & Identity. Cluster Peer Auth checks each internal request, Authority & Rotation rotates the capability authority while its key history replicates, Tenancy & Read Scoping scopes each tenant's reads, and Runtime Assurance Tiers ties a grant to its runtime evidence.
- Shared State. Budgets Across Nodes reads the quorum commit on each admission, Revocation Propagation carries a revocation to every node's revoked set, and Receipt Aggregation finds any node's receipts from one store.
- Fleet Evidence. Evidence Export checks receipts offline against keys you trust, Operator Reports traces a report to its receipts, and Receipt Dashboard traces any call to its capability chain.
- Fleet Operations. Kubernetes Admission admits pods by their capability token, Capability Leases binds a destructive step to its lease, Policy Rollout releases a policy across nodes, Cluster Recovery brings a peer back, and Cluster Upgrade replaces binaries with a path back.