Cluster Overview

Four claims, before anything else

A cluster in Chio is many nodes under one authority: one signing keypair, one admin service token, one shared-secret peer credential, one trust domain. It is a deterministic leader plus a repair loop. It is not a consensus protocol, and four claims fix the ceiling before the mechanism is worth reading. The first three are rows of docs/standards/CHIO_BOUNDED_OPERATIONAL_PROFILE.md, the profile the release is allowed to claim. The fourth has no row there because there is nothing to bound.

The last row is the one operators misread. The internal replication surface is exactly 10 routes, and this is all of them:

No policy delta, no guard module distribution, no chio.yaml sync. Policy is enforced by whoever holds the authority, never shipped to a member: configure_capability_authority refuses a reputation or runtime-assurance issuance policy on the --control-url path and tells you to run chio trust serve --policy instead. Nothing checks that the file two nodes loaded is the same file: the health body reports issuancePolicyConfigured and runtimeAssurancePolicyConfigured as booleans, never a digest, and load_policy sets both to None outright on the plain Chio-YAML path, so only a HushSpec policy file populates them at all. Policy Rollout works that row through in full: every negative verified against the code, and what an operator does in place of the mechanism.

The cluster model

The filing test for this rung: does answering the question require another node’s disk under the same key? If one process can answer it by reading its own, it is Node. If it needs a signature from a party under a different key, it is Swarm. Four objects carry everything in between.

The peer credential is derived from the admin service token, which is what makes the trust domain singular rather than merely coordinated. Membership is the admin token plus a node id that normalizes to an entry in peer_urls. The node id is allowlisted first, the digest is compared in constant time, and only then are timestamps outside a 60-second skew window rejected. Invalid-signature failures are counted per node-id-and-endpoint pair under an unverified: bucket, 8 in 60 seconds, answered 429 with Retry-After: 60; skew failures after a valid digest are counted separately under the node id. Two consequences follow from the digest being derived from a shared secret rather than a per-node key. A peer cannot prove authorship of anything to a third party, and any holder of the admin token can mint valid headers for any allowlisted node id, including /v1/internal/cluster/partition, which is a live route in the shipped router and will partition a running cluster on request.

Two ways to cross into this rung

Add peer URLs. Repeated --peer-url on chio trust serve, with --advertise-url naming this node. URLs are normalized once at boot; loopback and private hosts are refused without --allow-local-peer-urls, and the node’s own URL is filtered out of its own peer set. A peer list that normalizes down to nothing is still a standalone node.

Borrow remote stores. --control-url with a required --control-token swaps a kernel’s local stores for HTTP-backed adapters: RemoteReceiptStore, RemoteRevocationStore, RemoteBudgetStore, RemoteCapabilityAuthority. Each store is local or borrowed, never both: every configure_* function errors on a database path and a control URL together. The flag takes a comma-separated endpoint list, and every request carries a 15-second client timeout.

How a clustered write lands

1. Leader selection, recomputed per status check. compute_cluster_consensus_locked builds a candidate list of this node plus every peer that is healthy, unpartitioned, and contacted inside the lease TTL, sorts it, and takes the first. Quorum size is peers.len().div_ceil(2) + 1, so a two-node cluster needs both. Without quorum there is no leader and the role is candidate. A change of leader increments the election term. There is no term voting, no log replication, no election message of any kind.
2. The lease. TTL is three sync intervals clamped to the range 500 ms through 5 s, with a lease id of {leader_url}#term-{epoch}. The freshness test truncates that TTL to whole seconds with a floor of one, so at the default --cluster-sync-interval-ms 500 the lease is 1.5 s but a peer drops out of the candidate set after one second without contact. The term is persisted in the authority store and adopted on boot only when the recorded generation and rotation timestamp still match the live authority; a rotated key discards the fence rather than carrying a lease across it.
3. Forwarding, which refuses early. forward_post_to_leader answers 503 before it opens a connection if quorum is missing, the lease is unavailable, the lease expired, or no leader is named. It makes at most two attempts, and the second only against a leader that changed; re-posting to the same failed URL is treated as terminal.
4. Budget writes wait on witnesses. A peer counts toward a budget write only when its contiguous acknowledgment head for that write’s origin has reached the write’s event_seq, and the peer is reachable, unpartitioned, and not pending a forced snapshot. The wait is bounded by a timeout that scales with peer count and is capped at 300 s; on expiry the handler rolls the local exposure back and answers 503, except on a replayed or already-captured write, where the admission is retained and only the 503 is returned.
5. Repair, pulled not pushed. run_cluster_sync_loop visits peers serially. Budgets are pulled first because they are quorum-critical, then tool receipts, child receipts, and lineage share the remaining round budget; revocations run on an independent budget so budget churn cannot starve revocation propagation. Each of those two lanes gets its own cap of 64 pages, 200,000 records, and 20 seconds of wall clock, and any single peer response body is refused past 64 MiB.
6. Contract violations demote. A page that rewinds past the cursor, repeats a sequence, breaks contiguity on a dense stream, or ships budget usage records with no mutation events drops the peer to unhealthy and force-resyncs it from a full snapshot. A delta window too large to page at all is a separate outcome: it flags the snapshot without demoting, because that is backlog rather than misbehavior.

Three failure domains node health cannot report

The first two live in the trust-control service and are visible only in the body of its own GET /health, never in its status code. The third lives in a kernel that borrowed its stores, one process removed from the sidecar routes GET /chio/live and GET /chio/health, which keep answering 200 while that node’s own receipt table is writable. Per-node probes will not show you any of them.

1. A leaderless cluster reports ok: true

The trust-control GET /health route is unauthenticated and returns "ok": true unconditionally. The authority and federation-registry snapshots degrade to available: false inside the body rather than failing the request, deliberately: a broken certification registry file should not hide the cluster verdict. The consequence is that quorum loss never changes the status code. It changes cluster.hasQuorum, cluster.role, and cluster.leaderUrl, and the write paths that forward to a leader start answering 503 with a message naming quorum, the lease, or the missing leader. That set is a subset of the mutating routes, not all of them: roughly a third of the POST routes call forward_post_to_leader, and the rest, credit bond issuance among them, keep writing to the local SQLite file and returning 200 with no quorum gate at all.

2. A reachable peer that cannot witness

Reachability and eligibility are different properties. PeerSyncState::default() sets health: Unknown and force_snapshot: true, and is_reachable() matches only Healthy. A freshly booted clustered node therefore has one reachable node, no quorum, role candidate, and every leader-forwarded write refused until the first sync round reaches a peer. The same flag outlives recovery: any peer whose cluster_status call failed, not only one that violated the wire contract, is marked unhealthy and force-snapshot, and a bare reachability probe flips it back to healthy without clearing that flag. Only a full clean round does. The cluster then reports healthy peers while budget writes time out on the quorum wait, because the witness test excludes forced-snapshot peers by design. The distinguishing fields, forceSnapshot, snapshotAppliedCount, and deltaRecordsSinceSnapshot, appear only on the authenticated /v1/internal/cluster/status endpoint, not in the public health body.

3. A borrowing node whose control plane stalls

A --control-url node has moved its receipt writes, revocation lookups, and capability issuance onto the network. Its sidecar readiness probe does not follow: it opens a transaction against the sidecar’s own local SQLite receipt tables and rolls it back, and it reports healthy without probing anything at all when no such store is attached. Two mechanisms keep the borrowing node fail-closed instead of merely slow.

Bounded remote writes. BoundedReceiptWriter runs blocking control-plane writes on a fixed pool of 2 worker threads behind a depth-2 queue. Once every worker is parked and the queue is full, further submissions return ReceiptStoreError::Timeout immediately rather than spawning a thread per write. A write that outlives the caller’s budget still completes, with its result discarded, because the caller already failed closed. Only the budgeted overrides route through that pool: the plain append_chio_receipt, append_child_receipt, and record_capability_snapshot calls, and every revocation lookup and receipt point-load, block on the client’s own 15-second timeout with no budget at all.

A denial rather than a panic. AuthorityKeyCache refreshes against the service on a 2-second TTL, and from_status rejects any status with no current key, so a primed cache always carries one. If that invariant is ever violated, the code must neither abort the process nor return a key an attacker could influence:

/// Fail-closed substitute for a missing current authority key.
fn deny_sentinel_public_key() -> PublicKey {
    tracing::error!(
        "remote capability authority cache missing current key; \
         returning a non-trusting sentinel so admission fails closed"
    );
    Keypair::generate().public_key()
}

The returned key is freshly generated and its private half is discarded on the spot, so it can never validate a real capability and a caller that folds it into a trust set gains no usable issuer. An unprimed cache produces a guaranteed-wrong denial, a verdict an operator can read in a receipt, instead of a crash or a silent admission. The error log line is the signal to alert on; nothing here changes a health status code.

Guarantees and limits

Claim: the budget overrun bound

Next Steps

• Leader & Failover · the deterministic pick, the lease, and what a write does when there is no leader
• Replication & Convergence · the pull loop, the round budgets, and what costs a peer its witness standing
• Cluster Peer Auth · the digest, the allowlist, and the skew window in full
• Budgets Across Nodes · quorum witnessing, compensating rollback, and the overrun bound as arithmetic
• Authority & Rotation · one keypair per cluster, its snapshot, and the custody it refuses to copy
• Health & Readiness · the counterpart page, and what one process can answer about itself
• Trust Control Plane · the operational depth: topology, running a node, failover, key rotation
• Swarm Overview · the next rung, where authority descends to parties you do not run