Cluster Peer Auth
Configure which trust-control nodes replicate with each other, and see how a receiver checks each internal request before its handler runs.
On this page
A trust-control node opens its replicated stores to an internal request that names a configured node, carries a digest keyed with the cluster’s shared service token, and lands within 60 seconds of the receiver’s clock. Anything else is refused before a handler runs.
The check guards the nine internal replication routes and the two authority mutation routes, where a forwarded write also carries the leader’s lease term. Trust between operators belongs to Federation, and tenant reads use the bearer path on Tenancy & Read Scoping.
The receiver runs five checks, in a fixed order, before any handler
Node A parses the headers, matches the claimed identity against its peer list, recomputes the digest, consults the failure throttle, and checks the clock. Each stage can refuse on its own.
A listed node with a fresh digest reaches the handler
Node B names itself by its advertised URL, binds the route and the time with the shared service token, and arrives inside the 60-second window. Node A clears B’s verified-failure history and the handler reads the replicated stores. Replication pulls carry no term, so the term check is skipped. Replication & Convergence.
The digest comparison uses subtle::ConstantTimeEq over two 64-character hex strings, so length reveals nothing. The allowlist runs first: an unknown identity gets 403 before any digest is computed, which tells a caller without the secret whether an identity is listed. Acceptance clears the identity’s verified-failure history, and the handler then applies its own state and operation checks.
The operator bearer token that opens operator routes does not open an internal one. Ask node A for its cluster status with the token alone and the peer check refuses before the handler reads any state:
curl -i -H "Authorization: Bearer $CHIO_TRUST_SERVICE_TOKEN" \
https://kernel-a.larkfield.example:8940/v1/internal/cluster/statusHTTP/1.1 401 Unauthorized www-authenticate: chio.cluster.peer.v1 missing or invalid cluster peer authentication
Read more in Failure responses.
The digest binds the node, the route, the time, and the term
The client canonicalizes a JSON object that includes the service token as one field and sends its SHA-256 hex digest. The receiver rebuilds the same object from its own configuration and the received headers.
pub(crate) fn cluster_peer_auth_signature(
service_token: &str,
node_id: &str,
endpoint: &str,
issued_at: i64,
term: Option<u64>,
) -> Result<String, CliError> {
let payload = canonical_json_bytes(&json!({
"scheme": CLUSTER_AUTH_SCHEME,
"serviceToken": service_token,
"nodeId": node_id,
"endpoint": endpoint,
"issuedAt": issued_at,
"term": term,
}))
.map_err(|error| {
CliError::cli_other_error(format!(
"failed to encode cluster peer auth payload: {error}"
))
})?;
Ok(sha256_hex(&payload))
}This is a keyed digest, distinct from an HMAC construction or a public-key signature. It covers the route constant, so a digest for the status route fails on the budgets route; the method, body, query string, and channel sit outside it. Protect the whole request with TLS at the reverse proxy in front of each node.
The receiver accepts a timestamp within 60 seconds either side of its current second, both boundaries included, and keeps no nonce or replay cache. A captured header set therefore authenticates again until its window closes; adding a term changes the digest and still leaves it reusable inside the window.
A forwarded authority write must carry the receiver’s lease term
POST /v1/authority and POST /v1/capabilities/issue accept an operator bearer credential or the peer path. Any one peer header selects the peer path, so a stray term header moves an otherwise valid bearer request onto it. After authentication, the receiver checks the term against its own lease:
| Condition | Status | Message |
|---|---|---|
| No parsed term | 401 | cluster authority mutation is missing the forwarded term |
| No local lease | 503 | cluster authority lease is unavailable for authority mutation |
| Lease invalid | 503 | cluster authority lease expired before authority mutation |
| Term differs | 409 | cluster authority mutation term does not match the current lease |
A lease is valid only while the receiver’s local view holds quorum. These checks compare the request with that node’s own lease and term; the forwarding that follows, and the persisted fence that refuses a stale term across restart, are on Leader & Failover.
Membership is the peer list each node starts with
Give every node the same service token and list the other nodes by the exact URL each one advertises. This is node B of three:
export CHIO_TRUST_SERVICE_TOKEN="$(cat /run/secrets/chio_service)"
chio trust serve \
--listen 0.0.0.0:8940 \
--advertise-url https://kernel-b.larkfield.example:8940 \
--peer-url https://kernel-a.larkfield.example:8940 \
--peer-url https://kernel-c.larkfield.example:8940 \
--authority-db /var/lib/chio/authority.sqliteThe receiver trims whitespace and trailing slashes from --advertise-url and --peer-url, then compares strings, so https://kernel-b.larkfield.example:443 and https://kernel-b.larkfield.example are different identities. Adding a node means restarting the nodes whose peer lists change. --allow-local-peer-urls admits private and loopback addresses, and --authority-db holds the replicated authority and its persisted fence; a peer started with --authority-seed-file is refused.
Read more in Trust Control Plane, which covers the full command, stores, and the TLS reverse-proxy recipe.
An authenticated peer holds operator-level access to the cluster
The digest proves the request was built with the shared token for a listed identity. The same token is the operator’s bearer credential on operator routes, so any holder of it can claim any listed identity; the capability authority key plays no part, and no per-node certificate ties a request to the machine that sent it.
An authenticated peer reads every replicated store, with no tenant scope or per-route peer permission, and can drive the receiver’s partition control, so keep peers and the token inside the operator boundary that holds the capability authority.
Headers, routes, and responses
Try x-chio-cluster-auth-term, 429, or snapshot.
Peer headers
The built-in peer client builds these headers from its configured node identity and the shared token. Authorization: Bearer … alone fails the peer check on an internal route.
x-chio-cluster-node-id
The caller’s claimed identity: its advertised base URL. The receiver trims whitespace and all trailing slashes, then compares the result with its configured peers.
Examplehttps://kernel-b.larkfield.example:8940
x-chio-cluster-auth-issued-at
Signed Unix timestamp in seconds. Both sides use wall-clock time.
x-chio-cluster-auth-signature
The digest. The receiver compares the supplied text with its recomputed hex string as text, without decoding or normalizing it.
x-chio-cluster-auth-term
Optional on the general peer contract. The built-in replication pulls omit it. A peer-authenticated authority mutation requires it.
An absent or non-text node id, timestamp, or digest is refused. An empty or whitespace-only digest is refused before comparison, and other surrounding whitespace makes the digest mismatch. A textual term that does not parse as an unsigned integer gets invalid cluster peer term header; a non-text term reads as absent, though its presence still selects peer mode on the authority routes, which then require a parsed term.
Peer-authenticated routes
GET /v1/internal/cluster/status: the receiver’s local leader view, lease, peer health, replication heads, and budget acknowledgment heads.GET /v1/internal/cluster/snapshot: a full snapshot of the configured replicated stores.POST /v1/internal/cluster/partition: local fault-injection controls. Changing the blocked-peer set can remove the receiver’s quorum; clearing blocks requests snapshot repair.GET /v1/internal/authority/snapshot: authority public key state, generation, rotation time, and trusted-key history, with no signing seed.GET /v1/internal/revocations/delta: revocation records under the negotiated cursor contract.GET /v1/internal/receipts/tools/delta: tool receipts after a sequence number.GET /v1/internal/receipts/children/delta: child receipts after a sequence number.GET /v1/internal/budgets/delta: budget mutation events and their cursor contract.GET /v1/internal/lineage/delta: capability lineage records after a sequence number.
These nine handlers authenticate the peer before they read cluster state or open a store. Delta handlers clamp the requested row limit to 1 to 200, and snapshot and store availability rules apply after authentication. Query and JSON extractors run before the handler body, so a malformed query or partition body can fail with an extractor error before authentication runs; a status other than 401 does not show that a credential was accepted. The /v1/internal/admission-authority route takes bearer authentication and belongs to joint-authority mode, which cannot run alongside configured peers.
Digest payload fields
scheme:chio.cluster.peer.v1.serviceToken: the shared service token, which never travels as a peer header.nodeId: the claimed, normalized peer identity.endpoint: the route constant, without query parameters.issuedAt: the signed Unix timestamp in seconds.term: the supplied term, or JSONnullwhen absent.
Failure responses
| Status | Diagnostic | Check |
|---|---|---|
401 | missing or invalid cluster peer authentication | Required headers and digest agreement |
401 | invalid cluster peer term header | The term parses as an unsigned 64-bit integer |
403 | cluster peer is not in the configured allowlist | The receiver’s peer list and the exact normalized spelling |
429 | Repeated invalid digests | The unverified identity- |
429 | Repeated verified failures | The identity-wide bucket, filled by skew failures |
401 | Timestamp in the future or outside the skew window | Clock agreement and when the headers were built |
With well-formed headers, an unknown identity gets 403, and a listed identity with a wrong digest gets 401 or the unverified-bucket 429. An empty peer list refuses every identity at the allowlist. The generic 401 carries WWW-Authenticate: chio.cluster.peer.v1; the term, skew, allowlist, and throttle responses carry no challenge, and both 429 responses carry Retry-After: 60. Only the digest comparison runs in constant time; the rest of the handler and its error responses do not. Compare the configured URL strings before reading a 403 as a credential failure.
Failure buckets
Two buckets keep a garbage digest that names a listed peer from spending that peer’s verified-failure budget. Each keeps 60 seconds of history with a burst threshold of eight, on different branches:
- Unverified. Keyed
unverified:sha256(node_id \0 endpoint)and consulted only after a digest mismatch. Eight recorded mismatches make the next mismatch get 429; a valid digest bypasses this bucket. - Verified. Keyed by the normalized node identity across endpoints and filled by skew failures after a digest match. When full, it returns 429 before checking even a corrected timestamp.
An entry exactly at now − 60 stays in the history; entries before it are pruned when the history is read, and a full bucket returns without adding a timestamp. Success clears the verified bucket only. The counters are process-local and reset on restart. The digest is computed before the mismatch throttle, so the buckets limit failure responses, and a poisoned limiter mutex skips throttling while the digest and timestamp checks stay in force.
Peer URL validation
Configured URLs use HTTP or HTTPS and carry no credentials, query string, or fragment. Without --allow-local-peer-urls, IPv4 checks refuse private, loopback, link-local, multicast, unspecified, and shared-address-space values; IPv6 checks refuse loopback, unspecified, unique-local, multicast, and link-local values, and mapped IPv4 addresses follow the IPv4 rules. Localhost names are refused, and every DNS answer is checked at startup, once. Validation parses each URL and keeps its trimmed text.
The default advertised URL comes from the bound local address, so a wildcard listener with local addresses disallowed can fail validation; pass the reachable identity explicitly. The service binds a plain TCP listener, so an advertised HTTPS URL relies on the reverse proxy for TLS. Joint-authority storage cannot be configured alongside peers. A peer with neither --authority-db nor --authority-seed-file starts, and its authority snapshot and issuance operations then fail for missing authority storage. Use CHIO_TRUST_SERVICE_TOKEN to keep the token out of process arguments.
Implementation and tests
crates/platform/chio-control-plane/src/trust_control/report_validation.rs: digest, header parsing, allowlist, failure buckets, and the authority-auth selector.crates/platform/chio-control-plane/src/trust_control/service_runtime/client/auth.rs: the built-in GET and POST peer headers.crates/platform/chio-control-plane/src/trust_control/service_types/paths.rs: header names, scheme, and limiter constants.auth_helpers_and_metered_billing_validation_cover_error_pathsincluster_and_reports.rscovers matching and mismatching digests, an expired timestamp, and success after spoofed failures fill the unverified bucket. The cluster scenario incrates/products/chio-cli/tests/trust_cluster.rsrefuses a stale forwarded term without changing the authority generation.