Cluster Peer Auth

One shared secret, no per-node key

The key is literal. Every node is started with the same service token, --service-token or, as the flag help itself recommends, CHIO_TRUST_SERVICE_TOKEN so the secret is not readable from ps. That one secret is both the admin bearer credential and the key the peer digest is computed under.

Say the shape of it plainly before the mechanism. This is a shared-secret message authentication code, not a per-node public key. There is no peer keypair, no certificate, no mutual TLS identity. Any holder of the service token can mint a valid header for any allowlisted node id, so the node id is checked for membership, never proven as origin. Membership itself is static configuration: peers are the --peer-url flags the process was started with. There is no enrollment protocol, no join token, and nothing anywhere in crates/ that adds or removes a peer while the service is running.

What is on the wire

Four headers, all defined in trust_control/service_types/paths.rs alongside the route constants they authenticate. The client half is TrustControlClient::build_internal_get_request and its POST twin; the server half is report_validation::validate_cluster_peer_auth.

The scheme string chio.cluster.peer.v1 is not a header of its own. It is a field of the signed payload, and it is the whole value of WWW-Authenticate on one specific rejection: the generic cluster_peer_auth_error() 401. The 403, the two 429s, the term-parse 401, and both skew 401s carry no challenge header at all.

Which endpoints require it

Nine routes call validate_cluster_peer_auth as the first statement of their handler, before touching cluster state or opening a store. Nothing beyond that call gates what an authenticated peer may read: there is no per-route scope and no tenant narrowing. The only bound on a single request is page size, which list_limit clamps to MAX_LIST_LIMIT (200) on the five delta routes.

Two public routes accept either credential. POST /v1/authority and POST /v1/capabilities/issue run through validate_authority_mutation_auth, which switches into peer mode when any one of the four headers is present and falls back to the admin bearer token otherwise. The selector is header presence, not header validity, so a bearer request that carries a stray x-chio-cluster-auth-term is routed into peer mode and fails 401 with the bearer token ignored. One further internal route, /v1/internal/admission-authority, is bearer only and is not part of this mechanism: it serves the joint admission authority, which TrustServiceConfig::validate refuses to configure alongside peers and which the handler refuses again at request time when a cluster is live.

The digest

pub(crate) fn cluster_peer_auth_signature(
    service_token: &str,
    node_id: &str,
    endpoint: &str,
    issued_at: i64,
    term: Option<u64>,
) -> Result<String, CliError> {
    let payload = canonical_json_bytes(&json!({
        "scheme": CLUSTER_AUTH_SCHEME,
        "serviceToken": service_token,
        "nodeId": node_id,
        "endpoint": endpoint,
        "issuedAt": issued_at,
        "term": term,
    }))
    .map_err(|error| {
        CliError::cli_other_error(format!(
            "failed to encode cluster peer auth payload: {error}"
        ))
    })?;
    Ok(sha256_hex(&payload))
}

canonical_json_bytes is the RFC 8785 canonicalizer from chio-core-types, so key order and number formatting are fixed and both sides serialize identically. The secret is a field of the object rather than an HMAC key: this is a keyed digest, not HMAC-SHA256 and not a signature anyone can verify without the secret. The receiver recomputes the whole value from its own configuration and compares with subtle::ConstantTimeEq, the same comparison the admin bearer path uses. That comparison short-circuits on length before comparing contents, which subtle 2.6 documents on the slice impl. The expected value is always 64 hex characters, so the only thing timing distinguishes is a wrong-length header.

What the digest binds: the scheme, the shared token, the claimed node id, the route constant, the issue timestamp, and the term. term is Option<u64> and serializes as JSON null when absent, so a termless digest and a term-bearing one are different values over the same route. Sending the term header changes what must be signed. Binding the route constant is what stops a header captured on /v1/internal/cluster/status from being replayed against /v1/internal/budgets/delta.

What it does not bind: the request body, so the blockedPeerUrls array a partition POST carries is authenticated only by the caller holding the token; the query string, because the client signs the route constant while sending cursors as query parameters (request_internal_get_json(&url, path, term)); and the transport, because nothing here is channel-bound.

The body case is not academic. Once a caller is allowlisted and holds the token, one POST to /v1/internal/cluster/partition can mark every other peer partitioned on the receiver, which drops the receiver below quorum, invalidates its authority lease, and forces a full snapshot on each peer when the flag clears. The digest does not distinguish that body from an empty one.

Verification, in order

Order matters more than any single check here, so read the table as a sequence. Every failure body is JSON, {"error": "<message>"}, with the status shown.

These endpoints do not accept an Authorization: Bearer header at all. A bearer-only request to /v1/internal/cluster/status fails at step 1, before any token comparison, and comes back 401 with the chio.cluster.peer.v1 challenge. The peer headers are the only way in. Any runbook that curls an internal route with -H "Authorization: Bearer $CHIO_TRUST_SERVICE_TOKEN" is wrong on this build: reproduce try_internal_cluster_status in crates/products/chio-cli/tests/trust_cluster.rs instead, which signs the three headers by hand.

Two failure buckets

The split into two buckets is the point. Failures recorded before the digest verifies are keyed by unverified:sha256(node_id \0 endpoint); failures recorded after it verifies, which today means only clock skew, are keyed by the node id itself. The unverified bucket is consulted only on the mismatch branch, so a correct header is never answered 429 by it no matter how full it is. If both shared a counter, anyone able to reach the port could send eight garbage signatures naming node B and lock node B out of replication for a minute without holding any secret. auth_helpers_and_metered_billing_validation_cover_error_paths in trust_control/cluster_and_reports.rs drives exactly that: CLUSTER_AUTH_FAILURE_BURST spoofed failures, an assertion that the next one is 429, then a valid header for the same node id that still authenticates.

The verified bucket is written by that same test, through one expired timestamp, but nothing drives it to its own 429. That branch is implemented and unexercised.

Both buckets live in one process-local LazyLock<Mutex<HashMap<String, Vec<u64>>>>. Nothing is shared between nodes and nothing survives a restart, so this is a burst damper on one node, not a cluster-wide lockout.

The term gate on authority mutation

Replication pulls send no term: every internal client method passes None. A forwarded authority write must send one. Followers do not rotate keys or issue capabilities locally; they forward to the current leader, and validate_authority_mutation_auth adds three checks on top of peer auth.

• No term on a peer-authenticated mutation: 401 cluster authority mutation is missing the forwarded term.
• No authority lease: 503 cluster authority lease is unavailable for authority mutation. An invalid one: 503 cluster authority lease expired before authority mutation. lease_valid is has_quorum && not expired, so a minority partition cannot mutate authority state.
• Term does not equal the current lease term: 409 cluster authority mutation term does not match the current lease.

Those three refusals are this page’s half: they run in the auth layer, before any handler. What the term is, the persisted fence row it is compared against past this point, and the three-node kill-restart-replay test that exercises the whole path belong to Leader & Failover, which owns the counter. Read one thing from there before designing against the 409: the term is a per-node counter over a per-node observation history, so the comparison is against the receiver’s own lease term, never a cluster-agreed value.

Membership is configuration

The allowlist step 5 compares against is the peer map, and Leader & Failover owns how that map is built: once, at startup, from --peer-url, through normalize_cluster_config_url, with no runtime join or leave. One part of that validation is worth reading out here, because it decides which hosts can ever be a peer at all. Without --allow-local-peer-urls, the host is run through chio_external_guards::denied_external_guard_ip. For IPv4 that denies private, loopback, link-local, multicast, unspecified, and the 100.64.0.0/10 shared address space. For IPv6 it denies loopback, unspecified, unique-local, multicast, and unicast link-local, folding an IPv4-mapped address back onto the IPv4 test. A domain is resolved and every returned address is checked; localhost and .localhost names are refused by name. That resolution happens once, at startup, so it is a configuration check and not a live rebinding defence.

# Receipt, revocation and budget store flags omitted; see Trust Control
# Plane for the full command. Prefer CHIO_TRUST_SERVICE_TOKEN over
# --service-token so the shared secret is not visible in ps.
$ chio --authority-db /var/lib/chio/authority.sqlite trust serve \
    --listen 0.0.0.0:8940 \
    --advertise-url https://ctl-a.internal:8940 \
    --peer-url https://ctl-b.internal:8940 \
    --peer-url https://ctl-c.internal:8940

Because a node authenticates inbound peers against its own configured list, adding a fourth node means editing flags and restarting every node that must accept it.

Two startup refusals bound the configuration further, and one apparent third does not exist. Configuring peers alongside --authority-seed-file is a hard startup error, and a local joint authority database cannot run with peers at all. But --authority-db is not enforced at startup despite what the first error message says. A clustered node started with neither flag binds and serves; only then does /v1/internal/authority/snapshot answer 409 clustered authority replication requires --authority-db and issuance answer 409 trust control service requires --authority-seed-file or --authority-db. Without the database the persisted election term also resets to zero on every restart.

Guarantees and limits

Next Steps

• Leader & Failover · the deterministic pick, the lease, and what the term actually is
• Replication & Convergence · the pull loop that sends every one of these headers
• Authority & Rotation · what the authority snapshot carries and what it withholds
• Secrets & Signing Keys · custody for the one token this whole mechanism rests on
• Trust Control Plane · running a node: flags, stores, and the full serve command
• Node Observability · watching 401, 403, and 429 rates on the internal routes