Authority & Rotation

One keypair, one trust domain

A cluster is many nodes under one capability authority: a single Ed25519 keypair that signs every CapabilityToken the cluster issues, alongside a history of every public key that authority has ever used. Both live in the SQLite database behind --authority-db, which is why configuring --peer-url alongside --authority-seed-file is rejected at boot with clustered trust control requires --authority-db instead of --authority-seed-file: a clustered node needs a store, not a bare seed. Configuring neither is not rejected. The cluster comes up, the snapshot route answers 409, the per-round authority sync is a no-op, and the node has no capability authority at all.

The mechanism is small on purpose. The interesting part is the seam it draws between the key a cluster verifies against, which replicates freely, and the key a node signs with, which does not replicate at all.

Three tables and two accessors

SqliteCapabilityAuthority::open creates the schema on first contact. Three tables carry the lifecycle; a fourth, authority_budget_anchor_commits, is an append-only commitment chain for budget snapshot anchors, held immutable by BEFORE UPDATE and BEFORE DELETE triggers that RAISE(ABORT). It signs with the same seed, which matters for the role-separation row at the end, but its own lifecycle is out of scope here.

Two accessors read the seed, and confusing them is the fastest way to misread this subsystem.

if keypair.public_key() != status.public_key {
    return Err(AuthorityStoreError::Fence(format!(
        "local signing seed public key {} does not match replicated authority public key {}",
        keypair.public_key().to_hex(),
        status.public_key.to_hex(),
    )));
}

The lifecycle, in order

Bootstrap

Opening the store generates a fresh keypair and inserts it at generation 1 with ON CONFLICT(singleton_id) DO NOTHING, so a second handle on the same file adopts the existing row rather than overwriting it. The follow-up insert into authority_trusted_keys uses the public key derived from whatever seed the row now holds, not the freshly generated one, and is likewise a no-op when that key is already present. sqlite_capability_authority_persists_and_rotates_across_handles rotates through one handle and reads the new key back through the other.

Rotate

rotate() generates a new keypair, overwrites the singleton row, and appends. Nothing is removed:

UPDATE authority_state
SET seed_hex = ?1, public_key_hex = ?2, generation = ?3, rotated_at = ?4
WHERE singleton_id = 1;

INSERT INTO authority_trusted_keys (public_key_hex, generation, activated_at)
VALUES (?1, ?2, ?3)
ON CONFLICT(public_key_hex) DO NOTHING;

The generation is the previous generation plus one, saturating. rotated_at is Unix seconds. Capabilities minted before the rotation keep verifying because the kernel checks a signature against the whole trusted set, not the current key: trusted_issuer_keys() unions the configured CA keys, the authority’s trusted keys, and the kernel’s own public key. sqlite_capability_authority_issues_with_current_rotated_key issues either side of a rotation and asserts the two issuers differ and that the second one is the rotated key. It verifies only the second signature; that the first still verifies is a property of the trusted set, not something that test checks.

Rotation applies to future sessions only

Every authority status the control plane returns carries the field applies_to_future_sessions_only, hardcoded true on every branch of load_authority_status and rotate_authority, and serialized as appliesToFutureSessionsOnly. It is a statement about what rotation does not do. A live session keeps the capabilities it was issued, signed by the key that was current when it opened. Nothing re-signs them, nothing revokes them, and nothing tears the session down.

Two tests cover this and they are not interchangeable. Both open session A, rotate, open session B, and assert that every capability in A still names the old issuer while every capability in B names the new one. mcp_serve_http_admin_authority_rotation_only_affects_future_sessions does it against --authority-seed-file through /admin/authority, so it proves the property on the seed-file backend, not on the clustered store. mcp_serve_http_shared_authority_rotation_propagates_across_nodes is the SQLite equivalent: it asserts backend is sqlite and the generation moves by exactly one, across two processes pointed at one --authority-db file. Neither asserts the old capabilities still verify, only that their issuer field is untouched.

Snapshot and merge

Replication of the authority is not a delta stream. Every sync round, each node fetches the peer’s whole authority record from GET /v1/internal/authority/snapshot and merges it locally, which is why the authority has no cursor in Replication & Convergence. The payload is four fields:

pub struct AuthorityTrustedKeySnapshot {
    pub public_key_hex: String,
    pub generation: u64,
    pub activated_at: u64,
}

pub struct AuthoritySnapshot {
    pub public_key_hex: String,
    pub generation: u64,
    pub rotated_at: u64,
    pub trusted_keys: Vec<AuthorityTrustedKeySnapshot>,
}

apply_snapshot merges in two stages. Trusted keys are unioned first, upserting each key to MAX(generation) and MIN(activated_at), so the history only ever grows and each key keeps its earliest observed activation. Then the current pointer is replaced only when the remote record is strictly newer: a higher generation wins, and a tie on generation is broken by comparing (rotated_at, public_key_hex) as a tuple. The merge is symmetric, not leader-authoritative. Every node pulls from every peer, so the highest generation anywhere in the cluster becomes the current key everywhere: sqlite_capability_authority_rotates_and_applies_newer_snapshot rotates a replica and watches the primary adopt it.

What the snapshot withholds

The snapshot carries no seed. The comment above the merge says so in one line, and the store test that guards it is named after the property:

// Cluster snapshots replicate verification history, not signing custody.

The consequence is exact. After a follower applies a leader’s snapshot, its authority_state row names the leader’s public key at the leader’s generation, while seed_hex still holds the seed it was opened with. That node still answers authority status and the full trusted key set, so any kernel reading from it verifies exactly what the leader’s kernel verifies, and it still performs its node-local signing through local_keypair(). It cannot issue a capability, because current_keypair() refuses the mismatch. sqlite_capability_authority_snapshot_updates_public_view_without_copying_seed asserts both halves in one test: the replicated public view advances, and current_keypair() errors with the follower’s own key named in the message.

Claim: a follower reads through failover, but does not inherit the signer

The rotation request path

POST /v1/authority is the control plane’s way to advance a generation on a running cluster. Five gates, in order:

• Authenticate. validate_authority_mutation_auth treats the request as a peer forward if any x-chio-cluster-* header is present, and otherwise falls back to the bearer service token. Its three refusals, all before any handler runs, are in Cluster Peer Auth.
• Forward. forward_authority_post_to_leader sends the request to the current leader unless this node is it, refreshing the term from the leader’s own status first. It refuses 503 before forwarding anything when the cluster has no quorum, no known leader, or no valid lease, and retries at most twice, re-resolving the leader between attempts. Rotation is unavailable during a partition, not merely delayed. See Leader & Failover for how that leader is picked.
• Fence. enforce_authority_mutation_fence reads the persisted fence row and refuses a stale term three ways, each a 409; Leader & Failover states them. What matters here is when the step is off rather than what it rejects: an unclustered node has no lease view, a node without --authority-db has no fence row, and the generation check is skipped while the row is still unfenced, meaning term 0 with no leader.
• Rotate, then re-fence. Rotation changes the generation, which by construction invalidates the fence row the pre-rotation check just passed, so refresh_authority_mutation_fence reseeds it against the new generation immediately after.
• Verify visibility. respond_after_leader_visible_write re-reads the authority and returns the status only when the generation and public key match what the rotation produced. Otherwise it is a 500 reading rotated authority was not visible on the leader after write, with the rotation already committed.

The fence binds a term to a specific authority generation, so a node that comes back from the dead cannot replay an old term against a key that has since moved. sqlite_capability_authority_enforce_cluster_fence_rejects_stale_rotation seeds a fence, rotates, and asserts the enforcement then fails closed naming the stale generation. At boot, build_cluster_state applies the same rule: a persisted fence is adopted only when its recorded generation and rotation timestamp still match, and is otherwise discarded with a warning.

How an edge kernel notices

Kernels running against --control-url hold a RemoteCapabilityAuthority wrapping a cached authority status. The cache TTL is AUTHORITY_CACHE_TTL, two seconds, and both the current key and the trusted set refresh lazily on read. Two seconds is a staleness bound on reads, not a propagation bound: nothing polls in the background, so an idle edge holds its last key indefinitely, and refresh_status_if_stale discards the result of a failed refetch without retrying or reporting an error, so an edge that cannot reach the control plane keeps serving the stale key silently. Issuance is stricter: a capability whose issuer is not the cached current key, or whose cache entry is already stale, forces a synchronous refetch, and that one does fail the issuance if it cannot complete.

Cache installation is monotonic. ensure_not_older_than rejects a status that moves the generation backwards, one that keeps the generation while changing the key material or the rotation timestamp (reported as equivocated), one that advances the generation while moving rotated_at backwards, and one that drops generation metadata once a versioned status has been seen.

Guarantees and limits

Next Steps

• Secrets & Signing Keys · the custody half: seed files, loss and compromise, and the delegated-signing patterns that apply to the kernel key rather than to this one
• Leader & Failover · how the term this page fences on is computed, and where a rotation request is forwarded
• Replication & Convergence · the pull round that refetches the authority snapshot whole, every round
• Capabilities · the token this keypair signs, and what a verifier checks
• Rotate Keys & Revoke · the operator runbook that pairs rotation with revocation