Authority & Rotation
Rotate the cluster’s capability authority while its public key history replicates to every peer and the signing seed stays where you put it.
On this page
A cluster can fail over without copying its signing key. Every peer receives the authority’s public key, generation, and trusted key history; the seed that signs capabilities stays on the node that holds it, and issuance refuses wherever the two disagree.
Run the clustered authority with --authority-db. A peer started with --authority-seed-file is refused at startup, and a peer with neither starts with no capability authority to synchronize: its authority snapshot route returns 409. Seed storage and recovery are on Secrets & Signing Keys.
A snapshot carries verification state and leaves the seed behind
After a follower applies the leader’s snapshot, its authority_state row names the leader’s public key at the leader’s generation, and its seed_hex still holds the seed it opened with.
The follower answers authority status and serves the full trusted key set, so a kernel reading from it verifies capabilities the leader signed. Other trust configuration and admission checks stay local to each node, and its node-local signing continues through local_keypair(). Issuance goes through current_keypair(), which refuses a seed whose public key differs from the replicated one with AuthorityStoreError::Fence.
Leader promotion moves no seed and triggers no rotation, so a promoted node issues only when its seed already matches. Give every node that may issue the same seed under your custody, or point the issuing processes at one shared authority database.
Peers merge authority snapshots by union, so trusted history only grows
Each sync round, a node fetches the peer’s whole authority record from GET /v1/internal/authority/snapshot and merges it, which is why the authority has no cursor in Replication & Convergence. apply_snapshot merges in two stages:
- Union. Every trusted key is upserted at
MAX(generation)andMIN(activated_at), so each key keeps its earliest observed activation. - Pointer. The current key moves only for a strictly newer record: a higher generation, or a tie broken by comparing
(rotated_at, public_key_hex).
The union runs before the pointer comparison, so even a snapshot from a generation the node has passed adds its keys, and no path deletes one. The merge is symmetric: each node adopts the newest record it has received, and convergence follows the pulls that complete. A snapshot carries no signature of its own; the peer digest authenticates it, which makes each node’s peer list part of its issuer trust configuration.
The snapshot payload
pub struct AuthorityTrustedKeySnapshot {
pub public_key_hex: String,
pub generation: u64,
pub activated_at: u64,
}
pub struct AuthoritySnapshot {
pub public_key_hex: String,
pub generation: u64,
pub rotated_at: u64,
pub trusted_keys: Vec<AuthorityTrustedKeySnapshot>,
}Rotation changes the signer for new sessions and keeps old keys trusted
rotate() generates a new keypair, overwrites the singleton row, and appends the new public key to the trusted history. The generation becomes the previous one plus one, saturating, and rotated_at records Unix seconds.
UPDATE authority_state
SET seed_hex = ?1, public_key_hex = ?2, generation = ?3, rotated_at = ?4
WHERE singleton_id = 1;
INSERT INTO authority_trusted_keys (public_key_hex, generation, activated_at)
VALUES (?1, ?2, ?3)
ON CONFLICT(public_key_hex) DO NOTHING;Capabilities minted before a rotation keep verifying, because the kernel checks signatures against trusted_issuer_keys(): the configured CA keys, the authority’s trusted_public_keys(), and the kernel’s own public key. Every authority status carries appliesToFutureSessionsOnly: true; a live session keeps the capabilities it was issued, signed by the key current when it opened.
Rotation retires nothing already issued. For a compromised key, rotate to change the signer for new capabilities, then revoke the outstanding ones by capability id in the revocation store; Rotate Keys & Revoke pairs the two.
A rotation request passes five gates on its way to the leader
POST /v1/authority advances the generation on a running cluster. Each rotation is an explicit administrative request; nothing rotates on a schedule, on promotion, or on a compromise signal.
- Authenticate.
validate_authority_mutation_authtreats the request as a peer forward when anyx-chio-cluster-*header is present, and as an operator bearer request otherwise. Its refusals are on Cluster Peer Auth. - Forward.
forward_authority_post_to_leadersends the request to the current leader, refreshing the term from the leader’s status first. It returns 503 before forwarding when the node has no quorum, no known leader, or no valid lease, and retries at most twice, re-resolving the leader between attempts. - Fence.
enforce_authority_mutation_fencereads the persisted fence row and refuses a stale term with 409. The step is skipped on an unclustered node, on a node without--authority-db, and while the row is unfenced (term 0, no leader). - Rotate, then re-fence. Rotation changes the generation the fence was recorded against, so
refresh_authority_mutation_fencereseeds the row at the new generation immediately. - Confirm.
respond_after_leader_visible_writere-reads the authority and returns the status when the generation and public key match the rotation. Otherwise it returns 500,rotated authority was not visible on the leader after write, with the rotation already committed.
The term fences a stale authority write
A forwarded authority write carries its term in x-chio-cluster-auth-term, covered by the peer keyed digest. B refuses the replayed term 1 write with 409 before anything is written, and admits the term 2 write, which advances the authority generation by one. B persists the fence (term, leader URL, authority generation) in its authority database, so a restart keeps the old term closed. Authority & Rotation.
The persisted fence binds a term to this database’s authority generation and rotation time, so it fences writes against one database’s authority state; Leader & Failover describes how terms advance across nodes. At boot, build_cluster_state adopts a persisted fence only when both still match, and otherwise discards it with a warning. The gates live on the control-plane handler, while rotate() is a store method: POST /admin/authority on an mcp serve-http node that shares the --authority-db file and has no --control-url rotates with no term or fence check. Treat write access to the authority database as leadership.
Authority store reference
Try refresh threshold, local_keypair, or generation.
Authority tables
SqliteCapabilityAuthority::open creates the schema on first contact.
authority_state
Holds seed_hex (the signing half), public_key_hex, generation, and rotated_at. public_key_hex is the one nullable column: when it is null or empty, the reader derives the public key from seed_hex, which is how a database written before replication still opens.
authority_trusted_keys
generation and activated_at for each issuer key this store retains. The table has no validity interval, retirement reason, or delete path, so a key stays in the trusted history of every node that merged it.
authority_cluster_fence
leader_url, election_term, and copies of the authority_generation and authority_rotated_at the term was recorded against.
authority_budget_anchor_commits
The commitment chain for budget snapshot anchors, held immutable by BEFORE UPDATE and BEFORE DELETE triggers that RAISE(ABORT). It signs with the store’s seed.
Seed accessors
local_keypair()
Returns whatever seed this node holds. Node-local signing that speaks for the node resolves through it: the OID4VP verifier request signer, the reputation and policy feed export signer, and the remote session-resume integrity seed.
current_keypair()
Returns the same seed only after its public key equals the replicated authority_state.public_key_hex. Capability issuance uses it; on a mismatch it returns AuthorityStoreError::Fence and issuance fails.
if keypair.public_key() != status.public_key {
return Err(AuthorityStoreError::Fence(format!(
"local signing seed public key {} does not match replicated authority public key {}",
keypair.public_key().to_hex(),
status.public_key.to_hex(),
)));
}Bootstrap
Opening the store generates a fresh keypair and inserts it at generation 1 with ON CONFLICT(singleton_id) DO NOTHING, so a second handle on the same file adopts the existing row. The follow-up insert into authority_trusted_keys uses the public key derived from the seed the row now holds, and is likewise a no-op when that key is present. sqlite_capability_authority_persists_and_rotates_across_handles rotates through one handle and reads the new key back through the other.
Seed-file backend
--authority-seed-file is the standalone development backend. It rotates by overwriting the seed file, reports no generation and no rotated_at, and keeps a one-element trusted set holding only the new key, so capabilities issued before a rotation stop verifying against it. Clustered nodes refuse it at startup.
Edge kernel cache
Kernels started with --control-url hold a RemoteCapabilityAuthority over a cached authority status. The cache TTL is AUTHORITY_CACHE_TTL, two seconds, and the current key and trusted set refresh lazily on read. Two seconds is the refresh threshold, not a maximum staleness or propagation bound: nothing polls in the background, so an idle edge holds its last key until a read, and refresh_status_if_stale discards a failed refetch without failing that read. Until a refresh succeeds, the edge keeps its stale verification state.
Issuance is stricter: a capability whose issuer is not the cached current key, or whose cache entry is stale, forces a synchronous refetch, and a refetch that cannot complete fails the issuance. Cache installation is monotonic. ensure_not_older_than refuses a status that moves the generation backwards, keeps the generation while changing the key material or rotation time (reported as equivocated), advances the generation while moving rotated_at backwards, or drops generation metadata after a versioned status.
Signing roles
With --authority-db, the kernel signer and the capability authority are separate keypairs: the kernel keeps its own, and the store keeps another. The store’s one seed also signs budget snapshot anchor commitments, OID4VP verifier requests, and reputation and policy feed exports, and derives the remote session-resume integrity seed, with no per-artifact scope. commit_budget_snapshot_anchor_set reads the seed directly, so the anchor chain signs with the node’s local seed.
The store reads a raw Ed25519 seed from its own row through read_keypair_from_connection and Keypair::from_seed_hex. CapabilityAuthority is a public trait; a custom implementation takes no part in apply_snapshot.
Authority operations
- Rotate the issuing key. New sessions use the new issuer; existing capabilities keep their original issuer and old keys stay trusted. The generation increments and the time is recorded.
- Rotate through a follower. The follower forwards to the leader, then receives the new authority state by replication.
- Retry a stale term. The authority refuses the request with 409 and leaves the generation unchanged.
- Refresh an edge kernel. Reads refresh the cached authority after two seconds; a failed refresh leaves the previous state in use.
- Retire a compromised issuer. Revoke its outstanding capabilities and rely on capability lifetimes; the trusted history keeps the key for verification.
- Authenticate replication. Peers present the shared-secret digest (
chio.cluster.peer.v1, 60-second skew window); a holder of the service token can authenticate as any listed peer.
Source tests
sqlite_capability_authority_snapshot_updates_public_view_without_copying_seed: the replicated public view advances, andcurrent_keypair()errors naming the follower’s own key.sqlite_capability_authority_rotates_and_applies_newer_snapshot: the union merge and the fencedcurrent_keypair()together, with a rotated replica adopted by the primary.sqlite_capability_authority_issues_with_current_rotated_key: issuers differ on either side of a rotation, and the second is the rotated key.sqlite_capability_authority_enforce_cluster_fence_rejects_stale_rotation: after a rotation, enforcement refuses the fence naming the stale generation.mcp_serve_http_shared_authority_rotation_propagates_across_nodes: on the SQLite backend, two processes sharing one authority database see the generation move by one; sessions opened before the rotation keep the old issuer, and sessions opened after it get the new one.mcp_serve_http_admin_authority_rotation_only_affects_future_sessionsshows the same shape on the seed-file backend.run_trust_control_cluster_proving_scenario: a rotation posted to a follower is forwarded, applied once on the leader, and converges on the follower, with the generation moving 1, 2, 3 across two live processes.trust_control_cluster_rejects_stale_authority_term_after_failover_and_restart: a replayed pre-failover term is refused with 409 and the generation unchanged.
The store lives in crates/platform/chio-store-sqlite/src/authority.rs; the mutation gates live in the trust-control authority handlers.