Cluster Upgrade

Four things this is not

Chio ships exactly one upgrade procedure for a clustered trust-control plane, and it stops every node. Section 6 of docs/release/OPERATIONS_RUNBOOK.md orders it: qualify the candidate commit, obtain the exact binary set, take backups, drain external callers, stop the running Chio processes, replace the binary, restart chio trust serve first and dependent chio mcp serve-http edges second, then run post-upgrade smoke checks. Step 5 is plural and unordered. Nothing in the sequence upgrades one node while another serves.

The four negatives below are each checkable against the source. State them to whoever asked for a zero-downtime window before you plan one.

The one stream that does negotiate

Revocation replication is the exception, and it is a stronger mechanism than a defaulted field. prepare_peer_revocation_sync runs once per peer per sync round, before the revocation lane, and reads the contract that peer advertises off the status response already in hand. revocation_peer_contract returns Current when the peer named both a cursor version and a stream id, and Legacy when it named neither; sync_peer_revocations then dispatches on that value, so a node whose own cursor version is 4 pulling a peer that advertises none replays that peer’s whole revocation projection rather than failing the lane. The field’s doc comment states the purpose in as many words: it is advertised even when the stream is empty so a follower can distinguish an empty current stream from a legacy peer that has no stream-epoch support.

Read the boundary of that affordance carefully before planning anything on it. It covers one of the six replicated streams. ensure_revocation_cursor_version admits exactly one version and refuses a peer stamped 2 or 3 with UnsupportedRevocationCursorVersion, so the fallback is to a peer that advertises no version at all, not to any earlier one. Half a pair is IncompleteRevocationStreamContract, which marks the peer Unhealthy and ends the visit. The four delta streams beside it, budgets plus the two receipt streams plus lineage, negotiate nothing: their cursors are bare integers with no version alongside them, and a peer whose page shape changed is demoted rather than accommodated. Replication & Convergence carries the dispatch in full.

The one token that looks like a version and is not negotiated is the auth scheme. Every internal cluster route authenticates with chio.cluster.peer.v1, and that string is not advertised or compared, it is hashed: cluster_peer_auth_signature canonicalizes a JSON object whose first member is scheme and takes a SHA-256 over it. A build that changed the scheme constant would not negotiate down. It would produce a digest the receiver cannot reproduce, and every status probe, delta pull, and snapshot fetch between the two versions would answer 401 missing or invalid cluster peer authentication.

pub(crate) fn cluster_peer_auth_signature(
    service_token: &str,
    node_id: &str,
    endpoint: &str,
    issued_at: i64,
    term: Option<u64>,
) -> Result<String, CliError> {
    let payload = canonical_json_bytes(&json!({
        "scheme": CLUSTER_AUTH_SCHEME,
        "serviceToken": service_token,
        "nodeId": node_id,
        "endpoint": endpoint,
        "issuedAt": issued_at,
        "term": term,
    }))
    .map_err(|error| {
        CliError::cli_other_error(format!(
            "failed to encode cluster peer auth payload: {error}"
        ))
    })?;
    Ok(sha256_hex(&payload))
}

Before you stop anything

Five things have to be true, and four of them are cheap to check.

The peer set does not change

build_cluster_state reads --advertise-url and every --peer-url once, normalizes them, drops the node’s own URL from its peer map, and hands back an Arc<Mutex<ClusterRuntimeState>> that is never rebuilt while the process runs. The same list is the authentication allowlist: validate_cluster_peer_auth rejects a caller whose x-chio-cluster-node-id is not in config.peer_urls with 403 cluster peer is not in the configured allowlist.

A version upgrade that keeps the same three URLs therefore changes nothing about membership. Adding or removing a node is a different operation with a different failure mode: it needs every remaining node restarted with the new list, which means it is a fleet stop of its own, and it must not be bundled into a version upgrade if you want a clean rollback. Do one at a time.

Clustered nodes need an authority database

build_cluster_state refuses to start a clustered node that was given --authority-seed-file: clustered trust control requires --authority-db instead of --authority-seed-file. That refusal is what makes an upgrade resumable, because the persisted cluster fence lives in that database and is what a restarted node reloads instead of starting from term zero. A node whose invocation drifts to a seed file during an upgrade does not degrade, it fails to boot.

Record what you are leaving

Section 4 of the runbook ends its backup procedure with one line that does more work than the six sqlite3 .backup commands above it: record the binary version and git commit used for the backup snapshot. Nothing in the running system records it for you. There is no build version on GET /health, none on GET /v1/internal/cluster/status, and none in a receipt. If you do not write down which commit produced the binary that wrote a given backup, you cannot later tell whether that backup and a candidate binary are compatible.

The pre-upgrade evidence is node-local

Receipt flush and checkpoint operations run on the node that owns the database, over a local path. local_receipt_db_path refuses a --control-url outright:

if backend.control_url.is_some() {
    return Err(CliError::cli_other_error(format!(
        "{command_name} requires local --receipt-db; remote receipt operator operations are not supported in this release"
    )));
}

So a three-node cluster is three sets of the same commands, run against three different files, and there is no fleet view of the result. Run them with the binary that is currently serving, before the swap. The reason is in the next constraint.

Qualify the candidate first

Step 1 of the upgrade procedure is ./scripts/qualify-release.sh on the candidate commit. For the cluster specifically the narrower gate is ./scripts/qualify-trust-control.sh, seven named checks written to target/release-qualification/trust-control/logs/ with a qualification-report.md, an artifact-manifest.json, and a SHA256SUMS file beside them. Five of the seven run cluster drills, and three of those are the ones Cluster Recovery walks through in detail: quorum-heal, stale-leader-fencing, and replay-and-failover.

The procedure

Run it in this order. Every step here but the fourth is the runbook’s, and inserting the fourth puts everything after it one ahead of the runbook’s numbering. The notes are the cluster-specific consequences the runbook does not spell out.

1. Qualify the candidate. ./scripts/qualify-release.sh on the candidate commit, plus cargo xtask qualify bounded-chio and ./scripts/qualify-trust-control.sh for the ship-facing gate. Local qualification is not sufficient on its own: the runbook requires hosted CI and Release Qualification workflow results before any external tag or publication, and says outright not to promote from local qualification evidence alone.
2. Obtain the exact binary set. One binary for every node. There is no mechanism that distributes it and none that verifies two nodes are running the same one.
3. Take backups. Section 4: sqlite3 <db> ".backup <target>" for the receipt, revocation, authority, budget, verifier-challenge, and edge-session databases, plus a copy of the file-backed registries and policies under /etc/chio. Per node, because each node has its own. Record the version and commit.
4. Take the pre-upgrade receipt evidence. chio receipt health, chio receipt flush --timeout-ms 5000, chio receipt checkpoint status, and chio receipt checkpoint create --kernel-seed-file ... on each node, with the currently serving binary. Not part of the runbook’s numbered upgrade steps; it belongs here because the checkpoint chain is the thing a restore has to rejoin, covered in Backup & Restore.
5. Stop write traffic or drain external callers. There is no maintenance mode. A node below quorum still answers reads from its own disk with no staleness marker, so quiescing writers is something you do upstream of Chio, not to it.
6. Stop every Chio process. SIGTERM, then wait out the drain. Under the reference systemd units that is KillSignal=SIGTERM with TimeoutStopSec=35s, sized as the 25 second drain deadline plus a flush margin.
7. Replace the binary. On every node. Nothing runs both versions against one state directory: WAL mode makes each store a single-writer, single-host database, so two binaries sharing one receipt file corrupt it.
8. Restart trust-control first, edges second. Bring the lexicographically lowest --advertise-url up first; the reason is in the next section. Only then start the dependent chio mcp serve-http edges. The bounded profile makes this an operating rule, not a preference: chio trust serve starts before chio mcp serve-http.
9. Run the post-upgrade smoke checks. Per node, and read the verification section below before you copy the runbook’s curl lines, because one of them is authenticated the wrong way.

If SDK packages ship with the same release, the runbook’s step 9 also runs ./scripts/check-chio-ts-release.sh, ./scripts/check-chio-py-release.sh, and ./scripts/check-chio-go-release.sh. Those check package contents, not the running cluster.

Stopping a node is a quorum event

Quorum is computed locally, from a fixed formula over the configured peer count, and it does not care why a peer is absent. One line decides how many nodes you may have down at once.

let quorum_size = cluster.peers.len().div_ceil(2) + 1;
let mut candidates = vec![cluster.self_url.clone()];
for (peer_url, peer_state) in &cluster.peers {
    let contact_is_fresh = peer_state
        .last_contact_at
        .is_some_and(|last_contact_at| now <= last_contact_at.saturating_add(lease_ttl_secs));
    if peer_state.health.is_reachable() && !peer_state.partitioned && contact_is_fresh {
        candidates.push(peer_url.clone());
    }
}
candidates.sort();
let reachable_nodes = candidates.len();
let has_quorum = reachable_nodes >= quorum_size;
let leader_url = if has_quorum { candidates.first().cloned() } else { None };

peers excludes the node itself, so for a cluster of N the quorum is (N-1).div_ceil(2) + 1. Read the tolerance column before planning any procedure that stops one node at a time.

How fast a stopped node disappears

A peer leaves the candidate set on the first consensus computation after its contact stamp goes stale, and the staleness window is derived from the sync interval rather than configured directly:

pub(crate) fn authority_lease_ttl(sync_interval: Duration) -> Duration {
    let scaled = sync_interval
        .checked_mul(3)
        .unwrap_or_else(|| Duration::from_secs(5));
    scaled
        .max(Duration::from_millis(500))
        .min(Duration::from_secs(5))
}

The consensus computation then truncates that to whole seconds with a floor of one: Duration::from_millis(lease_ttl_ms).as_secs().max(1). At the CLI default --cluster-sync-interval-ms 500 the lease is 1500ms and the freshness window is one second, so a stopped node is out of every survivor’s candidate set roughly a second after its last answered probe. The cluster drills run at 2000ms, which clamps the lease to the 5 second ceiling and gives a 5 second window, so their observed convergence timings say nothing about a default deployment.

Which stop moves the leader

The leader is candidates.first() after a lexicographic sort of the reachable set, so leadership changes exactly when the lowest-sorting reachable node joins or leaves. Stopping any other node changes reachableNodes and nothing else. Every leadership change increments the local election_term. The term is per-node only in the sense that nothing forces equality: seed_cluster_authority_from_snapshot raises a node’s term to the one a peer’s snapshot carries when that one is higher, and persists it through seed_cluster_fence, and nothing anywhere lowers it. That is why the verification below asserts on leaderUrl agreement rather than term equality.

The practical consequence is about the restart, not the stop. Bring the lowest-sorting node up first and the cluster elects it once and keeps it for the rest of the window. Bring it up last and every other node elects an interim leader, then hands leadership over when the lowest node appears, costing an extra term bump and an extra round of forwarding churn. The drill that proves the underlying pick asserts it directly: reserve_cluster_nodes sorts its three URLs and trust_control_cluster_rejects_stale_authority_term_after_failover_and_restart then asserts initial_leader == url_a, the lowest.

What a stop actually waits for

serve_async installs a shutdown controller, hands the HTTP server a graceful-shutdown future, and drains under ServeHygieneConfig::drain_timeout, which is DEFAULT_DRAIN_TIMEOUT, 25 seconds. The cluster sync loop watches the same signal and is joined afterwards inside whatever is left of that one window, never a fresh wait stacked on top:

fn cluster_join_budget(drain_timeout: Duration, elapsed_since_signal: Duration) -> Duration {
    drain_timeout.saturating_sub(elapsed_since_signal)
}

Two properties follow, and both matter to an upgrade. First, teardown fits in one drain budget, which is why the reference unit sets TimeoutStopSec=35s and the runbook makes it a deploy contract that the platform stop grace is at least that: Cloud Run timeoutSeconds, ECS stopTimeout, Kubernetes terminationGracePeriodSeconds. Second, an outbound peer call still running at the deadline is abandoned on purpose. The comment in init.rs says why: outbound peer sync is best-effort catch-up that resumes on the next boot, so dropping it never loses a receipt. Trust-control writes budget and revocation state synchronously inside its handlers, so finishing in-flight requests is the entire drain.

The drills do not exercise that path. They stop a node by dropping a ServerGuard, whose Drop calls Child::kill. A failover drill proves what survivors do when a node vanishes without warning; it proves nothing about the graceful path an upgrade takes.

What a restarted node has to redo

Every peer entry is constructed from PeerSyncState::default(), and the default is deliberately pessimistic: health: Unknown, every cursor zero or None, and force_snapshot: true. Peer state is entirely in memory and none of it is persisted. So after a fleet restart, every node’s first successful visit to every peer pulls a full cluster snapshot before any delta page, in both directions.

That is the single largest cost of the stop-the-fleet procedure, and it scales with retained state rather than with the size of the change. build_cluster_state_snapshot materializes every revocation, tool receipt, child receipt, lineage row, budget projection, and budget mutation event into one response body. The receiver reads it under MAX_PEER_RESPONSE_BYTES, 64 MiB, and the whole build, transfer, and decode has to finish inside CONTROL_HTTP_TIMEOUT, 15 seconds. Neither side streams and neither side pages. Cluster Recovery covers what happens when a snapshot cannot be fetched, including the fact that the failure is silent and shows only as forceSnapshot: true against a frozen snapshotAppliedCount. Size an upgrade window against those two ceilings, on the node with the largest stores, before the first stop.

The term does survive a restart, conditionally. On startup build_cluster_state opens the authority database, reads its status and its cluster fence, and adopts the persisted term and leader URL only when the fence’s authority generation and rotation stamp still match the authority’s. When they do not it logs discarding stale persisted authority fence after authority rotation and starts at term zero. It also filters the persisted leader URL down to this node or a currently configured peer, so a fence written before a membership change cannot resurrect a URL that is no longer in the peer map. Do not rotate the authority key and upgrade in one window unless you want to reason about both.

The stamp that makes rollback a state restore

Every store under chio-store-sqlite writes two marks into its SQLite file: PRAGMA application_id set to 0x4348494F, which is ASCII CHIO, and a per-store revision row in a keyed table named chio_store_schema_versions. The revision is keyed rather than kept in PRAGMA user_version because the sidecar co-locates several stores in one file, and one database-wide pragma cannot give them independent revisions.

Two of the six databases step 3 backs up carry neither mark. PassportVerifierChallengeStore::open runs a bare CREATE TABLE IF NOT EXISTS batch with no application id and no revision row, and nothing in chio-mcp-remote stamps the edge session file. Both open on any binary. Everything below is about the four stores in the table that follows, and about nothing else.

The open path fails closed on three conditions, and the third is the one an upgrade has to plan around.

#[derive(Debug, thiserror::Error)]
pub enum SchemaVersionError {
    #[error("sqlite error: {0}")]
    Sqlite(#[from] rusqlite::Error),
    #[error("database application_id {found:#x} is not a Chio store (expected {expected:#x})")]
    ForeignDatabase { found: i32, expected: i32 },
    #[error(
        "database carries the Chio application_id but none of this store's tables ({expected_anchors:?}); refusing to open another store's database"
    )]
    MismatchedStore { expected_anchors: Vec<String> },
    #[error(
        "database schema version {found} is newer than this binary supports ({supported}); refusing to open"
    )]
    FutureSchema { found: i32, supported: i32 },
}

The module header states the intent in one sentence: a foreign, misdirected, or future database is refused before any write, so a rollback to an older binary or a mistargeted path is caught at open rather than after data has been commingled or a newer schema misread. Each store carries its own compile-time revision constant, and the four a clustered trust-control node opens are independent of each other.

Migrations run forward on the writable open path, with CREATE TABLE IF NOT EXISTS and column probes. They are additive and idempotent, and nothing in the tree migrates a revision backwards. So the asymmetry is total: a new binary opens an old database and moves it forward, and from that moment the old binary refuses it with FutureSchema if any revision moved. A binary-only rollback works when no store’s constant changed between the two builds and fails at open when one did. You cannot tell which case you are in from the running system, only by comparing the constants in the two source trees, so plan for the failing case.

Verifying the fleet came back

The runbook lists four curl lines for step 8. Three of them work as written. The one against /v1/internal/cluster/status does not, and the reason is worth knowing because it is the same reason there is no fleet-wide upgrade command.

So /health is the post-upgrade check, on every node, one at a time. Four assertions are worth making and one is worth avoiding.

• clustered: true and cluster.peerCount equal to N-1 on every node. A node that came back with a truncated --peer-url list is otherwise indistinguishable from a healthy one.
• hasQuorum: true and reachableNodes equal to N on every node, not just on the leader.
• The same leaderUrl reported by every node, and a role of leader on exactly one of them.
• unhealthyPeers, unknownPeers, partitionedPeers, and lastErrorCount all zero. The health block reports counts and never the error strings, so a non-zero lastErrorCount sends you to the internal status route for lastError.
• Do not assert that electionTerm matches across nodes. It is a per-node counter incremented on each local leadership change, and also raised to a peer’s term when that peer’s snapshot carries a higher one, so two nodes can legitimately sit at different terms.

Then confirm convergence rather than reachability: on each node, snapshotAppliedCount should have advanced and forceSnapshot should be back to false for every peer entry. Those two fields are only on /v1/internal/cluster/status, which needs a peer-authenticated call; if you cannot make one, the next best evidence is a written receipt visible from every node, which is what the drills assert instead of counters.

Rollback is a restore

Section 7 of the runbook is six steps and the first sentence is the whole design: rollback is a full binary-and-state rollback to the last known good backup.

1. Stop the candidate processes. All of them, same as the upgrade.
2. Restore the previous binaries. On every node.
3. Restore the backed-up SQLite and registry files if the candidate performed writes that must be discarded. Read that conditional carefully: it is written as a choice about discarding writes, and it is also the only way to get a database whose stamped revision the previous binary will open. If any store’s revision moved, the restore is mandatory and skipping it produces a node that refuses to boot.
4. Restart the previous version with the original arguments. Original means original: the peer list and the advertise URL are read once at startup, and a node that comes back with a different set is a different cluster member.
5. Re-run the same health and admin smoke checks. The ones from the upgrade, with the same caveat about the internal status route.
6. Record the failed candidate commit and attach the qualification logs and any cluster and admin diagnostics to the incident report.

Guarantees and limits

Next Steps

• Node Upgrade · the per-node half: one process’s drain, its store migrations, and the checks that belong on the box
• Cluster Recovery · the snapshot path every restarted node runs, and what a failed snapshot fetch does not tell you
• Backup & Restore · what to copy before step 3, what a copy does not carry, and the checkpoint chain a restored receipt log rejoins
• Leader & Failover · the deterministic pick that decides which stop moves leadership, and the term fence a returning node meets
• Cluster Peer Auth · the chio.cluster.peer.v1 digest, the allowlist, and why a bearer token cannot read the internal status route