Node Upgrade
Replace a node’s binary in a maintenance window, keep a restorable copy of its state, and verify it before traffic returns.
On this page
After an upgrade the node runs the exact candidate you qualified, over the state it held before, and you hold the previous binary and a rehearsed snapshot as the way back.
The upgrade is a stop, a binary swap, and a restart inside one maintenance window. The candidate can migrate a store the first time it opens it writable, so the binary, the stores, the keys, and the configuration move as one unit. Clusters roll through Cluster Upgrade.
StartCallers reach no node from the stop to the reopen
Traffic stops before the snapshot, the binary changes while nothing runs, and callers return only after the restarted node passes its checks. The previous binary stays on disk until the candidate is accepted. Node Lifecycle.
1. Qualify the exact candidate
Obtain the candidate as a signed release and install it on a staging host with the installer, which verifies the release signature before it installs. Then check two things before the window opens: that the binary is the release its signature names and passed the release lanes, and that this node's state and units can move to it.
chio release verify --evidence qualification
chio release check 0.2.0 --state /var/lib/chio --unit chio-trust.servicebinary /usr/local/bin/chio chio 0.2.0 x86_64-unknown-linux-gnu
checksum matches SHA256SUMS for v0.2.0
signed by release.yml on backbay-labs/chio, issuer token.actions.githubusercontent.com
qualification bound to v0.2.0, every gate passed
verified
target chio 0.2.0, verified
stores authority /var/lib/chio/authority.db schema 7 to 8, migrates on first start
receipts /var/lib/chio/receipts.db schema 12, unchanged
sessions /var/lib/chio/sessions.db schema 3, unchanged
flags chio-trust.service: no renamed or removed flags
rollback binary-only rollback is unavailable after the authority migration; keep the snapshot
readyYou should see verified and ready. Record the checksum line in the maintenance record, because step 4 installs that file and no other. The rollback line decides your way back: here the authority store migrates on first start, so the snapshot in step 3 is the only rollback. chio release check reads the stores read-only and exits 1 when a unit passes a flag the target removed.
Read what the release lanes cover in release qualification lanes.
2. Stop traffic and every writer
Take the node out of ingress, then stop the edge before the authority service it depends on. A locally hosted MCP edge has no separate trust-control process to stop.
systemctl stop chio-mcp-edge.service
systemctl stop chio-trust-control.serviceWait for each process, and every wrapped tool process, to exit. The reference units allow 35 seconds: the 25-second drain plus the flush margin. Read the drain outcome and confirm no writer still owns the state before you copy it.
3. Back up the whole recovery set
Record the current binary’s checksum and version, the service arguments, the policy files, the key and credential locations, and every configured state path. Then copy what the node holds:
| State | Copy | Restore depends on |
|---|---|---|
| Receipt and standalone stores | Receipt, authority, budget, revocation, and verifier databases in use | SQLite’s backup API |
| Remote MCP sessions | Session database and the local admission sidecar | Admission kernel seed and lock-root material |
| Joint authority store | Database with its identity | Protected rollback anchors and file identity |
| Keys and configuration | Seeds, policies, registries, verifier policies, certifications | Preserved permissions and access controls |
For a stopped receipt store, use SQLite’s backup form with your own paths and a private, existing backup directory:
set -e
umask 077
source_db="/var/lib/chio/receipts.sqlite3"
backup_db="/var/backups/chio/receipts-$(date -u +%Y%m%dT%H%M%SZ).sqlite3"
test -f "$source_db"
test ! -e "$backup_db"
sqlite3 -readonly "$source_db" ".backup '$backup_db'"
sqlite3 -readonly "$backup_db" "PRAGMA integrity_check;"ok
You should see ok from the structural integrity check. It checks SQLite structure; signed evidence and the joint store’s identity are checked by rehearsing the restore. Rehearse the chosen procedure before you accept the snapshot, and keep the joint store’s identity checks and protected anchors intact, because they are part of its security contract. Node State on Disk and Backup & Restore cover each store.
4. Swap the binary while nothing runs
Keep the previous binaries, and install the qualified candidate at the path the unit runs, /usr/local/bin/chio. That path sits outside the writable state directory, so a running service cannot rewrite its own binary. Restart with the same command line the process had before.
Copy a command line only between services of the same kind. On chio mcp serve-http, --session-db names the remote session database; on chio trust serve, it names a joint authority store. Local durable admission rejects separately configured budget and revocation databases.
Read more in service arguments.
5. Start dependencies before their clients
An edge that uses --control-url needs its authority service first. Start it, check it, then start the edge:
systemctl start chio-trust-control.service
# Inspect /health and authenticated /v1/authority before starting the edge.
systemctl start chio-mcp-edge.serviceWatch the logs through startup and the first new session. Some stores open inside handlers or per session, so a listening process can still meet a store refusal on its first request. Remote session restore drops a persisted session that fails reconstruction and keeps starting, so compare the restored sessions with your inventory.
6. Verify before you reopen traffic
Answer each question against the restarted node:
- Authority. Trust-control
/healthand authenticated/v1/authorityshow the intended key and generation; read the nested fields even when/healthreturns 200. - Configuration and sessions. MCP
/admin/healthand/admin/sessionsshow the intended configuration and the expected sessions, restored or deliberately replaced. The Admin API example shows the request. A custom--server-versionis a label; the checksum from step 1 identifies the binary. - Evidence. The candidate opens the existing receipt store through the schema-checked path, and its audit accepts the persisted receipts.
- Governed request. One bounded allowed operation runs, one refusal explains itself, and both receipts can be retrieved and verified.
chio --receipt-db /var/lib/chio/receipts.sqlite3 receipt health --json
chio --receipt-db /var/lib/chio/receipts.sqlite3 receipt checkpoint status --json
chio --receipt-db /var/lib/chio/receipts.sqlite3 receipt audit --jsonInspect each report and exit status. receipt health samples read-only and runs no schema gate; checkpoint status and audit take the schema-checked path, and can refuse a store that still needs a writable migration. Run the Quickstart checks with a bounded test operation, read its receipt, then reopen traffic.
Keep the record: the artifact identity, the state inventory and restore rehearsal, the stop and startup logs, and the post-start checks with their receipts. Leave /v1/internal/cluster/status to cluster diagnostics; it requires cluster-peer authentication and configured peers, and can open stores writably.
A store refuses a schema newer than its binary
Each participating store compares its keyed on-disk schema revision with the revision the binary supports, and a writable open can migrate the store and advance its stamp. A binary that finds a newer stamp refuses to open the store:
database schema version {found} is newer than this binary supports ({supported}); refusing to openThe gate keeps a binary from reading state newer than it supports; it never downgrades a store. Releases can share a store revision, so a newer binary does not always mean a newer schema. The gate is per store: a receipt health sample skips it, and session records and file-backed policies, registries, and seeds follow their own validation, so one stamped database says nothing about the whole node.
Roll back the binary and its state together
If acceptance fails, keep traffic stopped, keep the candidate’s logs and post-upgrade evidence, and classify the failure before you change any state. Restoring the snapshot discards admissions, revocations, budgets, sessions, and receipts written after the snapshot.
- Stop the candidate and confirm its processes no longer own the state.
- Select the previous qualified binaries and the matching recovery set.
- Restore through the procedure you rehearsed for those store identities and protected anchors, with schema stamps and rollback checks left as they are.
- Restart with the original configuration and repeat the authority, evidence, session, and governed-call checks.
- Reconcile effects and unresolved operations before you allow retries or reopen traffic.
A binary-only rollback needs the previous binary to read every stored format and runtime contract the candidate left behind; after a migration, establish that first. A plain copy of a joint authority database can be rejected when its protected anchor or file identity no longer matches. In-memory OAuth codes and notification replay state end with the old process. Rotate authority keys, change the peer set, or move storage as separate changes, each with its own plan.
Release lanes, unit paths, and service arguments
Try quorum-heal, --session-db, or ReadWritePaths.
Release qualification lanes
Every release passes the bounded qualification lane and the trust-control drills before it is tagged; chio release verify --evidence qualification shows their results for the release you are moving to. The trust-control lane runs seven named checks: node-identity, quorum-heal, stale-leader-fencing, denied-event-replication, replay-and-failover, duplicate-event-rejection, and stale-lease-rejection. The drills describe what the first five exercise. The release inputs check fails a release that ships a generated or cached path, and pins the release evidence keys against the release manifest.
Promote the binary chio release verify accepted: a binary rebuilt from the same tag on another host is a different artifact, and nothing qualified it. A team that builds Chio itself runs the lanes from the Contributors qualification guide.
Unit paths
Both shipped units point ExecStart at /usr/local/bin/chio, and the MCP edge unit points at a separate /usr/local/bin/chio-mcp-upstream for the wrapped server. Both paths are outside ReadWritePaths, which is set to the state directory alone, so the swap happens from outside the unit.
Service arguments
Step 5 and the fourth rollback step both restart with the same command line, because several flags are invariants. TrustServiceConfig::validate refuses a joint authority database alongside --budget-db or --revocation-db (a joint authority database replaces separate budget and revocation databases), refuses one alongside --peer-url, and runs validate_distinct_database_paths over all six configured paths. It opens no database, so each failure is a fast argument error that says nothing about the state on disk.
The two meanings of --session-db
On chio mcp serve-http, --session-db is the remote session tombstone database, which carries no schema stamp. On chio trust serve, the same flag becomes joint_authority_db_path, which goes through SqliteAuthorityStore::provision and open_serving: a provisioned, lock-anchored, schema-checked store that validates nine store keys at provision time. Copying a command line between the two subcommands puts a tombstone database where a joint authority database belongs.