Node Upgrade

An upgrade is a stop, a swap, and a start

There is no in-place upgrade in Chio. No process reloads a binary, no command migrates state ahead of a restart, and nothing coordinates a version change across a cluster. The upgrade procedure in docs/release/OPERATIONS_RUNBOOK.md is nine numbered steps that stop the processes, replace the file on disk, and start them again in a fixed order. Every schema change rides the first writable open of each store after the swap.

Read the ceiling before planning a window. The runbook’s Bounded Operational Profile states trust-control as local or leader-local single-writer truth with deterministic leader selection and eventual repair, explicitly not consensus-backed HA. Nothing in that profile supports a rolling upgrade in which two binary versions serve the same state concurrently, and no test drives one. Plan a window in which the node is down.

Why a downgrade is safe to attempt

The reason step 7 is worth attempting in reverse is that a Chio store written by a newer binary does not open at all under an older one. It is refused, by name, before a single write touches the file. The gate is check_schema_version, and it runs on every store’s open path.

#[error(
    "database schema version {found} is newer than this binary supports ({supported}); refusing to open"
)]
FutureSchema { found: i32, supported: i32 },

The module header states the intent directly: a foreign, misdirected, or future database is refused before any write, so a rollback to an older binary or a mistargeted path is caught at open rather than after data has been commingled or a newer schema misread. Two properties make that a usable operator guarantee rather than a hope.

• A refusal never mutates the file. future_schema_is_refused_without_mutation asserts the recorded revision is unchanged after a refused open. A table-driven case sweeps every binary revision 0 through 3 against every on-disk revision 0 through 5 and asserts the same thing at every point where the disk is ahead, and asserts that an accepted open never downgrades the stamp either.
• The check runs before the pragmas. The receipt store’s open path calls the gate ahead of configure_sqlite_connection with a comment saying why: a foreign or future database must be refused before any write touches its header, so a mistargeted path is never mutated into WAL mode.

The revision is per store, not per file. The database-wide PRAGMA user_version cannot represent independent revisions for stores the sidecar co-locates in one file, so each store records its own under a stable key in chio_store_schema_versions. These are the ones an operator backs up and restores.

The revisions move independently, which is the whole point of the keyed table. co_located_stores_track_independent_schema_versions bumps a receipt revision inside a file an approval store also occupies, asserts PRAGMA user_version stayed 0, and reopens both stores at their own revisions. A release that touches nothing a given store persists leaves that store’s revision alone, and a downgrade across such a release works.

Steps 1 and 2: qualify the candidate

Run the production qualification lane from the repo root, on the candidate commit, before anything touches a host.

./scripts/qualify-release.sh

That script opens with ./scripts/ci-workspace.sh, the fast regression gate, then runs the provider replay conformance tests and calls ./scripts/qualify-trust-control.sh, qualify-portable-browser.sh, qualify-mobile-kernel.sh, and the four release checks. It requires node and python3 on PATH and exits 1 without either. Everything it produces lands under target/release-qualification/ with a SHA256SUMS and an artifact-manifest.json.

For the ship-facing bounded gate specifically, the runbook names two commands:

cargo xtask qualify bounded-chio
./scripts/qualify-trust-control.sh

qualify-trust-control.sh is seven named checks, each one a single test invoked with --nocapture and tee’d to its own log: node-identity, quorum-heal, stale-leader-fencing, denied-event-replication, replay-and-failover, duplicate-event-rejection, and stale-lease-rejection. The logs are the evidence, not the exit code. The first five are chio-cli cluster drills, and each one probes a loopback bind first and returns green without running when the environment denies it; the last two are chio-store-sqlite tests with no such probe. Read the drills for what each of the five exercises.

Deploy-time smoke checks are a separate, shorter list, and they are the ones worth running on the machine that builds the package:

./scripts/check-release-inputs.sh
./scripts/check-dashboard-release.sh
./scripts/check-chio-ts-release.sh
./scripts/check-chio-py-release.sh
./scripts/check-chio-go-release.sh

check-release-inputs.sh walks git ls-files and fails if any generated or cached path is tracked: __pycache__, the Python build and egg-info trees, the TypeScript dist and node_modules, the dashboard’s dist and node_modules, and the generated conformance results and reports. It then pins release evidence keys and a readiness-package SHA-256 against releases.toml. It says nothing about the binary.

Step 3: back up every store

Stop write traffic or open the maintenance window before this step, not at step 4. Step 3 delegates to the runbook’s backup procedure, and that procedure opens with the precondition: stop write traffic or place the service in a maintenance window before taking authoritative backups. Every durable store opens in WAL mode, so a plain cp of the .sqlite3 file is not a backup. Use SQLite’s own backup, which is what the runbook does:

sqlite3 /var/lib/chio/receipts.sqlite3 ".backup '/var/backups/chio/receipts.sqlite3'"
sqlite3 /var/lib/chio/revocations.sqlite3 ".backup '/var/backups/chio/revocations.sqlite3'"
sqlite3 /var/lib/chio/authority.sqlite3 ".backup '/var/backups/chio/authority.sqlite3'"
sqlite3 /var/lib/chio/budgets.sqlite3 ".backup '/var/backups/chio/budgets.sqlite3'"
sqlite3 /var/lib/chio/verifier-challenges.sqlite3 ".backup '/var/backups/chio/verifier-challenges.sqlite3'"
sqlite3 /var/lib/chio/edge-sessions.sqlite3 ".backup '/var/backups/chio/edge-sessions.sqlite3'"

Then the file-backed inputs, which no SQLite command covers and no schema stamp protects:

cp /etc/chio/enterprise-providers.json /var/backups/chio/
cp /etc/chio/verifier-policies.json /var/backups/chio/
cp /etc/chio/certifications.json /var/backups/chio/
cp /etc/chio/*.yaml /var/backups/chio/

Record the binary version and git commit used for the snapshot. That pairing is the only thing that tells a later operator which binary the backed-up stamps correspond to, and the rollback procedure needs both halves. chio --version answers the first half from the binary itself: Cli carries #[command(version, about)], so clap renders the crate version.

A receipt store is worth checking before it is copied, not after. Run the deep audit with the process stopped, or on a copy:

chio receipt health --receipt-db /var/lib/chio/receipts.sqlite3
chio receipt checkpoint status --receipt-db /var/lib/chio/receipts.sqlite3
chio receipt audit --receipt-db /var/lib/chio/receipts.sqlite3

What a clean store looks like on the second and third of those, against a database holding two committed entries and no checkpoint yet. Read status: healthy and the pending range; a store you can copy is one that reports its own uncheckpointed tail:

$ chio receipt checkpoint status --receipt-db ./chio.db
status: healthy
committed_entry_seq: 2
checkpoint_seq: none
checkpointed_entry_seq: 0
next_range: 1..=2
retention_watermark_entry_seq: none

$ chio receipt audit --receipt-db ./chio.db
status: healthy
committed_entry_seq: 2
checkpoint_seq: none
checkpointed_entry_seq: 0
next_range: 1..=1
retention_watermark_entry_seq: none

The three do genuinely different amounts of work. health samples a SQLITE_OPEN_READ_ONLY connection through receipt_store_health_read_only, which never creates the file and never opens a writer pool. checkpoint status and audit go through open_existing, which does run the schema gate. All three exit non-zero when the report is not healthy, and all three refuse --control-url with requires local --receipt-db; remote receipt operator operations are not supported in this release. Run them on the node that owns the database.

Steps 4 and 5: stop traffic, then stop the process

Stopping write traffic and stopping the process are two steps because the drain is bounded. Every service installs a SIGTERM handler and drains in-flight requests before it exits, capped by DEFAULT_DRAIN_TIMEOUT, 25 seconds. Anything still running at 25 seconds is abandoned.

/// Wall-clock ceiling on the drain: how long to wait for in-flight requests to
/// finish after the listener stops accepting. Operators must size the unit
/// `TimeoutStopSec` at least this high plus a flush margin.
pub const DEFAULT_DRAIN_TIMEOUT: Duration = Duration::from_secs(25);
/// Per-request processing ceiling before the request is denied with 408.
///
/// This stays strictly below [`DEFAULT_DRAIN_TIMEOUT`] so a request admitted just
/// before a stop signal reaches its own 408 and completes cleanly within the
/// drain window ...
pub const DEFAULT_REQUEST_TIMEOUT: Duration = Duration::from_secs(20);

The 20 second request ceiling is not universal, and trust-control is the exception. serve_async builds its hygiene config with request_timeout: None on purpose: a clustered budget authorize parks in a rollback-aware quorum wait that is bounded on its own and can legitimately outrun any single request ceiling, and a blanket timeout firing after the local exposure write but before that wait returns would leave a charged write the client only saw fail. The consequence for an upgrade is that a trust-control request can occupy the full drain window rather than self-terminating at 20 seconds, which is another reason step 4 comes before step 5.

Under a supervisor, the shipped units are already sized for it. Both docs/release/systemd/chio-trust-control.service and chio-mcp-edge.service set KillSignal=SIGTERM and TimeoutStopSec=35s, the drain deadline plus a flush margin, so systemd escalates to SIGKILL only after the drain can finish. The deploy contract in the runbook is to set the platform stop grace at least as high: Cloud Run timeoutSeconds, ECS stopTimeout, or Kubernetes terminationGracePeriodSeconds at 35 seconds or more.

The edge unit adds KillMode=mixed, and the reason is specific to the wrapped command after --. The upstream tool server lives in the same control group; under systemd’s default control-group kill mode it would take SIGTERM at the same instant the edge began draining, tearing the upstream out from under in-flight MCP calls. mixed signals the main process first and escalates to the rest of the group only after TimeoutStopSec. If you supervise the edge with something other than systemd, reproduce that behavior or expect the last 25 seconds of calls to fail against a dead upstream.

On trust-control there is a second consumer of the same budget. The cluster sync loop watches the same shutdown signal, and after the HTTP drain returns the process joins that loop within whatever remains of the drain window rather than starting a fresh wait. cluster_join_budget is drain_timeout.saturating_sub(elapsed_since_signal), and cluster_join_never_extends_teardown_past_the_drain_window asserts the sum stays inside one 25 second window at every sample point. A stop grace sized for the drain is therefore sized for the whole teardown. Outbound peer sync abandoned at the deadline is best-effort catch-up that resumes on the next boot.

Step 6: swap the binary

Replace the binary while nothing is running. The shipped units both point ExecStart at /usr/local/bin/chio, and the MCP edge unit points at a separate /usr/local/bin/chio-mcp-upstream for the wrapped server. Both paths are outside ReadWritePaths, which is set to the state directory alone, so a running service cannot rewrite its own binary and the swap has to happen from outside the unit.

Nothing in the swap changes arguments. Step 4 of the restore procedure and step 7 of the upgrade procedure both say to restart with the same command line the process had before, and the reason is that several flags are load-bearing invariants rather than preferences. TrustServiceConfig::validate refuses a joint authority database alongside --budget-db or --revocation-db (a joint authority database replaces separate budget and revocation databases), refuses one alongside --peer-url, and runs validate_distinct_database_paths over all six configured paths. It opens no database, so every one of those failures is a fast argument error and none of them tells you anything about the state on disk.

Step 7: trust-control first, edges second

The order is not a preference. The onboarding runbook states it as a precondition (chio trust serve starts first; chio mcp serve-http starts after trust-control readiness) and the code makes it a hard dependency whenever the edge carries --control-url: RemoteSessionFactory::new constructs a DurableAdmissionRuntime::open_remote against that URL, requires --control-token, and requires --session-db for the identity path. That construction happens during startup, before the router is built, so an edge pointed at a trust-control that has not come back is a failed start, not a degraded one.

systemctl start chio-trust-control.service
# confirm /health and /v1/authority answer before continuing
systemctl start chio-mcp-edge.service

What each process opens at startup differs, and it decides where a schema refusal shows up.

The edge’s startup row is the one operators misread. Session restore is not a lazy path: serve_http_async loads every persisted active session and calls restore_session on each, and that builds a kernel and opens all four stores. A restore that fails does not fail the boot. It logs dropping persisted MCP session record that could not be restored and deletes the record, so a store the edge binary refuses costs you the persisted sessions rather than the process.

A future-schema budget or revocation database therefore kills chio trust serve outright, and the error text is worth knowing because it is what appears in the journal. The store error is wrapped once by the runtime: failed to open trust-control budget store: ... and failed to open trust-control revocation store: ..., each carrying the database schema version N is newer than this binary supports (M) text inside it.

Step 8: the smoke checks, and what each one proves

The runbook’s post-upgrade set is four curls. It swaps the /v1/authority call used at first deployment and after a restore for /v1/internal/cluster/status, which is the one substitution worth undoing: keep both.

curl -s http://127.0.0.1:8940/health | jq
curl -s -H "Authorization: Bearer $CHIO_SERVICE_TOKEN" \
  http://127.0.0.1:8940/v1/internal/cluster/status | jq
curl -s -H "Authorization: Bearer $CHIO_ADMIN_TOKEN" \
  http://127.0.0.1:8931/admin/health | jq
curl -s -H "Authorization: Bearer $CHIO_ADMIN_TOKEN" \
  http://127.0.0.1:8931/admin/sessions | jq

The third curl is reproduced from the runbook, and a bare service-token bearer does not satisfy it. handle_internal_cluster_status calls validate_cluster_peer_auth before anything else, which requires x-chio-cluster-node-id, x-chio-cluster-auth-issued-at, and x-chio-cluster-auth-signature, a caller already listed in --peer-url, and a signature that matches the one derived from the service token. Anything else is 401 missing or invalid cluster peer authentication or 403 cluster peer is not in the configured allowlist. Past that gate, a node with no replication configured answers 404 cluster replication is not configured. On a single node, run the other three checks and the receipt commands below.

$ curl -si -H "Authorization: Bearer $CHIO_TRUST_SERVICE_TOKEN" \
    http://127.0.0.1:8940/v1/internal/cluster/status
HTTP/1.1 401 Unauthorized
content-type: application/json
www-authenticate: chio.cluster.peer.v1
content-security-policy: default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; connect-src 'self'; img-src 'self' data:
content-length: 58

{"error":"missing or invalid cluster peer authentication"}

The substitution to undo, in full. Both of these answer to the service token, and the second is the one that surfaces a refused authority store:

$ curl -s http://127.0.0.1:8940/health | jq '{ok, clustered, stores}'
{
  "ok": true,
  "clustered": false,
  "stores": {
    "budgetsConfigured": true,
    "receiptsConfigured": true,
    "revocationsConfigured": true,
    "verifierChallengesConfigured": false
  }
}

$ curl -s -H "Authorization: Bearer $CHIO_TRUST_SERVICE_TOKEN" \
    http://127.0.0.1:8940/v1/authority | jq
{
  "configured": true,
  "backend": "sqlite",
  "publicKey": "8d7ddbdce2368deb5143de554fb574fea652fe3e5610acdb2b1be110083fe485",
  "generation": 1,
  "rotatedAt": 1785603453,
  "appliesToFutureSessionsOnly": true,
  "trustedPublicKeys": [
    "8d7ddbdce2368deb5143de554fb574fea652fe3e5610acdb2b1be110083fe485"
  ]
}

Read each one for what it actually opens, because the coverage is uneven.

The stores block on both health routes is the trap. On trust-control it is four booleans, receiptsConfigured, revocationsConfigured, budgetsConfigured, and verifierChallengesConfigured, computed as is_some() over the configured paths. On the edge it is six booleans, and two of them, revocationsConfigured and budgetsConfigured, go green on --control-url alone, with no local database behind them. Not one of them opens a file. A green stores block means the flags were passed, which the process could have told you before it started.

The one field on either route that identifies the binary is server.serverVersion on /admin/health. It defaults to env!("CARGO_PKG_VERSION") and is overridden only by an explicit --server-version. If your unit passes that flag, the field reports your string and proves nothing about the swap; drop the flag if you want the route to confirm which binary is serving.

Add the receipt checks. They are the only ones in the set that open the store the upgrade most likely migrated, and they run the schema gate on the way in:

chio receipt health --receipt-db /var/lib/chio/receipts.sqlite3 --json
chio receipt checkpoint status --receipt-db /var/lib/chio/receipts.sqlite3 --json
chio receipt checkpoint verify --receipt-db /var/lib/chio/receipts.sqlite3 --json

$ chio receipt health --receipt-db ./chio.db --json
{
  "report": {
    "checkpointError": null,
    "dbSizeBytes": null,
    "healthy": true,
    "latestCheckpointSeq": null,
    "latestCheckpointedEntrySeq": 0,
    "latestCommittedEntrySeq": 2,
    "retentionError": null,
    "retentionWatermarkEntrySeq": null,
    "uncheckpointedEndSeq": 2,
    "uncheckpointedStartSeq": 1,
    "writer": {
      "acceptedTotal": 0,
      "committedTotal": 0,
      "failedTotal": 0,
      "firstAcceptUnixMs": null,
      "inflight": 0,
      "lastCommitUnixMs": null,
      "lastError": null,
      "queueDepth": 0,
      "saturatedTotal": 0,
      "timedOutInflight": 0,
      "timedOutTotal": 0
    },
    "writerLevel": "healthy",
    "writerLiveness": "unknown",
    "writerRestartTotal": 0
  },
  "schema": "chio.cli.receipt.health.v1"
}

Each emits a two-field envelope, schema and report, under a stable name: chio.cli.receipt.health.v1, chio.cli.receipt.checkpoint_status.v1, and chio.cli.receipt.checkpoint_verify.v1. checkpoint verify is a documented compatibility alias for chio receipt audit that never passes --repair and keeps emitting the legacy envelope name so existing automation does not break. Both exit non-zero when the chain or projection integrity fails, with the report’s checkpoint_error as the message.

Rollback is binary and state together

The runbook says it in one line: rollback is a full binary-and-state rollback to the last known good backup. Six steps, and the third is the one operators skip.

1. Stop the candidate processes.
2. Restore the previous binaries.
3. Restore the backed-up SQLite and registry files if the candidate performed writes that must be discarded.
4. Restart the previous version with the original arguments.
5. Re-run the same health and admin smoke checks used in the upgrade.
6. Record the failed candidate commit and attach the qualification logs and any cluster or admin diagnostics to the incident report.

Step 3 is conditional in the runbook and unconditional in practice whenever a store’s revision moved, because the candidate performed a write the moment it opened that store: the re-stamp. Try the binary-only rollback first if you like. It either works, meaning no store the old binary reads had its revision bumped, or it refuses at open with a message naming both numbers, and you have lost nothing but the restart. That is the whole value of the stamp during a rollback: the cheap attempt is safe, and its failure is unambiguous.

Restoring state has its own ordering. chio receipt audit --repair belongs here, between the restore and the restart, with every process stopped, because it cannot reach a live writer head. In a cluster, restoring one node’s stores is not the end of it: the restored node rejoins through the same full-snapshot path every repair uses, and the force-resync decision governs when that happens. A snapshot body over the 64 MiB peer response cap fails to decode on the receiving node, which is why the snapshot path range-encodes abandoned sequences and the delta path pages its bodies rather than emitting unbounded lists. If a resync does hit the cap, the node-local restore path is the way back.

What this procedure does not do

• It does not upgrade a cluster without downtime. One stream negotiates a contract between peers, the revocation stream, and a node at the current cursor version degrades to the tuple projection against an older peer rather than failing. Nothing else does: there is no read-only mode and no drain-to-follower step, and cluster convergence resumes on reachability alone once a node is back.
• It does not migrate anything ahead of time. There is no chio migrate. Migrations run inside the first writable open of each store, in-process, on the startup path. The only way to rehearse one is to open a copy with the candidate binary.
• It does not verify the binary. Nothing in steps 6 through 8 checks a signature, a digest, or a provenance attestation on the file that was installed. Guard Supply Chain covers what is verified, and it is guard bundles rather than the chio binary.
• It does not protect unstamped state. The session tombstone database, the JSON registries, the policy YAML, and the kernel seed file all carry the version of whatever last wrote them and say nothing about it.
• It does not roll forward. A store that a newer binary has migrated cannot be handed back, and nothing writes a lower revision. Forward is a restart; backward is a restore.

Guarantees and limits

Next Steps

• Node Lifecycle · the drain this runbook waits on: the signal path, the deadline, and the refusals that stop a boot
• Node State on Disk · the stamp in full: anchor tables, adoption of unstamped files, and why WAL makes a file copy the wrong backup
• Backup & Restore · backup tiers, recovery objectives, and the checkpoint chain a restored receipt log rejoins
• Cluster Recovery · what a restarted node does to rejoin, and the resync that repairs it
• Health & Readiness · the full contract behind the two routes step 8 curls