Cluster Upgrade

The qualified binary set runs on every trust-control node and edge, against the stores and custody they had before; the peers have repaired, and a governed workflow has passed its checks before traffic reopens.

Authority keys and fences decide who may issue grants; budgets, revocations, receipts, and lineage keep the history of governed work. The procedure stops the fleet, captures one recovery boundary, replaces the binaries, and restarts trust control before its edges.

One recovery boundary separates the two releases

Every writer stops before the backups, so the boundary is one point the whole fleet can return to.

The procedure replaces the stopped fleet as one release operation. Quorum during a partial stop is a local availability condition, so a mixed-version rolling upgrade runs only when the release qualifies that exact transition.

1. Qualify the exact candidate

Run the release gate in the candidate’s tagged source from backbay-labs/chio before the window, then build or obtain the exact artifacts you will install.

./scripts/qualify-release.sh

Keep the manifest, checksums, and logs with the release record, and confirm from the logs that the required drills ran; a drill can pass where its environment skips it. Node Upgrade details the release lanes, and Cluster Recovery lists the repair scenarios the cluster drills exercise.

Rehearse opening a copy of the deployment’s stores with the candidate and restoring it with the previous build, because qualification covers one build and sets no contract between arbitrary pairs. Review the replication formats and recovery behavior of that exact release pair as well. A cursor rule or a passing snapshot on one lane leaves authority, budget history, receipts, and the other lanes to their own checks.

2. Record membership, custody, and the recovery inventory

Record every advertised URL, configured peer URL, authentication setting, service argument, and state path. Each node builds its peer entries at startup and authenticates peers against its configured URL list, so a membership change runs as its own operation, apart from the release.

Clustered trust control refuses a seed-file authority and directs the operator to the authority database. Preserve that database and the key-custody arrangement together: a replicated public authority snapshot carries no signing seed for a returning node, as Authority & Rotation explains.

• Previous and candidate artifacts. Version, source revision, and checksum for each binary; a matching version string can name different bytes.
• Persistent stores. Authority, receipts, revocations, budgets, verifier challenges, edge sessions, and every other store the deployment enables.
• Policy, registries, and arguments. The files that rebuild the intended policy, providers, verifiers, certifications, peers, and paths.
• Identity and custody. The approved restoration procedure and any state kept outside the database copies, including rollback-protection requirements.
• A named boundary. The copies tied to this maintenance operation, with the writers that were stopped; separate live-file copies give no fleet-wide point in time.

Follow Backup & Restore for the copies and Node State for the store inventory.

3. Drain callers, stop writers, and back up

1. Close traffic. Stop new autonomous work and external writes. Account for callers with outstanding requests and any worker that can still write to the stores.
2. Stop the processes. Use each service’s normal shutdown path and wait for process exit. Inspect forced shutdowns and unresolved request outcomes.
3. Back up. Take the authoritative copies, retain the full inventory, and keep the previous binaries in place.

The standard HTTP drain budget is 25 seconds after the shutdown signal, and trust control joins its cluster sync task within the rest of that budget. Give the service manager a stop allowance covering the drain plus the flush margin in Node Upgrade.

A deadline produces a forced drain, and a detached connection handler can outlive the dropped serve future. Confirm process exit before you replace a binary or restore its stores, and reconcile every request whose caller got no conclusive result.

Keep candidate and previous processes off the same state during the transition. Store ownership and qualified compatibility make the transition safe; SQLite WAL mode alone qualifies no pair of versions.

4. Restart trust control, then its edges

Install the qualified binary set and restart the trust-control nodes with the recorded arguments, state, and custody. Start the dependent MCP HTTP edges once their control-plane dependencies are ready, and keep caller traffic closed while you check the fleet.

Opening a store runs its schema check, and the outcome depends on the stored revision:

• Newer than the binary supports. The check refuses to open the store. Restore a compatible recovery set or use a binary that supports the state; editing the revision stamp migrates nothing.
• A receipt store behind the binary. The writable open path runs the supported migration. The existing-store open path refuses the lower revision and points to a writable open.
• Unstamped. The shared helper adopts the file only when it is empty or carries a recognized store anchor.
• The same revision on both binaries. This avoids the future-schema refusal; serialized records, policy behavior, and external integrations still need the release pair’s own qualification.

Revisions are kept per store in chio_store_schema_versions, so co-located stores move independently. Compare the exact release pair against the actual files.

Quorum and leadership come back as each node’s local view

For a node configured with N distinct members including itself, the local threshold is ceil((N − 1) / 2) + 1, the same rule as peers.div_ceil(2) + 1. A peer counts while it is reachable, unpartitioned, and freshly contacted, so a stopped process becomes visible through those observations over time.

With matching membership on every node and quorum, a node selects the lexicographically lowest URL in its eligible set, and any change of selection increments its local election term. Contact histories differ from node to node, so the order in which you start the nodes leaves the selected leader open; read it from each node’s health.

New peer entries begin with unknown health, empty cursors, and a required snapshot, and a status probe can restore reachability before the snapshot applies. Check each affected peer’s snapshot progress and the records the workflow needs, as in Cluster Recovery; hasQuorum shows routing, never catch-up.

5. Verify before reopening traffic

Run the smoke checks against every affected service. These use the runbook’s local ports and token variables; substitute your listeners.

curl --fail-with-body -sS http://127.0.0.1:8940/health
curl --fail-with-body -sS -H "Authorization: Bearer $CHIO_SERVICE_TOKEN" \
  http://127.0.0.1:8940/v1/authority
curl --fail-with-body -sS -H "Authorization: Bearer $CHIO_ADMIN_TOKEN" \
  http://127.0.0.1:8931/admin/health
curl --fail-with-body -sS -H "Authorization: Bearer $CHIO_ADMIN_TOKEN" \
  http://127.0.0.1:8931/admin/sessions

You should see each request succeed. The internal cluster-status route takes cluster peer authentication instead of a service bearer token. Then collect the acceptance evidence:

1. Artifact and configuration. The installed checksum and the intended arguments, policy, paths, and custody on each node.
2. Authority and routing. Each node’s local quorum, selected leader, term, and authority state; resolve any disagreement before relying on mutations.
3. Peer repair. New snapshot progress on the affected peers plus the expected records.
4. Governed work. One allowed request and one deliberate policy refusal through the serving path, each with its receipt.
5. Operation-specific state. Budget acknowledgments, revocation behavior, or restored sessions for every enabled workflow.

Reopen traffic once these checks pass, and record unresolved outcomes and release-specific differences.

Rollback returns the binaries and the stores to the boundary

1. Stop. Close traffic and stop the candidate processes. Keep their diagnostics and list the writes and external actions after the backup boundary.
2. Restore. Reinstall the previous binary set. Restore the backed-up SQLite files, registries, and policies when candidate writes must go or the previous binary cannot read the candidate’s state.
3. Recover custody. Apply the procedures for identity, custody, and rollback-protected state, with the original arguments and paths.
4. Restart and check. Restart trust control and its edges, repeat the acceptance checks, and keep the failed candidate’s artifact identity and logs.

A restored database leaves completed external work in place: a payment or a delivered artifact stays done, so reconcile post-backup work before reopening writers. Surviving peers can still carry newer records, and snapshot repair applies authority state, upserts revocations and lineage, appends receipts, and imports budget history by each store’s rules; it never rewinds the fleet to the first snapshot that arrives.

Next: upgrade a single node