Protect durable state

Back up, archive, restore, and move Chio's authority-bearing stores without reviving a revoked grant, a consumed nonce, or a spent allowance. Each store gets private storage, each rollback anchor gets its own device, and a rehearsed restore shows the kernel refusing a stale snapshot.

A signature authenticates a record; an independently kept anchor shows the record is the current continuation. Restore the stores, keys, serving-owner state, and anchors as one relationship, with one serving owner, before admitting new work.

1. List the stores that hold authority

Served through SqliteAuthorityStore, budget, revocation, and admission operation state are tables of one authority database, under one global commit chain and one serving rollback anchor. That anchor is a separate file under the store's lock root. The receipt, execution nonce, security state, and key log stores are each their own database.

Give each store a distinct absolute path with private permissions, back it up consistently with its state contract, and retain the session bindings and release checkpoints the host requires; an ephemeral development store carries no restart contract. Backups see more state than any running service, so store access, decryption authority, and permission to activate a restore are separate responsibilities.

2. Put every rollback anchor on its own device

When a receipt store or a qualified finding-pool ledger opens with a rollback generation anchor root, that root must live on a separate device from the database it protects; two paths on one device roll back together. Mount it independently, outside database snapshots, as the reference runtime does with /srv/chio-receipt-anchors.

pub(crate) struct RollbackGenerationAnchor {
    file: File,
    root: PathBuf,
    path: PathBuf,
    expected_device: u64,
    expected_inode: u64,
    scope_sha256: String,
    process_lock: Arc<Mutex<()>>,
}

You should see the store open only when its generation matches the anchor's protected generation. The serving owner's anchor also keeps a retired export retired: DatabaseState.retired_export_id is derived from relocation state on every load, and each anchor write copies it into AnchorRecord.retired_export_id. Neither anchor is the economy's chio-anchor.

3. Archive receipts with a verified rotation

Receipt store schema version 5 rebuilds the security evidence index without its foreign key to the receipt table, so a receipt row can be archived and removed with no live constraint broken. A retention archive refuses a recorded schema version newer than the binary supports, and completing a rotation verifies every archived table against the live prefix, row for row, aborting the delete on any shortfall.

If a rotation leaves the store unopenable, repair it from its archive; the command removes claim-log rows whose source receipts were already archived and deleted:

chio receipt retention repair --receipt-db "$RECEIPT_DB" --archive "$ARCHIVE"

4. Rehearse the broker restoration in dependency order

1. Isolate. Restore away from production callers with the intended service UID, private directory ownership, and the retained migration anchors.
2. Restore. Bring back the broker's four databases, secret, attempt, admin replay, and receipt, from one consistent backup set, then provision the matching tenant master key and broker signing seed through the supervisor's sealed descriptors.
3. Reconcile. Start the authoritative hold service before the broker, and let broker startup reconcile each restored attempt under its original identity.
4. Exercise. Send one permitted request from a test provider account and one to the nearest unauthorized destination.

You should see the permitted request succeed with signed evidence and a provider-side observation, the other refused, and the tool still without a direct credential path. Record the signing identity and migration heads. A startup refusal names a wrong key, stale migration head, unknown socket owner, or unavailable authority; keep the state, since deleting it erases the evidence that prevents duplicate credential use.

5. Relocate a stopped process host

chio process export permanently retires serving at the source and writes the manifest a copy is checked against; chio process import re-anchors the copy and verifies it against that manifest. Rehearse on the disposable host from the containment rehearsal: one committed mailbox message and a withdrawn worker.

#!/usr/bin/env bash
# Selected stopped-host relocation. Export retires the source; never use a live host.
set -eu
umask 077
: "${CHIO_STATE:?Set the existing stopped disposable host directory}"
: "${CHIO_RESTORE:?Set a new absolute destination directory}"
: "${CHIO_EVIDENCE:?Set a new absolute evidence directory outside the host}"
case "$CHIO_RESTORE" in /*) ;; *) echo 'Destination must be absolute' >&2; exit 2;; esac
case "$CHIO_EVIDENCE" in /*) ;; *) echo 'Evidence directory must be absolute' >&2; exit 2;; esac
python3 "$(dirname "$0")/relocate_check.py" paths "$CHIO_STATE" "$CHIO_RESTORE" "$CHIO_EVIDENCE"
if [ -e "$CHIO_RESTORE" ] || [ -e "$CHIO_EVIDENCE" ]; then
  echo 'Refuse to overwrite a destination or evidence directory' >&2
  exit 2
fi
mkdir -m 0700 "$CHIO_EVIDENCE"
chio process status --state "$CHIO_STATE" > "$CHIO_EVIDENCE/before-status.json"
chio receipt list --receipt-db "$CHIO_STATE/receipts.db" --admin-all --limit 100 \
  > "$CHIO_EVIDENCE/before-receipts.jsonl" 2> "$CHIO_EVIDENCE/before-page.stderr"
chio process export --state "$CHIO_STATE" > "$CHIO_EVIDENCE/export.json"
cp "$CHIO_STATE/relocation.json" "$CHIO_EVIDENCE/original-relocation.json"
cp -a "$CHIO_STATE" "$CHIO_RESTORE"
chio process import --state "$CHIO_RESTORE" > "$CHIO_EVIDENCE/import.json"
chio process status --state "$CHIO_RESTORE" > "$CHIO_EVIDENCE/after-status.json"
chio receipt list --receipt-db "$CHIO_RESTORE/receipts.db" --admin-all --limit 100 \
  > "$CHIO_EVIDENCE/after-receipts.jsonl" 2> "$CHIO_EVIDENCE/after-page.stderr"
# This compares the selected first page, not proof of complete receipt history.
cmp "$CHIO_EVIDENCE/before-receipts.jsonl" "$CHIO_EVIDENCE/after-receipts.jsonl"
python3 "$(dirname "$0")/relocate_check.py" identity "$CHIO_EVIDENCE"

Download relocate_stopped_host.sh and relocate_check.py into one directory. Point CHIO_STATE at the stopped rehearsal's .chio/cancellation/host and the other variables at new absolute directories outside it; the script refuses overlap and an existing destination.

export CHIO_STATE="$PWD/.chio/cancellation/host"
export CHIO_RESTORE="$HOME/chio-restored-host"
export CHIO_EVIDENCE="$HOME/chio-relocation-evidence"
bash relocate_stopped_host.sh

IMPORT original store/export identity; selected receipt page unchanged
SOURCE retired; review restored cancellation, revocation and outstanding operations before serving

You should see both lines: the import keeps the original store_uuid and export_id, and the first receipt page is byte-identical. The script compares that bounded page and keeps both pagination logs; continue each cursor where more remain. This relocates a stopped host; it is never a live SQLite backup.

6. Verify that withdrawn authority stays withdrawn

Each receipt check reads the local database named by --receipt-db and refuses to run without it.

chio process status --state "$CHIO_RESTORE"
chio process state --state "$CHIO_RESTORE" --process worker
chio receipt audit --receipt-db "$CHIO_RESTORE/receipts.db"
chio receipt checkpoint verify --receipt-db "$CHIO_RESTORE/receipts.db"
chio receipt health --receipt-db "$CHIO_RESTORE/receipts.db"

You should see the withdrawn worker still cancelled. In the process-host rehearsal, the cancelled credential's new mailbox operation after import is refused as cancelled with the mailbox unchanged, the exported source refuses to serve, and a copy with changed host.json bytes fails import with "relocated file changed since export: host.json." Step 4 covers broker restoration. Inspection grants nothing; review cancellation, revocation, captured work, and anchors before serving. Process operations covers export and import.

A stopped kernel restarts stopped

The emergency stop is durable state too: chio emergency-stop, from the CLI, the process-host socket, or the mounted HTTP route, appends a signed record to the hash-chained stop record in the admission store, and each deny receipt names the epoch it observed.

On boot the kernel verifies the stop heads before the reconciliation sweep and reports ready_stopped while probes see a live process; it refuses at entry and again inside the dispatching crossing. Output returned during the stop waits in durable custody until resume. chio emergency-resume needs a different principal, an offline-verified quorum, or a single-operator cooldown, signs inside its own transaction, and leaves revoked grants revoked; see emergency stop.

Rehearse restart with live reservations, captured unknown effects, a pending key rotation, overlapping responses, and a partial rollback; old serving fences fail to mutate the recovered state. Rehearse an upgrade with retained receipts, a captured operation, and an unfinished response; the new runtime reads the original evidence unchanged and keeps accounting and control ownership.

A stale, moved, or uncertain store refuses to serve

An unknown outcome poisons the serving owner and reports SqliteServingOwnerError::OutcomeUnknown, which admission operation, budget, revocation, and receipt callers receive as their store's OutcomeUnknown. The repairs:

1. Behind the anchor. Recover the matching authorized snapshot and anchor and reconcile relocation and owner history; the anchor never moves down.
2. Exported. Import at the destination with the original manifest and keep the source retired. Recover a changed copy from the matching authorized one, never by rewriting its manifest.
3. Shared device. Provision an independent anchor device and revalidate the pair before opening the store.
4. Fenced. Stop the stale owner, reacquire serving ownership, and continue the original mutation identity.
5. Outcome unknown. Stop serving through the poisoned owner and reconcile the original outcome, keeping holds until it resolves; a captured unknown effect gets no fresh dispatch or automatic refund.