Backup & Restore

A node rebuilt from backup with its receipt history intact, its checkpoint head matched against a copy you hold outside the operator, and admission closed until revocations and pins are current.

In every deployment mode (in process, sidecar, local MCP bridge, process host), a node holds SQLite stores and one kernel signing key. Copy them on three tiers and rehearse the restore before a restored node admits calls or rejoins a partner.

1. Inventory the six data sets a restore needs

Each data set is a separate backup target with its own recovery point and retention policy.

A trust-control node also keeps --revocation-db and --budget-db. You should end with each store's path, exactly as your services pass it, and a recovery point for each.

2. Take a consistent snapshot of the live receipt store

The receipt store is one SQLite database per kernel node, or one replicated across trust-control followers, opened with these pragmas:

PRAGMA journal_mode = WAL;
PRAGMA synchronous = FULL;
PRAGMA busy_timeout = 5000;
PRAGMA foreign_keys = ON;
PRAGMA auto_vacuum = INCREMENTAL;

Copy it with SQLite's online backup API, which writes a consistent, self-contained snapshot of the live database, committed WAL content included.

set -e
umask 077
mkdir -p /backups
backup_snapshot="/backups/receipts-$(date -u +%Y%m%dT%H%M%SZ).sqlite"
test -f /var/lib/chio/receipts.sqlite
test ! -e "$backup_snapshot"
sqlite3 -readonly /var/lib/chio/receipts.sqlite \
  ".backup '$backup_snapshot'"
sqlite3 -readonly "$backup_snapshot" "PRAGMA integrity_check;"

ok

You should see ok. The integrity check covers SQLite structure; step 3 checks the signed history.

• Copy through SQLite. A filesystem copy needs every writer stopped and a consistent database and WAL set. A WAL checkpoint followed by live file copies races new writes; the WAL lifecycle explains why.
• Record the boundary. With synchronous = FULL a commit is durable when it returns; a commit after the backup finished belongs to the next snapshot. Continuous point-in-time recovery needs its own capture and restore mechanism.
• Size the volume from pragmas. PRAGMA page_count times PRAGMA page_size gives the database size without a filesystem stat, consistent in WAL mode; the retention path computes it this way.
• Include retention archives. Rotation moves history into archive databases. Back up the live and archived stores together and verify their checkpoint coverage together.

3. Prove the snapshot's checkpoint chain before you keep it

Open the snapshot with the installed chio and read its checkpoint status.

$ chio --receipt-db "$SNAPSHOT" receipt checkpoint status

status: healthy
committed_entry_seq: 1834720
checkpoint_seq: 4928
checkpointed_entry_seq: 1834496
next_range: 1834497..=1834720
retention_watermark_entry_seq: none

You should see status: healthy: the command walked the stored checkpoint chain and found it intact. Record committed_entry_seq as the snapshot's recovery point; next_range is history after the last checkpoint.

# Last retained receipt sequence; this is not a row count.
$ sqlite3 -readonly "$SNAPSHOT" "SELECT MAX(seq) FROM chio_tool_receipts;"
# Last stored checkpoint; checkpoint status checks its signed chain.
$ sqlite3 -readonly "$SNAPSHOT" \
    "SELECT MAX(checkpoint_seq) FROM kernel_checkpoints;"

1834720
4928

4. Ship an encrypted copy off the host

Send the snapshot to versioned object storage for long-term retention. Configure the bucket, its permissions, the KMS key, and lifecycle retention first; this nightly cron script then uploads and keeps no local copy.

# /usr/local/bin/chio-backup.sh - run nightly via cron
set -euo pipefail

DB="/var/lib/chio/receipts.sqlite"
BUCKET="s3://larkfield-chio-backups/prod"
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
SNAP_DIR="$(mktemp -d)"
SNAP="$SNAP_DIR/receipts-$STAMP.sqlite"
trap 'rm -rf "$SNAP_DIR"' EXIT
umask 077
test -f "$DB"

# 1. Online snapshot with consistency check.
sqlite3 -readonly "$DB" ".backup '$SNAP'"
sqlite3 -readonly "$SNAP" "PRAGMA integrity_check;" | grep -qx "ok"

# 2. Request storage-service encryption under the configured KMS key.
aws s3 cp "$SNAP" "$BUCKET/receipts-$STAMP.sqlite" \
    --sse aws:kms \
    --sse-kms-key-id "$CHIO_BACKUP_KMS_KEY_ID"

# 3. Drop the local copy.
rm -f "$SNAP"

# 4. The 90-day retention rule is an S3 lifecycle policy, configured
#    once per bucket outside this script.
echo "ok: $STAMP"

ok: 20261011T020000Z

You should see ok: and the stamp. With --sse aws:kms the storage service encrypts the object, which keeps that service inside the data's trust boundary. To keep the provider out, encrypt the file under a separate KMS key before upload.

The cold tier accepts a longer recovery point, typically nightly snapshots kept 90 or 365 days. A common starting schedule keeps hourly snapshots 7 days, daily 90 days, and monthly 7 years; the receipt log is often the document of record for agent decisions, so external rules decide.

5. Anchor checkpoints outside the operator

The kernel batches receipts into signed checkpoints. Each checkpoint body commits to a Merkle root over its batch and to the SHA-256 of the previous body, so the chain is its own integrity proof.

pub struct KernelCheckpointBody {
    /// First receipt seq in this batch.
    pub batch_start_seq: u64,
    /// Last receipt seq in this batch.
    pub batch_end_seq: u64,
    /// Number of leaves in the Merkle tree.
    pub tree_size: usize,
    /// Root from MerkleTree::from_leaves.
    pub merkle_root: Hash,
    // … issued_at, kernel_key
    /// Hash of the immediately preceding checkpoint body when this checkpoint extends a prior batch.
    // … serde attributes
    pub previous_checkpoint_sha256: Option<String>,
    // … chain_root (absent on v1 checkpoints)
}

An anchored checkpoint lets a relying party verify that any later checkpoint extends it and, through a ReceiptInclusionProof, that a receipt in a covered range was in its batch. Each body also names the kernel public key in use, which a restore depends on. Fields are listed under Checkpoint body fields.

# /usr/local/bin/chio-checkpoint-anchor.sh - run weekly
set -euo pipefail

DB="/var/lib/chio/receipts.sqlite"
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
umask 077
mkdir -p /var/lib/chio/anchors
ANCHOR="/var/lib/chio/anchors/checkpoint-$STAMP.json"
test -f "$DB"
test ! -e "$ANCHOR"

# 1. Advance the checkpoint chain. The checkpoint is signed by the
#    kernel keypair loaded from the seed file.
chio --receipt-db "$DB" receipt checkpoint create \
    --kernel-seed-file /var/lib/chio/authority.seed

# 2. Confirm the chain is intact before anchoring.
chio --receipt-db "$DB" receipt checkpoint verify

# 3. Export the latest checkpoint row for the anchor.
sqlite3 -readonly -json "$DB" \
    "SELECT * FROM kernel_checkpoints ORDER BY checkpoint_seq DESC LIMIT 1;" \
    > "$ANCHOR"

# 4. Retain $ANCHOR outside the operator: an internal transparency log,
#    object storage with object lock, or a witness that co-signs checkpoints.

created: true
checkpoint_seq: 4928
checkpoint_range: 1834273..=1834496
committed_entry_seq: 1834496
checkpointed_entry_seq: 1834496

You should see created: true and the range the new checkpoint covers. Keep the anchor file where the producer cannot rewrite it.

Read more in Bilateral Receipts, where the checkpoint chain is the integrity proof between two organizations.

6. Rehearse a restore, then open admission

Restore a recent snapshot from the tier you are testing into an isolated environment, and compare its checkpoint head with the anchor you retained. Hashes are shortened.

$ cp "$SNAPSHOT" /srv/drill/receipts.sqlite
$ jq -r '.[0] | "\(.checkpoint_seq) \(.merkle_root)"' "$ANCHOR"
$ sqlite3 -readonly /srv/drill/receipts.sqlite \
    "SELECT checkpoint_seq, merkle_root FROM kernel_checkpoints
     WHERE checkpoint_seq = 4928;"

4928 7d41…e09c
4928|7d41…e09c

You should see the same root from the anchor and from the store. Bring up a kernel against the restored store, run one call through it, and extend the chain with the restored seed:

$ chio --receipt-db /srv/drill/receipts.sqlite receipt checkpoint create \
    --kernel-seed-file /srv/drill/authority.seed

created: true
checkpoint_seq: 4929
checkpoint_range: 1834497..=1834721
committed_entry_seq: 1834721
checkpointed_entry_seq: 1834721

You should see checkpoint 4929 over a range that starts right after the anchored one. A healthy receipt checkpoint status afterwards confirms that the new checkpoint's previous_checkpoint_sha256 extends the retained head. Open admission only when four checks pass:

1. Integrity. PRAGMA integrity_check prints ok and the checkpoint status is healthy.
2. Chain. The anchored root matches the store, and the next checkpoint extends the retained head.
3. Revocations. Revocation deltas from the control-plane log, and from each partner in a federated deployment, close the staleness window within your freshness budget.
4. Pins. The restored seed matches the kernel_key in the retained checkpoints, and the trusted issuer keys and partner cards come from trusted configuration.

Tear down the drill and record the measured recovery time, the surprises, and any drift between runbook and reality. Drill monthly or quarterly to start, matched to your recovery obligations, and again after any change to storage, key custody, authority, or the restore procedure.

Each tier recovers to a different point

The recovery point objective (RPO) is the history you can lose, and the recovery time objective (RTO) the time to verified service. Measure both in drills.

The hot tier is a trust-control follower holding replicated receipt and authority state. Its sync interval is a polling cadence, so its recovery point is the last completed replication. Before promotion, inspect completed replication, writer ownership, and the authority the application will use.

Read more in Trust Control Plane, which holds the coordinator's contract.

Authority and partner state catch up before admission reopens

The capability authority database holds issued capabilities, their lineage, and the revocation list. A restored copy starts at the snapshot boundary and catches up from the authoritative log.

• Revocation freshness budget. A 24-hour-old snapshot leaves up to 24 hours of revocation deltas to replay from partners or the control-plane log; size snapshot cadence for that window.
• Lineage travels with capabilities. Retain capabilities and their delegation lineage together. Verification after restore also needs the intended issuer pins, current revocations, the validity window, and the other dispatch preconditions.
• Issuance is reassessed. Keep or replace each capability after checking expiry, revocation, signer continuity, and lost authority state; validity at snapshot time approves nothing.
• Partners rejoin from their cards. Restore partner pins from trusted configuration or the card's authority key history, and verify a newly advertised key against that history before it replaces a pin.
• Federation re-handshakes. Each side announces its checkpoint head and both agree on a fresh exchange schedule. Partner revocation logs are pulled from the retained acknowledged offset, because a valid signature leaves revocation status open.

A restore that fails a check stays closed to admission

The anchor covers history the snapshot lacks

Recover that history from another tier, or reconcile and record the gap before new writes. A new checkpoint extends the retained head; the missing receipts stay missing.

Revocations lag the freshness budget

Keep admission closed and replay deltas from the control-plane log and from each partner's revocation log at the retained offset until the window closes.

The signing seed did not survive

Rotate to a new keypair and re-anchor relying parties on it. Receipts already signed keep verifying while their public keys and trust history are retained; Secrets & Signing Keys has the procedure and how the checkpoint chain crosses a rotation.

Checkpoint fields, store tables, and encryption layers

Next: converge the restored node with its cluster