Node State on Disk

One writer per file

The kernel names its own boundary. crates/kernel/chio-kernel-core/src/evaluate.rs lists what its pure core does not do, and the first two items are revocation membership lookup and budget mutation, both fenced into chio-kernel proper as stateful. A guard stays Kernel content even when its verdict reads node state. The store it reads is this page.

Everything below is local persistence for a single OS process. A node can instead point its revocation and budget state at a remote control plane (DurableAdmissionRuntime::open_remote in chio-control-plane), in which case the bytes live on another node’s disk and the Cluster rung owns the question. What one process can answer by reading its own filesystem is what follows.

Two shapes of database

chio-store-sqlite is the concrete backend for every durable store trait the kernel defines. It opens files in two arrangements.

• The authority database. One file provisioned and served by SqliteAuthorityStore, holding one Connection behind a mutex. Logical stores are handed clones of that connection through open_alongside: budget, revocation, admission operations, tool outcomes, FROST custody, the economic state cache, fiscal state, channel lifecycle, and the channel release publisher. They never reopen the file and never start a second writer.
• Own-file stores. SqliteReceiptStore, SqliteRevocationStore, SqliteBudgetStore, SqliteApprovalStore, SqliteBatchApprovalStore, SqliteExecutionNonceStore, SqliteMemoryProvenanceStore, SqliteEncryptedBlobStore, and SqliteCapabilityAuthority each take a path and open it directly. Two of them share a file on purpose: chio api protect opens the receipt store first and then co-locates the approval store onto the same sidecar database.

A sidecar started with chio api protect --receipt-store /var/lib/chio/receipts.db produces this layout on disk:

That layout is the sidecar’s, not a universal convention. The same authority store opened by DurableAdmissionRuntime::open in chio-control-plane puts its lock root at <path>.locks and points the budget and revocation stores at the authority file rather than a sibling.

Reads through a pool, writes through one actor

Receipt queries are read-heavy and receipt appends are not, so the receipt store keeps two pools: DEFAULT_READER_POOL_MAX_SIZE = 8 and DEFAULT_WRITER_POOL_MAX_SIZE = 1 (chio-store-sqlite/src/lib.rs). Every write runs on the single writer connection through the commit actor’s run_write. Session anchors, request lineage, lineage statements, child receipts, and manual checkpoint creation all route through the writer. They used to run IMMEDIATE transactions on reader-pool connections, stacked behind inline checkpoint construction: that is the path that produced SQLITE_BUSY after a tool had already run.

The reader pool is read-only by routing discipline, not by open flags: both pools open the same file read-write, and the property is pinned by a test that grabs all eight reader connections, forces PRAGMA query_only = ON on each, and then drives the whole routed write path to completion (reader_pool_never_begins_a_write_transaction).

The actor group-commits: it drains up to RECEIPT_GROUP_COMMIT_MAX_BATCH = 64 appends into one transaction, waiting at most RECEIPT_GROUP_COMMIT_FLUSH_DELAY (500 microseconds) for each next append, on a bounded channel sixteen times the batch deep (1024). A full channel is a refusal, not a queue: ReceiptStoreError::Pool("sqlite receipt commit queue saturated"). The default append blocks until its batch commits, so a mediated allow is returned only after the receipt is durable.

Callers that cannot afford an unbounded wait use the deadline variants (append_chio_receipt_with_timeout, run_write_receipt_with_timeout), which fail closed with ReceiptStoreError::Timeout { operation, timeout_ms }. That is a caller-side deadline only. The command is already on the channel, so the actor still owns it and may still commit it, which is why expiry deliberately leaves the in-flight counter elevated for the liveness probe to read.

At start the actor seeds a verified head once, by running full claim-log validation and a full checkpoint-chain verification against the writer connection. After that, each append verifies only the delta: one indexed latest-checkpoint read plus a digest compare, and an aggregate scoped to rows past the head. Seeding runs on the actor thread, and writer health reports a poisoned head until it succeeds. If it fails, the writer publishes the error and refuses appends rather than accepting writes it cannot vouch for; recovering means restarting the process to reseed, or running chio receipt audit --repair with the kernel stopped. Incremental verification is the default, not the only mode: SqliteStoreOptions::incremental_verification is read-only after open and, set false, keeps full per-append verification for A/B checking a suspect database. The design is written up in docs/architecture/reliability/RFC-0006-storage-hot-path.md, which is still marked Draft; the routing, the fence, and the verified head are in the tree.

The authority database is arranged differently and deliberately so. There is no pool: one connection, one mutex, cloned into every logical store, which is what lets a budget authorization and its admission commit land in a single transaction.

The serving-owner row

SQLite serializes transactions. It does not serialize recovery ownership, and it will not tell a stale worker that it stopped being the writer. The serving owner does both. It is one row, and it binds the database identity to the process currently allowed to mutate it.

CREATE TABLE chio_serving_owner (
    singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
    store_uuid TEXT UNIQUE NOT NULL,
    database_path TEXT NOT NULL,
    database_device INTEGER NOT NULL CHECK (database_device >= 0),
    database_inode INTEGER NOT NULL CHECK (database_inode >= 0),
    lock_root TEXT NOT NULL,
    lock_device INTEGER NOT NULL CHECK (lock_device >= 0),
    lock_inode INTEGER NOT NULL CHECK (lock_inode >= 0),
    owner_epoch INTEGER NOT NULL DEFAULT 0 CHECK (owner_epoch >= 0),
    lease_id TEXT,
    opened_at_ms INTEGER
);

Provisioning is a separate step

SqliteAuthorityStore::provision runs before anything serves. When the path does not exist it creates the database with create_new, mode 0600 and O_NOFOLLOW. When it does exist, provisioning adopts it only if it is already a regular file, mode exactly 0600, owned by the effective user, with link count one, and only if the authority tables in it are still pristine. It requires the database’s parent directory and the lock root to be owned by the effective user and neither group- nor world-writable; mints a canonical UUID-v7 store id; creates exactly one <store_uuid>.lock inode in the lock root, mode 0600 with link count one and the lock root’s own uid and gid; and records the device and inode of both files. The owner table and its single row are created in one IMMEDIATE transaction, so an interrupted provision rolls the table back with the row and leaves a recoverable database rather than a half-built one. Files and both parent directories are fsynced. It is idempotent only when the UUID and the inode metadata match exactly.

Provisioning is a library call, not an operator step. It runs from DurableAdmissionRuntime::open and from the sidecar’s own startup path, so the process that will serve the database is the process that creates it, and the ownership record is written under the same uid that the serving checks later enforce.

Opening claims the epoch

open_serving canonicalizes the path, takes an advisory lock on the lock root, re-reads the provisioning record, and refuses a database whose path, device, inode, mode, or link count no longer matches what was provisioned. A symlink or hardlink alias is refused. It then takes try_lock on the lock inode and holds it for the serving lifetime, re-validating the record after the lock is held so a race cannot slip a different file underneath it.

Only then does it mutate. In one IMMEDIATE transaction it closes the previous lease row at the current admission-authority head, increments owner_epoch, writes a fresh lease_id, and inserts the new open lease. Lease history lives in chio_serving_leases and is append-and-close-only: triggers ABORT on any delete and on any update that is not the single legal transition from open to closed. A startup check requires the leases to form a dense chain of epochs whose authority intervals tile without gaps.

What the caller receives is a StoreMutationFence of store_uuid, lease_id, and owner_epoch. Every mutation re-reads the owner row inside its own transaction and compares all three fields; a mismatch returns Fenced { expected_epoch, actual_epoch } before a row changes. Two further checks run alongside: PRAGMA data_version is compared against the value observed at open, so a change made outside the serving connection poisons the owner with OutcomeUnknown; and a two-slot rollback anchor written into the lock file records the commit heads and chain digests, so restoring an older database snapshot under a live lock root is caught instead of silently accepted.

What an open refuses

Ownership works in the other direction too. Once a database carries an owner row, a standalone open of one of its logical stores is refused: SqliteRevocationStore::open on a provisioned authority file reports that the store requires the joint serving owner at its current epoch, rather than writing to it unfenced.

The schema stamp

Before ownership, before pragmas, before any write, an open answers one question: is this file ours, and can this binary read it? Every store calls the same gate.

/// ASCII "CHIO" as a big-endian `i32`, stamped into every Chio operator store.
pub const CHIO_SQLITE_APPLICATION_ID: i32 = 0x4348_494f;

/// Table holding each co-located store's schema revision, keyed by a stable
/// per-store identifier. `PRAGMA user_version` is database-wide and so cannot
/// give the receipt and approval stores (which the sidecar keeps in one file)
/// independent revisions; a keyed table can.
const SCHEMA_VERSION_TABLE: &str = "chio_store_schema_versions";

/// Failure to validate or apply a store's schema stamp.
#[derive(Debug, thiserror::Error)]
pub enum SchemaVersionError {
    #[error("sqlite error: {0}")]
    Sqlite(#[from] rusqlite::Error),
    #[error("database application_id {found:#x} is not a Chio store (expected {expected:#x})")]
    ForeignDatabase { found: i32, expected: i32 },
    #[error(
        "database carries the Chio application_id but none of this store's tables ({expected_anchors:?}); refusing to open another store's database"
    )]
    MismatchedStore { expected_anchors: Vec<String> },
    #[error(
        "database schema version {found} is newer than this binary supports ({supported}); refusing to open"
    )]
    FutureSchema { found: i32, supported: i32 },
}

The revision itself is not PRAGMA user_version. That pragma is database-wide, and co-located stores in one file need independent revisions, so each store records its own under a stable key in chio_store_schema_versions. The receipt store sits at its revision and the approval store at another in the same sidecar file without either making the other refuse to open.

The shared application id proves a file is a Chio store but not which one, so anchors carry the rest of the identification: the receipt store answers to chio_tool_receipts plus the legacy names http_receipts and tool_receipts, and the joint authority open passes capability_grant_budgets and revoked_capabilities. The two legacy names are generic enough that an unrelated database could carry one by coincidence, so on an unstamped file the receipt store also requires a receipt payload column (raw_json or receipt_json) before it will adopt on that name alone.

A database with no user tables is adoptable, which is how a fresh file and a pre-stamping legacy file both open cleanly. Emptiness is not a universal pass: the application_id is read first, and a file stamped by some other application is refused as ForeignDatabase whether or not it holds any tables.

The refusal an operator actually meets is the mistyped path. Point a receipt command at a revocation database and the anchor check answers by name, before anything is written:

$ chio receipt checkpoint status --receipt-db ./state/revocations.sqlite
error [CHIO-CLI-RECEIPT-STORE]: receipt store error: not found: receipt database ./state/revocations.sqlite is not an initialized Chio receipt store: missing table chio_tool_receipts
context: {"source":"not found: receipt database ./state/revocations.sqlite is not an initialized Chio receipt store: missing table chio_tool_receipts"}
suggested fix: Check the configured receipt store path, permissions, and schema health before retrying.
$ echo $?
1

Both halves of the stamp are readable from the file with any SQLite client, which is the fastest way to answer "is this ours?" without starting a node. 1128810831 is 0x4348494f, ASCII CHIO:

$ sqlite3 ./state/revocations.sqlite "PRAGMA application_id; PRAGMA journal_mode;"
1128810831
wal

WAL is single-writer and single-host

Every durable store opens in WAL mode with synchronous = FULL and busy_timeout = 5000. Foreign keys are not uniform: the serving owner, revocation, budget, approval, and receipt stores set foreign_keys = ON; the capability authority and execution-nonce stores do not, so do not read referential enforcement into either of those two files. The receipt store adds auto_vacuum = INCREMENTAL. WAL coordinates readers and writers through a shared-memory index that works only when every connection is on the same host and a local filesystem. That is the operational rule behind the whole page, and it is the one operators break most often.

A store is more than one file, and the extras matter to anyone writing a backup script. One allow evaluated with a receipt database and a session database, then the directory listed. The session database carries the durable admission authority, so it brings a keypair pair and a lock directory with it, all at 0600:

$ chio --receipt-db ./receipts.sqlite3 --session-db ./sessions.sqlite3 \
    check --policy policy.yaml --tool hello_world --server hello
$ ls -la
-rw-r--r--   1 chio chio      636 policy.yaml
-rw-r--r--   1 chio chio  1007616 receipts.sqlite3
-rw-------   1 chio chio  1507328 sessions.sqlite3
-rw-------   1 chio chio       65 sessions.sqlite3.kernel.pub
-rw-------   1 chio chio       65 sessions.sqlite3.kernel.seed
drwxr-xr-x   4 chio chio      128 sessions.sqlite3.locks

$ ls -la sessions.sqlite3.locks
-rw-------   1 chio chio      339 .chio-path-6c1464fd638c23b0c2d18bd7979daf5df45748c70a498e5e05d52ece3fb3dc3b.identity
-rw-------   1 chio chio     2048 019fbe46-df16-7bb1-9fc6-fa1440bd44b6.lock

The -wal and -shm siblings appear beside each database while a connection is open and are checkpointed away on a clean close. A backup that copies the .sqlite3 file alone, while a writer holds an unmerged WAL, copies a database missing its most recent commits; that is what Backup & Restore uses .backup rather than cp for.

The receipt store does not assume the setting took, and it is the only store that checks. After issuing the pragmas it re-reads four of them on the connection it just configured (journal_mode must be WAL, synchronous must be 2, busy_timeout must be at least 5000, foreign_keys must be 1) and refuses the open if a value did not stick, so a filesystem that quietly declines WAL stops the process at startup instead of running without the durability the store advertises. The other stores issue the same pragmas with no read-back:

let journal_mode: String = connection.query_row("PRAGMA journal_mode", [], |row| row.get(0))?;
if !journal_mode.eq_ignore_ascii_case("wal") {
    return Err(ReceiptStoreError::Conflict(format!(
        "sqlite receipt store journal_mode must be WAL, got {journal_mode}"
    )));
}

Non-durable paths are classified, not guessed. is_in_memory_sqlite_path recognizes :memory:, file::memory:, and any file:...?mode=memory URI, and the sidecar refuses to boot on one unless the operator passes --allow-ephemeral-receipts. Cloud Run and Azure Container Apps have no per-instance persistent disk, so the reference manifests run the log on a local in-memory volume: WAL works, and the log is gone on every revision recycle. For durable receipts there, front a client-server audit store or move to a platform with a per-instance disk.

Guarantees and limits

Next Steps

• Revocation Store · the node-local list, the read-only view the core is handed, and the fail-closed durability gate
• Budget Store · authorize, capture, release, reconcile at the single-node atomic guarantee level
• Backup & Restore · what to copy, and what a copy of a provisioned database does not carry with it
• Performance & Tuning · what one receipt costs in this schema, measured through dbstat against a live store
• Sidecar HTTP Service · volumes, per-instance disks, and the platforms with none
• Fail-Closed Semantics · what the kernel does when a store refuses to open or a writer stops