Retention & Archive

Deleting evidence without breaking it

A receipt database is append-only by construction. The receipt tables and the claim_receipt_log_entries projection carry reject_delete triggers that RAISE(ABORT), every persisted checkpoint’s Merkle root is rebuilt from that projection on verification, and the projection is cross-checked against the source tables by exact receipt-id set equality. A plain DELETE therefore does not lose history, it is refused. Drop the guards first and you get the worse outcome: the next append, the next health check, and the next open all fail on set drift, and a restart does not recover. One drift shape has an in-code repair; it is the last section of this page.

Retention is the supported way out. It archives a contiguous prefix of the log into a second SQLite file, deletes exactly that prefix from the live database in one transaction, and records a watermark that tells every later verifier which range is now checked against the archive rather than rebuilt from the live claim log. Everything here is one process reading and writing its own disk. The mechanism is in the tree: the maintenance thread in crates/kernel/chio-kernel/src/receipt_store.rs, the Rotate writer command and the co-archive-then-delete path in crates/platform/chio-store-sqlite/src/receipt_store/evidence_retention.rs. The design record, docs/architecture/reliability/RFC-0007-retention-without-bricking.md, is still headed Status: Draft and listed as minor-gaps in the reliability index. Read the RFC for why the shape is what it is; read the code for what it does.

Pruning here is what makes the cluster receipt streams non-dense, so a peer cannot distinguish a row this page archived from one it never received: Receipt Aggregation states that as a published non-claim.

One thread, one actor, three durable objects

• The maintenance thread. RetentionMaintenanceHandle::spawn takes a dedicated Arc<dyn ReceiptStore> clone of the store and one OS thread. It sleeps check_interval_secs in 200ms slices so a stop is responsive, calls ReceiptStore::rotate_receipts inside catch_unwind, records the outcome, and sleeps again. It sleeps before its first rotation, so a freshly started kernel never rotates at boot. Dropping the handle sets the stop flag and joins, so shutdown waits out at most one 200ms slice plus any rotation already running. This thread is the only caller of rotate_receipts in the tree. No CLI verb and no HTTP route forces a rotation.
• The writer actor. Rotation does not run on the maintenance thread. dispatch_rotate increments the writer in-flight counter, try_sends a ReceiptCommitCommand::Rotate to the same single commit actor that owns appends, and blocks on the reply. Serialization with appends comes free from that queue, with no second lock. The send is non-blocking: a full commit channel refuses the rotation for that interval rather than queueing it.
• The archive file. A separate SQLite database, stamped with the Chio application_id and the receipt-store schema revision. create_archive_schema creates thirteen evidence tables in it, plus the schema-version table: nine receive copied rows, and the four checkpoint-projection tables are empty shells so that opening the archive as a receipt store does not fail on a missing table. Archive tables carry no REFERENCES clauses; it is a write-once evidence copy, not a live database enforcing cascades.
• The watermark ledger. receipt_retention_watermark records each rotation’s archived_through_entry_seq, cutoff, and archive path. Three triggers make it append-only and strictly increasing at the database level, because a later verifier trusts this value to skip work.
• The tombstones. receipt_retention_tombstones keeps one small row per archived receipt id. Deleting a live receipt also deletes its UNIQUE(receipt_id) sentinel, so without a tombstone an archived id looks brand new to the append path’s ON CONFLICT DO NOTHING and could be inserted a second time. Two BEFORE INSERT triggers on the receipt tables abort any insert of a tombstoned id, and two more make the tombstones themselves immutable.

What an operator can actually set

Attaching a store is where a policy that could never be honored is rejected. try_set_receipt_store_handle refuses three configurations outright rather than spawning a worker that would log a failure every interval: a store whose supports_retention is false, a tenant_id against a store whose supports_tenant_scoped_retention is false (which is every store in the tree, including SqliteReceiptStore), and retention alongside checkpoint_batch_size == 0. The last follows from the mechanism: the watermark only advances to a checkpoint boundary, so with checkpointing off it could never leave zero.

The signer that makes a prefix prunable

Retention can only free what a checkpoint has already committed, so its throughput is the checkpoint signer’s throughput. The same attach that spawns the maintenance thread installs a BackgroundCheckpointSigner { keypair, max_batch } as one more command on the writer actor, with max_batch taken from checkpoint_batch_size (DEFAULT_CHECKPOINT_BATCH_SIZE = 100, and 0 disables checkpointing per ADR-0008). A store that advertises kernel-signed checkpoint support but fails to install a signer has its attach refused rather than appending forever without producing any.

The build runs after commit_receipt_batch has already sent every append durability response, so it does not extend append latency. It is off the append response path, not off the writer thread: it takes the same single writer connection, and it runs before the co-drained flush waiters are released, which is what makes flush_receipt_writes a checkpoint barrier and not merely a durability barrier. Each checkpoint is O(batch) and the loop repeats while the head owes max_batch entries. A build failure, including a caught panic, records last_error and leaves the head untouched; it never fails the append that triggered it, because that append is already durable.

One rotation, in order

The actor arm refuses before it reads anything if the writer’s verified head is poisoned. Then it runs a full verify_checkpoint_chain_integrity and a full claim-log set-equality validation, in both verification modes. That is deliberate and it is the expensive part. An incremental head attests new appends only; it never notices that a checkpoint-covered source row and its projection row were both deleted out of band after the store opened. Rotating over that drift would co-archive the survivors, delete the rest, and stamp a watermark the archive cannot back. The rebuild is off the append hot path, which is the code’s argument that it is affordable, but it still holds the single writer for its duration. See the limits table for what that costs.

The cutoff comes from resolve_rotation_cutoff. An explicit cutoff wins. Otherwise both triggers are evaluated and the rotation takes whichever frees more; if neither fires, the rotation returns zero archived before reading a single checkpoint. The time trigger fires when the oldest claim-log entry predates now - retention_days * 86400. The size trigger fires when live_db_size_bytes, which is (page_count - freelist_count) * page_size, exceeds max_size_bytes, and its cutoff is one second past the median claim-log timestamp. Both details are load-bearing: excluding freelist pages is what makes a size-driven rotation converge instead of re-firing forever on freed-but-unreclaimed pages, and the extra second is what keeps a burst of receipts sharing one timestamp from blocking their own batch. The number an operator reads is not the number the trigger compares: the in-process health report publishes db_size_bytes (page_count * page_size, freelist included), and the read-only sampler behind chio receipt health publishes neither, nulling that field.

What holds the watermark back

Archival is expressed as one monotone entry_seq watermark W: the largest checkpoint batch_end_seq whose entire covered prefix qualifies. Because checkpoints tile the prefix contiguously, the archived range is always [1, W] and no checkpoint ever straddles it.

SELECT COALESCE(MAX(kc.batch_end_seq), 0)
FROM kernel_checkpoints kc
WHERE NOT EXISTS (
    SELECT 1 FROM claim_receipt_log_entries e
    WHERE e.entry_seq <= kc.batch_end_seq
      AND e.timestamp >= ?1
)
AND NOT EXISTS (
    SELECT 1 FROM chio_authorization_receipt_consumptions ac
    JOIN claim_receipt_log_entries ae ON ae.receipt_id = ac.authorization_receipt_id
    JOIN claim_receipt_log_entries ce ON ce.receipt_id = ac.consumer_receipt_id
    WHERE ae.entry_seq <= kc.batch_end_seq
      AND ce.entry_seq > kc.batch_end_seq
)
-- ... three further NOT EXISTS clauses follow, plus a sixth on settle_attempts
-- ... when that table is present; see the table below

The first clause is the age rule, and it reads the whole prefix rather than the boundary row, so one non-monotonic timestamp inside the prefix cannot smuggle an unaged receipt into the archive. The other four are all the same shape: never archive a row that live evidence still points at.

Every guard reads the whole covered prefix, not the boundary row, and that costs liveness as well as buying safety. One receipt with an open reconciliation, or one unresolved settle_attempts row against it, disqualifies every checkpoint whose batch_end_seq reaches its entry_seq. That pins W at the last boundary below it for as long as the row stays nonterminal, and at zero when the row sits in the first batch, however old the rest of the log is. A stuck reconciliation is a stuck retention policy; a flat retention_watermark_entry_seq under a growing database is the symptom.

In incremental mode the result is then capped at the newest checkpoint boundary the pre-flight audit actually verified, so a checkpoint row appended by a second handle since the head was seeded cannot widen the prune. A W of zero, or one that does not exceed the recorded high-water mark, returns zero archived without touching anything.

Copy first, delete once

The archive path is vetted before the ATTACH. A target that evaporates on DETACH, or one that aliases the live database by path, by symlink, or by inode, is refused: the identity checks would compare a table to itself and pass trivially while the delete removed the only copy. The path is then canonicalized, so the ledger records an absolute location a later reader resolves the same way from any working directory.

The copy is idempotent. Append-only evidence uses INSERT OR IGNORE. The two reconciliation tables use INSERT OR REPLACE, because their rows are mutated in place by later reconciliation and a stale archived copy would otherwise fail the identity re-check on every subsequent rotation and stall the prefix permanently.

Then verify_co_archival_complete runs, and presence is not what it checks. For each of nine tables it counts the live prefix rows whose archive counterpart matches on that table’s key column and NULL-safely on every remaining column. Any shortfall raises RetentionArchiveIncomplete { table, live, archived } before a single delete. A count-only check would pass while a reused archive held the right ids under divergent bytes, and the delete would leave no faithful copy behind.

DROP TRIGGER IF EXISTS chio_tool_receipts_reject_delete;
DROP TRIGGER IF EXISTS chio_child_receipts_reject_delete;
DROP TRIGGER IF EXISTS claim_receipt_log_entries_reject_delete;

INSERT OR IGNORE INTO receipt_retention_tombstones
    (receipt_id, receipt_kind, archived_through_entry_seq, tombstoned_at)
    SELECT receipt_id, receipt_kind, {w}, {now}
    FROM claim_receipt_log_entries WHERE entry_seq <= {w};

DELETE FROM settlement_reconciliations WHERE receipt_id IN (
    SELECT receipt_id FROM claim_receipt_log_entries
    WHERE entry_seq <= {w} AND receipt_kind = 'tool_receipt');
DELETE FROM metered_billing_reconciliations WHERE receipt_id IN (
    SELECT receipt_id FROM claim_receipt_log_entries
    WHERE entry_seq <= {w} AND receipt_kind = 'tool_receipt');
DELETE FROM chio_authorization_receipt_consumptions WHERE authorization_receipt_id IN (
    SELECT receipt_id FROM claim_receipt_log_entries
    WHERE entry_seq <= {w} AND receipt_kind = 'tool_receipt');
DELETE FROM chio_tool_receipts WHERE seq IN (
    SELECT source_seq FROM claim_receipt_log_entries
    WHERE entry_seq <= {w} AND receipt_kind = 'tool_receipt');
DELETE FROM chio_child_receipts WHERE seq IN (
    SELECT source_seq FROM claim_receipt_log_entries
    WHERE entry_seq <= {w} AND receipt_kind = 'child_receipt');
DELETE FROM claim_receipt_log_entries WHERE entry_seq <= {w};

That batch, the watermark insert, and the recreation of every immutability guard all run inside one BEGIN IMMEDIATE transaction. SQLite executes CREATE and DROP TRIGGER transactionally, so a rollback restores the rows and the guards together and a failed delete can never leave the store append-only in name but not in enforcement. The transaction also re-runs the co-archival identity check, the archive-path pin, and the archive-backing check now that it holds the write lock: the pre-flight versions of all three are fast fails, and a second handle can commit a dependent row or repoint the ledger in the window between them.

The source tables and the projection lose exactly the same receipt-id set in that one transaction, which is why the set-equality validator passes afterward with no exemption of its own. Cross-database atomicity is never required: the copy finishes first, the delete touches only main, and a crash between them leaves the live store intact and the rotation re-runnable. Only then does the actor detach and run PRAGMA incremental_vacuum and PRAGMA wal_checkpoint(TRUNCATE) to give the pages back.

A watermark is trusted, not believed

Chain verification exempts every checkpoint with batch_end_seq <= W from the live Merkle rebuild, since its claim-log rows are gone on purpose. Signature parsing, column agreement, transparency-projection rows, and predecessor linkage all still run; only the rebuild is skipped. That exemption is the most dangerous thing on this page: a forged W would silence verification for the range it names. The ledger triggers enforce monotonicity, and monotonicity is not archival, so trusted_retention_watermark requires three independent facts and returns 0 if any of them fails.

The third check ties the exemption to evidence rather than to the absence of evidence, and it is not cheap. archive_path_backs_prefix opens the archive read-only and re-derives the Merkle root of every checkpoint at or below W from the archived rows, so it is O(archived history) and grows with every rotation, forever. It sits inside verify_checkpoint_chain_integrity, which runs when the writer seeds its head at open, in receipt_checkpoint_status behind every health report, and in the rotation pre-flight. Budget for a startup and a health check whose cost tracks total archived receipts, not live ones. The one place it could have reached the append hot path, the adopted claim-log delta check, is gated behind a cheap comparison against the raw ledger value first, so a steady-state append on a rotated store never opens the archive.

What a rotation refuses

A failure does not stay in the logs. The worker calls record_retention_rotation_outcome with the message, which lands in ReceiptStoreHealthReport.retention_error and forces healthy to false for as long as it is set, so a store serving under a retention policy that is not being honored cannot read green. A later success clears it. This is in-process state, and it is the reason chio receipt health is not enough on its own: receipt_store_health_read_only reads retention_watermark_entry_seq off the file, but hardcodes retention_error: None, defaults the whole writer block, and nulls db_size_bytes, because a read-only observer cannot see the owning writer’s memory. A store whose retention has been failing every hour for a week still prints healthy there. Read the failure from the process that owns the database. For how a degraded writer is judged, see Health & Readiness.

Guarantees and limits

One recovery path exists, for a store damaged by a delete that left orphaned projection rows behind. chio receipt retention repair --archive <path> opens through open_existing, so the tool does not brick trying to diagnose a bricked store. It removes only claim-log rows absent from both source tables, only when the named archive holds a byte-identical copy of each, only up to the smallest checkpoint boundary covering them, and only when every row in that boundary’s prefix is itself an orphan whose archived rows re-derive the signed roots. If a watermark already covers that boundary, the --archive you pass must be the one the ledger pinned, because the tombstones are stamped from it. Anything else aborts untouched, and both output modes exit non-zero when the drift survived. Nothing here recovers the other shape, a delete that took the source rows and their projection rows together; that store fails the rotation pre-flight and needs a restore.

Next Steps

• Receipts & Audit · the signed body, canonical JSON, and the checkpoint whose boundary this page prunes to
• Node State on Disk · the single writer, the reader pool, and the schema stamp every rotation runs through
• Health & Readiness · the supervised commit writer whose liveness a failing rotation shares, and what readiness denies on
• Backup & Restore · why the archive is now part of the backup set, not an optional extra
• Node Overview · what else one process owns on its own disk