PlatformDurable State
Node
Approval Store & Channels
The tables a paused call waits in, the channels that carry it to a human, and the transaction that resolves it.
The verdict next door, the state here
ApprovalGuard demand a human, which binding checks a presented token must survive, and what HitlVerdict comes back. That is one mediated call, and the value it can return is The Three Verdicts. This page owns everything that outlives the call: the rows on this node’s disk, the channels that leave the process, and the resume path back in.The half that remembers
An approval store answers two questions for one process: what is still outstanding, and what has already been spent. Both are answerable by reading one node’s own filesystem, which is what files them here rather than a rung up. Nothing on this page contacts a peer. Running two nodes is precisely where the single-use guarantee stops holding, and that limit is stated below rather than papered over.
The contract is ApprovalStore in crates/kernel/chio-kernel/src/approval.rs, deliberately synchronous because the kernel does not run on an async executor and both implementations (an in-memory reference and SqliteApprovalStore) are synchronous too. The store is real and durable. Be exact about what drives it.
- The guard is test-only.
ApprovalGuardis constructed in one file,chio-kernel/src/kernel/tests/approval_flow.rs; everywhere else it is a re-export inlib.rs. The seven-guard pipeline does not stand in for it either:kernel/dispatch.rstreats aGuardthat returnsPendingApprovalas an unsupported state and fails closed. - The sidecar writes rows over HTTP.
POST /approvals/submittakes operator-supplied context, materializes anApprovalRequestwithgoverned_intent: Noneand a defaultpolicy_idofpolicy-hermes-hitl, and names the sidecar’s own signing key as the sole entry intrusted_approvers. Its doc comment states the boundary: manual flow only, the held call is not auto-resumed. Resolving records a decision; it does not release anything. - The kernel’s own pending path skips this store.
Verdict::PendingApprovalis raised by budget admission when a grant’s cumulative-approval requirement trips with no token attached. It signs aThresholdApprovalProposal, parks it on the durable admission operation, and returns aDenyreceipt readingcumulative approval required. Nochio_hitl_pendingrow is written on that path.
Durability is inherited from the receipt store
chio api protect selects SqliteApprovalStore only when it was given a durable --receipt-store path. It refuses to boot without one unless --allow-ephemeral-receipts is set, and in that mode the approval store is InMemoryApprovalStore, the threshold collector is its in-memory twin, and the durable admission stores are absent, so every pending row and every collected vote dies with the process. An in-memory SQLite path counts as no path. Everything below about files applies to the durable configuration.Five tables, two stores
SqliteApprovalStore creates every table below on open, all with CREATE TABLE IF NOT EXISTS, alongside the shared chio_store_schema_versions row it stamps. One type implements two traits against them: ApprovalStore for the first three, ThresholdApprovalCollectorStore for the last two.
| Table | Holds | Keyed on |
|---|---|---|
chio_hitl_pending | Outstanding requests, stored as the serialized ApprovalRequest in payload with eight fields lifted into columns for querying. Indexed on subject_id and on expires_at. | approval_id |
chio_hitl_resolved | The audit row a resolution leaves: outcome, timestamp, approver hex, token id. Indexed on (subject_id, policy_id, outcome), which is what count_approved reads. | approval_id |
chio_hitl_consumed_tokens | The replay registry. One row per token actually spent. | (token_id, parameter_hash) |
chio_threshold_approval_collectors | One signed threshold proposal per row, with its requirement, its state, an optimistic version, and the whole record as canonical JSON in record_json. Reads come from that blob. | proposal_id |
chio_threshold_approval_collector_votes | Individual approver tokens, written so SQL enforces uniqueness rather than to be read back: nothing selects from this table outside the v1 migration. | (proposal_id, token_id) |
Batch approvals are a second store type entirely. SqliteBatchApprovalStore takes its own path and holds one table, chio_hitl_batches, indexed on (subject_id, revoked). No shipped binary opens it.
Both are own-file stores in the sense Node State on Disk defines: they take a path, open it directly, and never join the serving-owner arrangement the authority database uses. The schema gate is the shared one, under the store key approval at revision 2 and batch_approval at revision 0. What is specific here is the anchor set. A standalone open accepts exactly one anchor, chio_hitl_pending, so a path mistargeted at a receipt, revocation, budget, or authority database is refused rather than written with HITL tables. open_colocated_with_receipt_store widens the set with three receipt anchors, because chio api protect keeps both stores in one file and opens the receipt store first, so the approval store meets a database already carrying somebody else’s anchor and no approval table yet. The pair is pinned by standalone_open_refuses_a_receipt_sidecar_that_colocated_open_adopts.
Pending row to resolved row
Writing the request
store_pending is idempotent on approval_id, but only for a byte-identical payload. The whole rule is one statement:
INSERT INTO chio_hitl_pending (...) VALUES (?1, ..., ?9)
ON CONFLICT(approval_id) DO UPDATE SET payload = excluded.payload
WHERE chio_hitl_pending.payload = excluded.payload
RETURNING payloadA retry carrying the same request updates a row to itself and returns it. A retry carrying a different one fails the WHERE, so the update is skipped, RETURNING yields nothing, and the call reports approval_id ... already exists with different payload rather than overwriting in-flight HITL state. The in-memory store reaches the same outcome with an equality compare before insert. Two things about that refusal are worth knowing. It is an ApprovalStoreError::Backend, not a typed conflict, so a duplicate /approvals/submit comes back as 500 internal_error. And the comparison is on the serialized string, not on JSON semantics, so a payload that changes shape across a binary upgrade reads as a conflicting retry rather than a matching one.
Leaving the process
A channel is a delivery sink, and the trait is two methods: name and dispatch. Two implementations exist in-tree and neither has a caller outside tests, because the only code that dispatches channels is ApprovalGuard::evaluate. In the shipped sidecar, notification is poll-only: GET /approvals/pending is the whole delivery mechanism, and it returns every matching row unless the caller passes a limit.
WebhookChannel is a blocking ureq POST with a 5s default timeout and one optional static header, deliberately one because every real caller uses a single auth secret. Being static it cannot be a per-body HMAC, whatever the doc comment suggests, and the channel retries nothing even though the trait says implementations own retries. Its body is { event, approval, callback_url } with event fixed at approval_requested, approval the entire serialized request including its trusted approver keys, and callback_url a host-less relative path. RecordingChannel appends every dispatch to an unbounded in-memory vector; its doc comment attributes it to an api-poll dispatch mode that does not exist in the tree.
Where the guard does run, dispatch failure is not a refusal. It iterates channels sequentially, logs a redacted warning on Transport, Remote, or Config, and leaves the pending row in place so the poll route still serves it. A webhook outage delays a call; it does not drop or allow one.
Doc comments describing behavior that is not there
dispatch returns a ChannelHandle of channel, channel_ref, and an optional action_url, and its doc comment says the kernel records it alongside the request so the dispatch can be cancelled later. Neither half is true in the tree: the guard discards the Ok value, no table has a column for it, and the trait has no cancel method. ApprovalRequest::callback_hint is documented as something a dispatcher fills in after sending and is None at every construction site in the repo. The module header links ChioKernel::evaluate_tool_call_with_hitl, which no longer exists. Treat all three as reserved shape, not behavior.The resolve transaction
resolve is one SQLite transaction doing six things in a fixed order: read the pending row’s policy_id and parameter_hash inside the transaction to avoid a check-then-act race, refuse if the bound token is already in the consumed registry, refuse if a resolution row already exists, insert the resolved row by selecting straight from the pending row, insert the consumed-token row, and delete the pending row. Either the request stops being pending and the token stops being spendable, or neither happens. That shape is why record_consumed and is_consumed are separate trait methods rather than folded into resolve, as the trait comments say. One consequence of the order: because the pending row is read first and deleted last, a second resolve of the same id returns NotFound, not AlreadyResolved. The already-resolved branch is only reachable when a pending row was reinserted after a resolution.
The caller above it is resume_with_decision, and its ordering carries a stated reason. It loads the pending record, reports already resolved: {id} ({outcome}) when only a resolution exists, checks the replay registry, verifies the token against the stored request, and only then compares the HTTP envelope’s outcome to the signed decision. That last check runs before the store is touched: a mismatched pair, where the token says denied and the body says approved, would otherwise have already flipped the request to resolved and moved the approved counter before the error was returned.
| Condition | What resume returns | Status on respond |
|---|---|---|
URL id disagrees with the signed request_id | Rejected before the store is read | 400 |
| No pending row and no resolution | ApprovalRejected("unknown approval id") | 403 |
| A resolution row already exists | ApprovalRejected("already resolved") | 403 |
The token is already in chio_hitl_consumed_tokens | ApprovalRejected("approval token already consumed (replay)") | 409 |
The store raises Replay from inside the transaction | ApprovalRejected("replay detected") | 409 |
| Binding, approver-trust, time, lifetime, or signature check fails | ApprovalRejected(reason) | 403 |
| A concurrent respond deleted the pending row between the load and the transaction | Internal("approval store: approval request not found") | 500 |
Read that third column carefully. The resume path collapses every distinction into one string-carrying variant, and the HTTP mapping recovers 409 by testing whether the message contains replay. Anything the resume path does not classify as ApprovalRejected becomes 500. The typed mapping that turns NotFound into 404 and AlreadyResolved into 409 is real, but it applies only where a handler calls the store directly, as GET /approvals/{id} does. Match on the code, and do not read a 403 on respond as a signature failure.
The batch endpoint, POST /approvals/batch/respond, runs the same path per entry and never aborts the set: each decision lands as resolved or rejected with its own message, under one summary of total, approved, denied, and rejected. It is a loop over independent transactions, not a transaction over the loop. The list is uncapped, and the response is 200 even when every entry was rejected, so the summary is the only thing worth reading.
The TTL ceiling, and the deadline that is not one
MAX_APPROVAL_TTL_SECS = 3600 caps the token lifetime: expires_at - issued_at above an hour is rejected in verify_against, because the single-use registry’s retention horizon is pinned to that value and a longer token could outlive its own replay entry. The sidecar clamps in the same units on the way in, ttl_seconds.min(3600) with 3600 as the default when the field is zero or absent, and mints 600-second tokens on the operator-respond shortcut.
A request’s expiry is advisory
ApprovalRequest::expires_at is written, indexed, and never enforced. verify_against compares the clock only to the token’s own window, and no resolve path reads the request’s deadline, so a request that lapsed weeks ago still resolves cleanly. Nothing sweeps expired rows out of chio_hitl_pending either: the only DELETE in the store is the one inside resolve. Filtering is opt-in, through ApprovalFilter::not_expired_at on list_pending, which the caller must pass as a query parameter. Operators who need auto-deny at the deadline should treat that as unbuilt.Two paths beside the single request
Threshold: m-of-n on one node
The threshold collector is the shipped multi-approver path. chio api protect routes four endpoints at it under /approvals/threshold/proposals for create, get, respond, and deliver, and opens SqliteApprovalStore a second time against the same file to back it. Those routes are the only writers of the collector tables; the kernel’s own cumulative-approval proposal is parked on the durable admission operation instead. A proposal must be signed by a trusted policy authority, and in the sidecar that set holds one key, the sidecar’s own signer, which is also the key its embedded kernel signs proposals with. It must further agree with its requirement on threshold and eligible-set digest before it is stored.
Each submitted token must agree with the proposal on request id, intent hash, subject, and the proposal’s own artifact digest, sit inside three proposal-relative bounds (issued_at at or after creation, before the deadline, and expires_at no later than the deadline) as well as its own validity window against now, come from an approver in the eligible set, and survive a submitter-separation check when one is required. Its decision must be Approved: the collector has no way to record a denial, only cancel.
Storage is optimistic. Every mutation reads the record inside a transaction, refuses if the version moved or the state is terminal, and writes with the expected version and previous state in the WHERE clause; a row count other than one is a conflict, not a retry. The vote table enforces the rest in SQL:
canonical_token_digest TEXT NOT NULL UNIQUE,
...
PRIMARY KEY (proposal_id, token_id),
UNIQUE (proposal_id, approver_fingerprint),
UNIQUE (proposal_id, canonical_token_digest),
FOREIGN KEY (proposal_id)
REFERENCES chio_threshold_approval_collectors(proposal_id)The column-level constraint is the strict one: a canonical token digest is unique across the whole file, not per proposal, so one signed token cannot be submitted to two proposals even where the rest would allow it. The foreign key is weaker than it looks, because PRAGMA foreign_keys is set only on the connection that ran the migration.
Revision 2 exists for the primary key. The v1 table made token_id globally unique, so two proposals collided on an id that only has to be unique within one; opening a v1 database rebuilds the table scoped to the proposal and copies every row across in one transaction, asserted by open_migrates_v1_threshold_vote_ids_to_proposal_scope. Delivery has one more subtlety worth knowing: token expiry is bounded above by the proposal deadline but not below, so a set that was quorate at submission can hold lapsed tokens by delivery time. deliver filters to currently valid tokens and fails with threshold approval quorum is no longer satisfied rather than delivering a stale set, and the proposal stays Ready so the same signer can replace their expired token.
Batch: pre-approving a class
A BatchApproval lets one human pre-approve a class of calls: a subject, a server and tool pattern, an optional per-call and total amount, an optional call count, a time window, and used counters carried on the row so consumption can be reconciled without going back to the receipt log. find_matching narrows in SQL to non-revoked rows for the subject inside the window, then applies pattern, count, and amount filters in Rust and returns the first hit. There is no ORDER BY, so which batch wins when several match is unspecified. Pattern matching is exact, full wildcard, or one trailing-star prefix. Amount handling has one rule worth stating plainly: a call carrying no monetary intent matches only a batch that constrains neither per-call nor total amount, so putting a cap on a batch narrows it to priced calls and nothing announces that. Consumption is not atomic with matching either. record_usage is a separate unconditional UPDATE, so concurrent callers can both clear a max_calls check and overshoot it, and store is INSERT OR REPLACE, so re-storing a batch id overwrites its used counters.
Nothing consults it yet
BatchApprovalStore has both backends, a schema, and a passing test. It has no caller. find_matching and record_usage are invoked from exactly one kernel unit test, against the in-memory backend, and nowhere in any product crate, so no mediated call is currently satisfied by a batch approval. It is a finished store waiting for a guard.Guarantees and limits
| Status | Claim | Evidence |
|---|---|---|
| Shipped | Resolution is atomic: the resolved row, the consumed-token row, and the pending delete commit together or not at all, with both the replay and double-resolve checks inside the same transaction. | SqliteApprovalStore::resolve |
| Proved by test | Pending rows, the consumed registry, and resolutions survive a process restart: a second store handle on the same file lists the pending request and resumes it to Approved. | persistence_survives_restart, chio-store-sqlite/tests/approval_store.rs |
| Proved by test | A token spent once cannot be spent again, even after the pending row is reinserted; a second record_consumed for the same pair is a Replay error rather than a silent no-op. | resolve_rejects_replay, record_consumed_is_idempotent_on_first_write_only |
| Proved by test | A threshold proposal recovers its collected votes across three separate store opens and records delivery before returning the token set. | threshold_collector_recovers_votes_and_persists_delivery_before_return, chio-store-sqlite/src/approval_store.rs |
| Limit | Single-use is per store. The registry is a local table keyed on (token_id, parameter_hash); two kernels with separate stores cannot reject each other’s replays. Sharing one file instead runs into the unsupported row below. | chio_hitl_consumed_tokens; InMemoryApprovalStore::consumed_key |
| Limit | In the sidecar the approver is the sidecar. /approvals/submit installs the process signing key as the only trusted approver and /approvals/{id}/operator-respond signs the decision with it, falling back to that same key for the token subject when the request carries none. The human factor is reaching the route, not holding a key. | chio-api-protect/src/proxy/approval.rs |
| Limit | Every approval route sits behind the sidecar-control gate, and that gate admits any loopback caller when no control token is configured. The token comes from CHIO_SIDECAR_CONTROL_TOKEN in the environment, so an operator who never sets it makes approving a local, unauthenticated POST. | require_sidecar_control_request in proxy/sidecar.rs |
| Limit | Without --authority-seed-file the sidecar generates a keypair per boot, which it warns about at start. Pending rows outlive the key named on them, so after such a restart operator-respond returns sidecar signer is not a trusted approver for this request and the row is unresolvable. | signer_seed_hex in proxy/state.rs; approver check in operator_respond_approval_handler |
| Limit | Pragmas are issued once, on the pooled connection that ran the migration: journal_mode, synchronous, busy_timeout, foreign_keys. WAL is a property of the file and persists; the other three are per-connection and are not reissued, so the other connections the pool hands out carry SQLite defaults and do not enforce the vote table’s foreign key. Sibling stores in the same crate install busy_timeout through with_init; this one does not. | approval_store.rs::run_migrations against execution_nonce_store.rs and governed_approval_replay_store.rs |
| Limit | A request’s expires_at is recorded and indexed but never enforced, and no sweeper removes lapsed pending rows. | No comparison against expires_at in verify_against, resume_with_decision, or resolve; the only delete is inside resolve |
| Not durable by default | Under --allow-ephemeral-receipts with no durable receipt path, the approval store and the threshold collector are the in-memory implementations and the durable admission stores are absent, so the kernel’s cumulative-approval path cannot even park a proposal. | durable_receipt_db branch in proxy/state.rs |
| Not wired | Approval channels. Neither WebhookChannel nor RecordingChannel is constructed outside kernel tests, because only ApprovalGuard dispatches them. Notification in the sidecar is polling. | Repo-wide grep for both types in crates/ |
| Not wired | Batch approvals. Both backends and the schema exist; no product crate calls find_matching. | One kernel unit test is the only caller in crates/ |
| Not wired | The single-request ApprovalGuard in a shipped product. The sidecar drives the store over HTTP instead. | ApprovalGuard::new appears only in chio-kernel/src/kernel/tests/approval_flow.rs |
| Unsupported | Two processes writing one approval database. Unlike the authority file, this store takes no serving lock and mints no epoch fence, so nothing detects a second writer; the arbitration is SQLite’s file locking alone, on connections that mostly lack a busy timeout. | No serving_owner reference in approval_store.rs; contrast the serving-owner row |
| Design only | Timeout actions and escalation chains. The HITL protocol document specifies timeout_action, tiered escalation, and an Escalated approval state; no TimeoutAction or EscalationChain type exists in any crate and no approval type here has an escalated variant. The unrelated Escalated case state in chio-governance is a different mechanism. | docs/protocols/HUMAN-IN-THE-LOOP-PROTOCOL.md sections 2 and 15 |
Two replay registries, one word
GovernedApprovalReplayStore, which reserves a (subject_id, request_id, intent_hash) tuple at the dispatch boundary before any payment or tool side effect and leaves the marker blocking when a commit result is unknown. Different table, different key, different crate module. A deployment that needs replay safety across restart installs a durable implementation of both.Next Steps
- Approval & HITL · the guard, the constraints that trigger it, and the seven binding checks a presented token must survive
- The Three Verdicts · who may construct
PendingApproval, and what every other runner does on sight of it - Node State on Disk · the schema stamp this store passes through, the sidecar file it shares, and the fence it does not take
- Revocation Store · the other node-local list the kernel consults before an allow
- Fail-Closed Semantics · what the node does when a store refuses to open mid-flight