Budget Store
Reserve a call's worst-case cost before dispatch, reconcile it to what was spent, and retry without charging twice.
On this page
A grant's spending limit holds under concurrent calls and crashes. The node reserves each call's worst case before the call runs and writes every change to one durable journal.
The budget store keeps that journal on one node. Money held for calls in flight and money already spent count against the same limit, so no call spends capacity another call is holding. Budgets Across Nodes coordinates the same limit across a cluster.
Held and spent money count against one limit
Before a priced call leaves the kernel, the store reserves its worst case: the quoted cost when the call carries a quote, otherwise the grant's max_cost_per_invocation. The call dispatches only after that hold is recorded.
The worst case is held before dispatch
Before a model call leaves the kernel, the hold ledger reserves its worst case against the grant’s USD 2.00 budget: the quoted cost when a quote is present, otherwise max_cost_per_invocation. The call dispatches only after the hold is recorded. Authoritative spend.
USD 2.00 budget, step by step. The store keeps amounts as u64 minor units. Composite admission’s quota and cumulative-approval state are left out.Budgets and meteringWhen the reply returns, the hold reconciles down to the reported cost and the unused reservation goes back to the limit. An abort before execution reverses the hold with no charge. A reply that never returns keeps its hold until the outcome is resolved, and the call is never dispatched again. A worst case that would cross the limit is refused before dispatch, and the kernel signs a deny receipt.
In minor units: a grant with a limit of 100 and a per-invocation bound of 8 has 70 spent and 20 held by other calls. Reserving 8 brings the committed total to 98, so another 8-unit call is refused (98 + 8 > 100). Reconciling a reported cost of 3 releases the other 5 and leaves the committed total at 93.
Each hold moves through journaled transitions to one final disposition
A hold opens at authorize, which runs three limit checks inside one IMMEDIATE transaction: the invocation count, then the per-invocation cap, then the total. An overflow while adding the total is refused as an error.
- Capture. Marks the invocation slot as used. Before capture, a pre-dispatch failure reverses the hold; after it, the only pre-dispatch exit is
cancel_captured_before_dispatch. - Release. Gives back part of the exposure. The hold closes as
Releasedwhen the remainder reaches zero. - Reverse. Removes the whole remaining exposure and the invocation count, and closes the hold as
Reversed. - Reconcile. Moves the reported cost into realized spend, releases the rest, and closes the hold as
Reconciled.
Every transition appends one row to the mutation journal, refusals included. A refused authorize still takes a sequence number and a commit index while writing no usage row, so a deny receipt can cite the budget commit that refused it.
The limit checks in Rust
if request.max_invocations.is_some_and(|max| invocation_count >= max)
|| request
.max_cost_per_invocation
.is_some_and(|max| request.requested_exposure_units > max)
{
return Ok(false);
}
let committed = checked_committed_cost_units(exposed, realized)?;
let requested = committed
.checked_add(request.requested_exposure_units)
.ok_or_else(|| {
BudgetStoreError::Overflow(
"committed cost + requested exposure overflowed u64".to_string(),
)
})?;
Ok(request
.max_total_cost_units
.is_none_or(|max| requested <= max))Read more in Authoritative Spend, which defines the receipt side of the hold lifecycle.
A retried mutation applies once, and an unknown outcome keeps its hold
The kernel derives each hold's identity from the request, capability, and grant before it touches the store, and every event id hangs off that hold id. Replaying an event id returns the recorded outcome instead of applying the mutation again.
The replay must match the original exactly: capability, grant index, kind, hold, amounts, and all three caps. A different mutation under a reused id fails, and so does a retry presented under a different lease, because each event carries the authority that wrote it.
When a tool server errors or the call's future is dropped after dispatch began, the node keeps the hold. Reversing it would reopen capacity for a call that may already have taken effect. The signed receipt names the retained hold in budget_authority.hold_id, and recovery works from the durable operation and hold state.
Read more in Unknown Outcomes.
Abandoned holds close without undercounting spend
A crash between authorize and reconcile leaves a hold open. The kernel sweeps open holds once at boot and then periodically, and age only selects which holds to examine.
A hold expires only after the recovery coordinator proves the request moved no money, either because it had no effect or because the contract waives the charge. The store then releases the remaining exposure, writes the disposition Expired, and leaves realized spend untouched. Every other old hold stays frozen and raises an operator incident.
Holds reserved under an execution nonce follow a stricter rule. Once the nonce expires without a reconcile, the hold settles at its worst case, because the caller's reconcile is the only evidence of what was spent and releasing at zero would undercount against a cumulative cap.
One node guarantees single-node atomicity, and its receipts say so
Authorize runs its checks, the usage update, the hold insert, and the journal append in one transaction, so no partial state is visible. That guarantee covers one process against one file, and the store reports it as SingleNodeAtomic.
Receipts carry the guarantee level, and a consumer can pin a floor and reject any receipt whose level ranks below it. The floor check fails closed on an unrecognized level. It ranks the label the receipt carries, so a receipt is as truthful as its signer and the backend that reported the level.
A separate process opening the same file meets the serving-owner fence. Several nodes sharing one limit coordinate through the cluster's replicated budget log, which Budgets Across Nodes describes.
Budget store records and transitions
Try reconcile, event_id, or SingleNodeAtomic. Each entry has its own link.
Store types and records
| Term | Defined in | Role |
|---|---|---|
BudgetStore | chio- | The trait: lifecycle methods, read methods, and three profile methods that describe the backend. |
Sqlite | chio- | The durable node-local backend: WAL, one writer behind a mutex. The schema also creates the replication, quota, and cumulative- |
In | chio- | A mutex over hash maps with the same journal, driven by the property tests. |
Budget | chio- | One row per (capability_id, grant_index): invocation_, total_, total_, seq. |
Hold | budget_ | One in-flight authorization: remaining_, invocation_, disposition, and a projection_kind of legacy or composite_v1. |
Budget | budget_ | The journal: one row per attempted mutation, allowed or denied, with the after-state and a dense event_seq. |
Budget | chio- | authority_id, lease_id, lease_epoch, stamped on every mutation and checked again on replay. |
Amounts are u64 minor units. MonetaryAmount (chio-core-types/src/capability/scope.rs) pairs units with an ISO 4217 currency string, and ToolGrant carries the optional max_invocations, max_cost_per_invocation, and max_total_cost; a grant with none of the three has no budget limit. ADR-0006 records why amounts are integers, and ADR-0016 makes the authorize-then-reconcile hold lifecycle normative.
Committed cost
Exposure is money the node has committed to allow while it waits for a reported cost; realized spend is money a completed call reported. A cap is compared against their sum, computed with a checked add so a wrap becomes a refusal.
fn checked_committed_cost_units(
total_cost_exposed: u64,
total_cost_realized_spend: u64,
) -> Result<u64, BudgetStoreError> {
total_cost_exposed
.checked_add(total_cost_realized_spend)
.ok_or_else(|| {
BudgetStoreError::Overflow(
"total_cost_exposed + total_cost_realized_spend overflowed u64".to_string(),
)
})
}The Kani harness verify_budget_checked_add_no_overflow, compiled under cfg(kani), proves fail-closed no-partial-commit and overflow-before-cap ordering for a standalone model of this arithmetic. The proof covers the model, and the store's transaction tests cover the store.
Transitions and their accounting
| Transition | Journal kind | invocation_ | total_ | total_ |
|---|---|---|---|---|
authorize | Authorize | +1 | + requested | unchanged |
capture | Capture | unchanged | unchanged | unchanged |
reverse | ReverseExposure | −1 | − exposure | unchanged |
cancel after capture | Cancel | −1 | − exposure | unchanged |
release | ReleaseExposure | unchanged | − amount | unchanged |
reconcile | ReconcileSpend | unchanged | − exposure | + realized |
expire | ExpireHold | unchanged | − remaining | unchanged |
authorize_budget_hold_atomic runs the limit checks, the usage upsert, the hold insert, and the journal append inside one IMMEDIATE transaction.
Hold dispositions
create_hold inserts at HoldDisposition::Open with invocation_captured = 0. release writes Released only when the remainder reaches zero and otherwise leaves the row open; reverse and cancel-after-capture write Reversed; reconcile writes Reconciled; the recovery release writes Expired. BudgetHoldDispositionView, the projection the trait returns, has the same five variants.
Capture sets a flag. Its statement sets invocation_captured = 1 and touches disposition through a CASE that fires only when remaining_exposure_units = 0. The reaper finds captured-but-unreconciled work with WHERE disposition = 'open' AND reserved_until IS NOT NULL.
Invocation and monetary lanes
Each journal row carries four state columns beside its counters: invocation_state_before, invocation_state_after, monetary_state_before, and monetary_state_after. append_mutation_event derives all four through appended_event_lifecycle on every event, denials included. They are per-event stamps on two machines, separate from the hold-level disposition.
| Lane | Enum | Variants, in declaration order |
|---|---|---|
Invocation | Budget | Absent, Authorized, Captured, Reversed, Denied |
Monetary | Budget | None, Exposed, Released, Reconciled, Captured, Reversed |
Both enums live in chio-kernel/src/budget_store/model.rs. On the write path the lanes are derived from the mutation kind, the allow bit, and the exposure amount. On import and replay, validate_legacy_event_lifecycle lists the before and after pairs each kind may carry and refuses anything else with budget mutation `X` has an invalid legacy lifecycle projection.
Authorize moves the invocation lane from Absent to Authorized, or to Denied on a refusal; capture moves it to Captured; reverse and cancel-after-capture move it to Reversed; release and reconcile leave it unchanged. The monetary lane reads Exposed after a non-zero authorize, Released once the remainder is given back, Reconciled after a settle, and None throughout when the authorized exposure was zero.
Four kinds belong to the composite path and never reach a legacy hold: ReserveInvocation, AuthorizeCumulativeApproval, ReverseInvocation, and CaptureSpend. validate_supported_import_record refuses an imported record carrying one with budget mutation `X` uses state unsupported by the sqlite budget store. CaptureSpend is the only kind that produces BudgetMonetaryState::Captured.
Authorize
For a legacy hold, the projection the node-local validation path creates, the pre-execution debit is grant.max_cost_per_invocation in minor units, and 0 when the grant sets no per-invocation bound. A grant that sets only max_total_cost therefore reserves zero exposure, reconcile clamps realized spend to the authorized exposure (realized_spend_units: realized.min(charge.cost_charged) in reconcile_budget_charge), and only max_invocations counts. Set max_cost_per_invocation on every grant whose monetary cap must hold on this path.
A server that reports more than the authorized exposure produces a cost_overrun: the receipt records settlement_status failed and the hold reconciles at the exposure, so the accounting records admitted spend. Detecting an overrun does not undo an external charge; inspect the receipt and the provider’s billing evidence to resolve the discrepancy.
A refusal appends a journal event with allowed = Some(false) and a commit index and writes no usage row. denied_authorize_hold_reports_durable_commit_metadata asserts one event and get_usage still None against InMemoryBudgetStore; the SQLite path allocates the sequence and appends the event in the same code path, under its own tests.
Capture, reverse, and cancel
Before capture, a pre-dispatch failure calls reverse_budget_hold, which decrements invocation_count and removes the exposure. After capture the store refuses reverse, and cancel_captured_before_dispatch is the distinct transition with its own journal kind. Reverse takes exactly the hold's remaining_exposure_units and refuses any other amount with budget hold `X` does not match reverse amount; partial give-back is release.
Reconcile
Reconcile raises an explicit Invariant error unless three conditions hold: the hold was captured (invocation was not captured before reconciliation), the reconciled exposure equals the hold's remaining exposure, and realized spend does not exceed exposed cost. The unused remainder is released by construction, so a reconciled hold can carry both a committed and a released amount under one terminal disposition.
Retained holds
The journal has no retain mutation. A hold kept after an unknown dispatch outcome stays open, and its exposure keeps counting against the cap. When the invocation produces a signed receipt, budget_authority.hold_id names the hold, and mark_runtime_admission_reservations_retained_fail_closed copies any live reservation ids across as retained_destructive_lease_id, retained_treaty_continuation_id, and retained_swarm_continuation_id. A canceled future need not return a receipt to its caller, so recovery reads the durable operation and hold state.
Hold and event ids
The id of a legacy hold is budget-hold:{request_id}:{capability_id}:{grant_index}. Event ids are {hold_id}:authorize, {hold_id}:capture-invocation:{commit_index}, {hold_id}:reconcile, and, for both reverse and cancel-before-dispatch, {hold_id}:authorize:rollback:{commit_index} (kernel/validation.rs, kernel/mod.rs). A nonce preflight hold prefixes nonce-preflight-, and under durable admission the id comes from the admission record.
On a standalone node the authority is kernel:{public key hex} with lease id single-node and epoch 1. existing_event_allowed returns the recorded outcome for a replayed id and fails a mismatch with budget event_id `X` was reused for a different mutation; validate_replay_authority compares the persisted BudgetEventAuthority with the presented one. Idempotency is keyed on the id the kernel derives, so a caller that mints a fresh id per retry gets a second mutation.
Opening, fencing, and integer range
The store opens with journal_mode = WAL, synchronous = FULL, busy_timeout = 5000, and foreign_keys = ON, stamps its schema revision under the budget key, and refuses a database already provisioned to a serving owner with BudgetStoreError::Fenced before it reads anything. Writes take TransactionBehavior::Immediate and check the serving-owner fence and authority anchor inside begin_write; reads use a deferred transaction and the same fence.
SQLite stores signed 64-bit integers, so every amount and sequence number passes through budget_u64_to_sqlite. A value above i64::MAX is refused with Overflow (budget field `X` exceeds SQLite INTEGER range), which halves the usable range on this backend.
The kernel serializes its own budget access behind a process mutex in with_budget_store and recovers from poisoning; with_revocation_store takes no such lock. The mutex orders one process's mutations, and the serving-owner fence covers a second process.
Serving-owner path
SqliteBudgetStore::open is what a node opens for itself. A store built with open_alongside a serving owner, as the control plane's durable-admission authority database is, refuses the unbound lifecycle mutations with joint sqlite authority rejects legacy {operation} mutation and runs the composite path with its own quota, approval, and admission-binding tables. Budgets Across Nodes documents that path.
BudgetStoreError::OutcomeUnknown reports a commit whose result cannot be established, so the caller recovers instead of retrying. commit_joint_transaction raises it under a serving owner. The unbound node-local path commits with transaction.commit()? and reports a commit failure as BudgetStoreError::Sqlite.
Trait defaults
Rich trait methods default to Invariant errors such as budget store does not support truthful rich release projections, and the _with_ids shims error when handed a hold or event id they cannot honor. try_charge_cost_with_ids_and_authority has no default, "so a backend upgrade cannot compile successfully and fail only on live calls."
Reapers and the hold sweep
reap_orphaned_holds settles holds against a map of realized amounts. reap_expired_reserved_holds handles holds marked reserved with a deadline and settles an expired, unreconciled hold at its worst case. The allow path marks a hold reserved when it mints an execution nonce, and the deadline is that nonce's expires_at; a hold with no nonce has a NULL reserved_until and is left to the hold sweep. Both reapers are idempotent and touch only open holds, and the trait default for each returns zero.
chio api protect runs the reserved-hold sweep every 30 seconds when a mediation kernel is configured. An application that embeds the kernel calls reap_expired_reserved_budget_holds on its own clock.
The kernel hold sweep runs at boot and then every hold_sweep_interval (default 300 seconds), listing open holds past hold_expiry_horizon (default 3,600 seconds) with list_open_holds_older_than. release_recovered_hold releases a hold only with a terminal resolution of no effect or a contractual zero charge, keyed by its resolution digest so a duplicate applies once. chio_budget_open_holds and chio_budget_holds_expired_total expose the sweep, and chio budget holds list and chio budget holds review <hold-id> show a hold's journal and resolution.
Backend profile methods
Three trait methods describe the backend. Their defaults are what an unclustered node reports: budget_guarantee_level() returns SingleNodeAtomic, budget_authority_profile() returns AuthoritativeHoldEvent, and budget_metering_profile() returns MaxCostPreauthorizeThenReconcileActual. SqliteBudgetStore keeps all three defaults.
BudgetAuthorityProfile and BudgetMeteringProfile each have one variant. BudgetGuaranteeLevel has four: SingleNodeAtomic, HaLinearizable, PartitionEscrowed, and AdvisoryPosthoc.
Currency
Every lifecycle method takes bare u64 units; currency lives on MonetaryAmount in the grant and on the reserved-hold row. The kernel converts a reported cost whose currency differs from the grant's through resolve_cross_currency_cost and a PriceOracle, which an operator installs with set_price_oracle. With no oracle the conversion fails closed: the hold reconciles at the full authorized exposure and the receipt records oracle_conversion.status = failed.
Guarantee floor
ADR-0016 makes the BudgetGuaranteeLevel taxonomy normative and requires it to be truthful: no ha_linearizable without a quorum store. receipt_meets_guarantee_floor ranks the four levels and fails closed on an unrecognized floor and on an unrecognized claimed level, so a typo cannot rank as the weakest level.
The predicate ranks a label. r4_receipt_claiming_ha_linearizable_fails_single_node_operator_floor pins that an honest single-node receipt fails a linearizable floor; a receipt re-signed with the same key after editing guarantee_level passes it. Signer admission and honest backend reporting stand between the two, which is why a store that reports a level it cannot back is a correctness bug.