Budgets Across Nodes

A monetary admission returns after a quorum holds its own event

The writing node counts itself. A peer counts once its acknowledgment head for the write's origin reaches the write's sequence.

In the three-node write drawn here, node A and node B hold event 58 and meet the quorum of two, while node C, acknowledged through 57, waits. A head is the last event before the first gap, so a peer missing a preceding event cannot witness a later one. Only the write's own origin counts: a peer that is ahead on another origin, or that awaits a forced snapshot, stays outside the witness set.

Admissions and the monetary transitions that follow them wait for this witness. Invocation-only increments and denied admissions return after one leader-visible re-read, and a denial still writes its journal event.

Read more in Replication & Convergence, which defines the eligible witness set.

Each write is forwarded, stamped, identified, and witnessed in order

Every mutating budget route runs the same five steps after its service-token check.

1. Forward. forward_post_to_leader checks quorum, then lease validity, then the leader URL, and answers 503 before opening a connection if any is missing. It makes at most two attempts, the second only against a leader that changed. A node without local quorum cannot mutate budgets.
2. Stamp. A clustered money write calls current_budget_event_authority, which answers 503 when the lease is unavailable or expired and otherwise stamps authority_id (the leader URL), lease_id in the form {leader_url}#term-{epoch}, and lease_epoch. A transition on an existing hold takes its authority from resolve_budget_hold_authority: an injected joint authority store's fence first, then the authority recorded on the hold, then the current lease.
3. Identify. When an optional eventId is absent, the handler mints cluster-budget-write-{pid}-{nanos}-{counter} from a process-local counter, so the witness waits on this write's own sequence. Waiting on the authority's highest sequence would let a concurrent write to another capability raise the target and roll back a write that had reached quorum.
4. Target. budget_write_token resolves (origin_id, event_seq, budget_term) from the stored event, with the event's own origin and lease epoch. A retry after leadership moved waits on the origin peers advertise the event under. Only a missing row falls back to the authority's highest sequence, which can over-target but never under-target.
5. Wait. wait_for_budget_write_quorum_commit nudges the background loop once with request_sync, then parks on a watch channel and fetches from no peer itself. It answers 503 when quorum disappears, on timeout, or when the progress channel closes while the node is clustered. The timeout is the larger of a sync-interval term clamped to 5 to 30 seconds and a term that scales with the peer count, capped at 300 seconds.

A successful wait adds the budgetCommit block to the response: budgetSeq and commitIndex (both the write's own event sequence), quorumCommitted, quorumSize, committedNodes, witnessUrls, and the authority and term that wrote it.

let budget_commit = match wait_for_budget_write_quorum_commit(&state, write).await {
    Ok(budget_commit) => budget_commit,
    Err(_) => {
        if already_captured || replayed_event {
            return plain_http_error(
                StatusCode::SERVICE_UNAVAILABLE,
                "budget authorization replay could not confirm quorum commit; admission retained",
            );
        }
        let rollback_result = rollback_budget_authorize_exposure(
            &state,
            &payload,
            authority.as_ref(),
            commit_index,
        );
        return match rollback_result {
            Ok(()) => plain_http_error(
                StatusCode::SERVICE_UNAVAILABLE,
                &format!(
                    "budget authorize became leader-visible at commit index {commit_index} but failed quorum commit; local exposure rollback succeeded"
                ),
            ),
            Err(error) => budget_internal_error(
                &error,
                "budget authorize quorum failure and rollback failed; admission retained",
            ),
        };
    }
};

Read more in Leader & Failover, which covers forwarding and the lease.

An admission that misses its quorum is reversed by a new event

When the witness wait fails on authorize-exposure, the node appends a reverse under an event id derived from the generation it cancels. A later authorize that raised the usage sequence in between stays intact.

fn budget_authorize_compensation_event_id(
    payload: &TryChargeCostRequest,
    authorize_event_seq: u64,
) -> String {
    if let Some(event_id) = payload.event_id.as_deref() {
        return format!("{event_id}:rollback:{authorize_event_seq}");
    }
    if let Some(hold_id) = payload.hold_id.as_deref() {
        return format!("{hold_id}:rollback:{authorize_event_seq}");
    }
    format!(
        "rollback:{}:{}:{}",
        payload.capability_id, payload.grant_index, authorize_event_seq
    )
}

When an authorize of 60 is followed by an unrelated hold of 40 and the first is rolled back, the usage row lands at one invocation and 40 units exposed. The rolled-back hold cannot reopen under its original id, and a retry under a fresh hold id receives a strictly higher commit index, so a caller retrying a compensated authorize opens a new hold.

Compensation covers newly admitted exposure only. Capture, release, and reconcile answer the same 503 through respond_after_budget_write_quorum_commit and keep their local mutation. A replayed event id or an already-captured hold answers 503 with admission retained, because a reverse there could remove exposure another request relies on or reopen a slot whose call may have taken effect.

The reverse, written by rollback_budget_authorize_exposure, is a local write that waits for no peer. If it fails, the response is 500, or 503 for a fenced store, with budget authorize quorum failure and rollback failed; admission retained. After any of these responses, read the operation and the hold state before retrying, since the counters may already have moved.

A lagging node imports history without forgetting or over-claiming

A node that falls behind installs another node's budget snapshot only after three checks on the import path.

1. Replay. The snapshot is replayed into a throwaway in-memory store and compared with what it claims, so a payload without its own complete final usage projection never reaches the database.
2. Local subset. validate_local_snapshot_subset requires the snapshot to retain every local mutation event identically, cover every local abandoned sequence range, and prove every local usage row.
3. Heads. A claimed covered_head must equal the head the imported records prove, and each per-origin acknowledgment head must match the same way.

A leader started with --authority-db whose snapshot carries a usage-history anchor attaches a budgetAnchorProvenance chain. The importer refuses one that comes from a node it does not call leader, names another leader or term, mismatches the anchor set, or rewinds or forks the chain it has persisted for that leader.

The chain is a chio.budget-snapshot-anchor-provenance.v1 record with one signed commitment per link, each binding a commit sequence, the previous chain digest, the anchor-set digest, the leader URL, and the election term. Each commitment is checked against a key carried in its own body, which proves the chain's continuity. The cluster authenticates the chain with a constant-time HMAC of cluster_authenticator under the cluster service token, so provenance is as strong as that shared secret. A snapshot without the chain goes through import_budget_snapshot with the three checks.

A clustered plane labels its budget guarantee advisory_posthoc

Three records answer what happened to one clustered budget write.

Receipts carry the guarantee level of the store that admitted the spend, and a consumer can pin a floor. budget_authority_guarantee_level returns advisory_posthoc whenever a cluster is configured, and a kernel that reaches budgets through --control-url holds a RemoteBudgetStore that reports AdvisoryPosthoc in every case. quorumCommitted: true states that one event reached a quorum; linearizable spend across nodes is outside what it claims.

The release profile treats clustered monetary budgets as provisional authorized exposure. Across a partition, the documented bound is overrun <= max_cost_per_invocation × node_count, which assumes each replica admits at most one invocation at the per-call cap before the merge catches up; the exposure bound works it through.

Read more in Authoritative Spend, which states what a receipt must carry before a spend counts as authorized.

Budget routes and records

Next: follow a revocation across the same nodes