Budgets Across Nodes

What a second node changes

One question: what changes when a second node holds the same authority keypair and the same admin token.

Three moving parts. Every mutating budget request is funnelled to one node. On the four money routes that node waits for peers to acknowledge the exact event it wrote before it answers. When that wait fails on an admission, the admission is compensated rather than left standing. What none of it buys is a hard cap: under a split brain the overrun is bounded, not eliminated, and the bound is arithmetic you can compute before you deploy.

Status: shipped. The live tests in crates/products/chio-cli/tests/trust_cluster.rs cover quorum refusal, compensation after a failed commit, denied-event replication, and snapshot catch-up. Read the coverage boundary carefully. The one live test that asserts the budgetCommit block, trust_control_cluster_replicates_state_and_fails_closed_without_quorum, is marked #[ignore] as flaky on CI, and its guarantee-level assertion still expects ha_quorum_commit, a string no shipped code emits. The witness arithmetic itself is pinned by an in-crate unit test over a simulated peer-ack state in cluster_and_reports.rs.

A clustered node stamps budgetAuthority.guaranteeLevel as advisory_posthoc, the weakest level ADR-0016 defines: rank 0, below single_node_atomic. An unclustered node emits no budgetAuthority block at all, so the single_node_atomic branch of budget_authority_guarantee_level never reaches the wire. Clustering buys availability on this plane and pays for it in enforcement strength.

Five routes, one authority

The route names and the store verbs are not one to one, and the mismatch is the most common way to write a wrong client. release-exposure performs the store's reverse, and reconcile-spend performs either release or reconcile depending on which fields the body carries. Three of the four money routes also accept a legacy form that carries no holdId, which lands on the corresponding _with_ids_and_authority ledger call instead of the hold lifecycle: authorize-exposure, release-exposure, and reconcile-spend. capture-invocation has no such form. The store column above names the hold-bearing path. The wire vocabulary is exposure-first throughout: exposureUnits, maxExposurePerInvocation, maxTotalExposureUnits on the way in; totalExposureCharged and totalRealizedSpend on the way back. Reconcile carries all three of authorizedExposureUnits, realizedSpendUnits, and reductionUnits, which budget_wrappers_use_split_budget_routes pins byte for byte.

On the three routes whose identifiers are optional, identity is checked after the service token and before the request is forwarded. A holdId without a non-empty eventId is a 400, and so is any empty identifier, in validate_budget_request_identity. A store fenced to another serving owner answers 503 rather than 500, and the epochs that were compared never reach the response body.

One budget write, in order

Forward or refuse. After the service-token check, and after identity validation on the three routes that run it, every mutating handler calls forward_post_to_leader, which checks quorum, then lease validity, then the leader URL, and answers 503 before opening a connection if any of them is missing. It makes at most two attempts, and the second only against a leader that changed. Shared with every other cluster-aware write; see Leader & Failover. A minority node therefore cannot write budget state at all: trust_control_cluster_requires_quorum_and_heals_after_partition isolates one node of three, asserts 503 on /v1/budgets/increment, writes the same request successfully through the majority, then heals the partition and waits for the isolated node to report the usage row it never wrote.

Stamp the authority. An unclustered node writes with no authority at all, and so does every /v1/budgets/increment regardless of clustering. A clustered money write calls current_budget_event_authority, which fails the write with 503 if the lease is unavailable or expired, and otherwise stamps authority_id (the leader URL), lease_id of the form {leader_url}#term-{epoch}, and lease_epoch. Transitions that carry a hold do something more careful: resolve_budget_hold_authority takes an injected joint authority store's fence first if one is present, then the authority recorded on the hold itself, and only then the current lease. So a capture or reconcile that arrives after leadership moved is fenced under the lease that authorized it rather than the one currently in force.

Mint the event id. A caller may omit eventId. The handler mints one anyway, as cluster-budget-write-{pid}-{nanos}-{counter} from a process-local atomic counter, and generated_budget_event_id_is_unique_per_call checks ten thousand ids in a tight burst are all distinct. This is not cosmetic. The witness has to wait on this write's sequence number, and falling back to the maximum sequence for the authority would let a concurrent write to an unrelated capability raise the target, parking the wait on a later event and rolling back a write that had itself reached quorum.

Build the witness token. budget_write_token resolves (origin_id, event_seq, budget_term) from the stored event. When that event already carries an authority, its own origin and lease epoch are used, so an idempotent retry after a leadership change waits on the origin peers actually advertise the event under. The authority-maximum fallback survives only for the row-not-found case, and it can only over-target the sequence, so the witness stays fail-closed. Which peers may then answer for it, and why a peer pending a forced snapshot may not, belongs to Replication & Convergence.

Wait, then compensate. wait_for_budget_write_quorum_commit nudges the background loop once with request_sync, then parks on a watch channel and fetches from no peer itself. It returns 503 the moment cluster quorum disappears, on timeout, and if the progress channel closes while the node is still clustered. The timeout is the larger of a sync-interval term clamped to the 5 to 30 second range and a term scaling with the peer count, capped at 300 seconds. Only after a successful wait does the response gain its budgetCommit block: budgetSeq and commitIndex (both the write’s own event sequence), quorumCommitted, quorumSize, committedNodes, witnessUrls, and the authority and term that authored it. The witness set always counts the writing node itself; peers count only when their contiguous ack head for this write’s origin has reached its sequence.

let budget_commit = match wait_for_budget_write_quorum_commit(&state, write).await {
    Ok(budget_commit) => budget_commit,
    Err(_) => {
        if already_captured || replayed_event {
            return plain_http_error(
                StatusCode::SERVICE_UNAVAILABLE,
                "budget authorization replay could not confirm quorum commit; admission retained",
            );
        }
        let rollback_result = rollback_budget_authorize_exposure(
            &state,
            &payload,
            authority.as_ref(),
            commit_index,
        );
        return match rollback_result {
            Ok(()) => plain_http_error(
                StatusCode::SERVICE_UNAVAILABLE,
                &format!(
                    "budget authorize became leader-visible at commit index {commit_index} but failed quorum commit; local exposure rollback succeeded"
                ),
            ),
            Err(error) => budget_internal_error(
                &error,
                "budget authorize quorum failure and rollback failed; admission retained",
            ),
        };
    }
};

The compensation is a forward mutation, not an undo. It appends a reverse under a derived event id that names the generation it is cancelling, so a later authorize that raised the usage sequence in between is not clobbered:

fn budget_authorize_compensation_event_id(
    payload: &TryChargeCostRequest,
    authorize_event_seq: u64,
) -> String {
    if let Some(event_id) = payload.event_id.as_deref() {
        return format!("{event_id}:rollback:{authorize_event_seq}");
    }
    if let Some(hold_id) = payload.hold_id.as_deref() {
        return format!("{hold_id}:rollback:{authorize_event_seq}");
    }
    format!(
        "rollback:{}:{}:{}",
        payload.capability_id, payload.grant_index, authorize_event_seq
    )
}

budget_authorize_compensation_tracks_the_authorize_generation drives exactly that shape: authorize 60, let an unrelated hold authorize 40, roll the first one back, and assert the usage row lands at one invocation and 40 units exposed. It then asserts the rolled-back hold cannot be reopened under its original id, and that a retry under a fresh hold id gets a strictly higher commit index. A caller retrying a compensated authorize needs a new hold, not a replay. End to end, trust_control_cluster_failed_quorum_does_not_leave_orphaned_exposure stops the only peer of a two-node cluster with SIGSTOP, asserts the 503, and waits for the usage row to return to zero invocations and zero exposure.

Taking another node’s budget history

A node that fell behind is repaired by import, and import is where a second node can do real damage. Three checks on the snapshot import path in chio-store-sqlite/src/budget_store/snapshot.rs bound it. A snapshot is replayed into a throwaway in-memory store first and compared against what it claimed, so a payload that does not carry its own complete final usage projection never reaches the real database. validate_local_snapshot_subset then requires it to retain every local mutation event identically, cover every local abandoned sequence range, and prove every local usage row. And a claimed covered_head must equal the head the imported records prove, with the same equality demanded of the per-origin acknowledgment heads. An import cannot make this node forget, and it cannot make it over-advertise.

A second import path adds provenance, and it is not a setting you turn on. The serving node attaches a budgetAnchorProvenance block only when it is the elected leader, its snapshot carries at least one usage-history anchor, and it was started with --authority-db. Absent the block, the importer falls back to the unverified import_budget_snapshot with the three checks above still applied. Present, it is also refused unless it came from the node this importer currently calls leader. The block is a chio.budget-snapshot-anchor-provenance.v1 chain carries one signed commitment per link, each binding a commit sequence, the previous chain digest, the anchor-set digest, the leader URL, and the election term. The head must name the expected leader and term and match the exact anchor set being installed, and the chain must not rewind or fork the one already persisted for that leader. Read the trust anchor carefully: each commitment is verified against a public key carried inside its own body, so the signatures prove chain continuity rather than signer admission. What actually authenticates the chain to this cluster is the constant-time HMAC comparison of cluster_authenticator under the cluster service token. Provenance here is exactly as strong as that shared secret.

The overrun bound, as arithmetic

Publish this number before you set a budget, not after an incident. Where nodes admit independently against their own replica, each one can approve up to one invocation at the full per-invocation cap before the merge propagates:

overrun <= max_cost_per_invocation * node_count

# the guide's own worked case: two nodes, a $5.00 per-invocation cap
#   -> worst case $10.00 admitted beyond max_total_cost
#      before the LWW merge propagates

Two qualifications matter more than the formula. First, the precondition does the work. Both source documents state the bound as what can be admitted before the merge propagates, and neither says what happens over a longer partition. Each isolated node keeps admitting until its own local view exhausts the cap, so read the bound as per convergence window rather than per partition. That reading is ours. A partition that outlasts many windows degrades toward node_count × max_total_cost, which follows directly from the three limit checks reading local state and is not something any test in the tree pins.

Second, the clustered trust-control path narrows the window rather than closing it. A node in the minority cannot admit anything, because forwarding refuses before the store is touched. The residual exposure is the asymmetric partition: each node computes quorum over its own candidate set, so two nodes that can both see a third can each hold quorum, name themselves leader, and admit concurrently. That case is stated in the Limit row of Leader & Failover, and it is the shape in which the arithmetic above still applies to a clustered deployment.

Guarantees and limits

Next Steps

• Node · Budget Store · the baseline this page distributes: hold dispositions, idempotency, reapers, the conservation law
• Replication & Convergence · who may witness a budget write, and the contiguity guard the acknowledgment head depends on
• Leader & Failover · how the write reached one node, and the asymmetric partition the overrun bound survives in
• Cluster Overview · the four claims the rung declines to make, stated before any mechanism
• Authoritative Spend · what a receipt has to satisfy before a spend counts as authorized at all