Replication & Convergence

Pull-only, one peer at a time

The delta and snapshot replication path is pull-only: no node pushes state to a peer. Every clustered node runs one background loop (run_cluster_sync_loop in crates/platform/chio-control-plane/src/trust_control/cluster/deltas.rs), visits its configured peers serially, and imports what each peer offers. Client writes travel a different path in the same cluster: forward_post_to_leader sends a budget, receipt, lineage, or fiscal POST that lands on a follower to the elected leader, so the write is applied once on the leader and reaches the other nodes only when they next pull. Read Budgets Across Nodes for that half.

Peer requests carry no keypair. Each one is a SHA-256 digest over a canonical chio.cluster.peer.v1 payload that embeds the shared service token, and the caller’s node URL must sit in the receiver’s own --peer-url allowlist inside a 60 second skew window; Cluster Peer Auth owns that wire. Membership is the --peer-url list, not a discovered set, and a peer marked unhealthy is unhealthy in this node’s view: standing is each observer’s own record, not a cluster verdict. The interval is --cluster-sync-interval-ms, default 500, floored at 50 by chio-cli.

Six streams, two lanes, one budget each

Five of the streams are incremental delta endpoints with a per-peer cursor. The sixth, the capability authority, is refetched whole every round and merged, so it has no cursor, no page guard, and no round budget of its own.

The authority merge is the sharp edge in that table. sync_peer_authority runs before both delta lanes, charges nothing to any budget, and returns early on failure: the visit records a sync error and ends before a single delta page is fetched and before the round finalizes, so that peer replicates nothing and witnesses nothing until the merge succeeds. A peer started without --authority-db answers 409 on that route (clustered authority replication requires --authority-db), which stalls every other stream from that peer.

The first four delta streams share one round budget, budget first. Revocations get a second, independent budget so a sustained budget backlog or a broken budget endpoint cannot starve revocation propagation. That lane is the whole of how a revoke recorded on one node reaches the others: there is no push, no gossip, and no separate revocation protocol, only this stream under its own budget and whichever page guard its contract selects. Both orderings are asserted directly: budget_pull_is_prioritized_first and revocations_are_not_in_the_shared_budget_round compare function pointers against peer_pullers().

pub(crate) const MAX_PULL_PAGES_PER_PEER_PER_ROUND: u32 = 64;
pub(crate) const MAX_PULL_RECORDS_PER_PEER_PER_ROUND: u64 = 200_000;
pub(crate) const PEER_ROUND_WALL_CLOCK_BUDGET: Duration = Duration::from_secs(20);

/// The per-round pull budget was reached. This is a LOCAL cap on how much a
/// single peer is pulled per sync round, NOT peer misbehavior: a large but
/// well-ordered backlog legitimately exceeds it. The puller stops the round and
/// resumes from the advanced cursor next round WITHOUT demoting the peer.
#[derive(Debug, PartialEq, Eq)]
pub(crate) enum RoundLimit {
    Pages,
    Records,
    Deadline,
}

The distinction in that doc comment is the whole design. A large, well-ordered backlog is honest and must not cost a peer anything; misbehavior on the wire must cost it everything. Four outcomes carry that split out of the pullers.

Only a protocol violation demotes, and the budget stream defines five of them: NonContiguousPage (a cursor-jump, an interior hole, or an out-of-order event), NonAdvancingPage (a non-empty page that does not move the merged cursor past the caller’s position), RecordsWithoutMutationEvents and AbandonedWithoutMutationEvents (a page of usage projections or abandoned slots with no events to anchor them), and InvalidAbandonedSequence. The records-only check runs before the over-cap check on purpose, so a malformed page always demotes and never gets routed to snapshot recovery.

RFC-0011 proposed making budget exhaustion itself a PeerProtocolError. The shipped code does not: demoting a peer for having a backlog would take an honest node out of consensus for being busy.

One peer visit, in order

The first gate is deliberate isolation, not failure. POST /v1/internal/cluster/partition sets partitioned on the named peers, which skips them entirely and flags each for a full snapshot; unblocking one resets its health to unknown and its delta counter to zero, so it resyncs from a snapshot rather than resuming a cursor formed before the block. The body carries the complete blocked set, not a delta: every configured peer absent from blockedPeerUrls is unblocked by the same call, and the node’s own URL is filtered out.

The one contract read off a peer

The second gate is where this node asks what a peer supports. prepare_peer_revocation_sync runs once per peer per visit, on the status response already in hand, and the answer selects which revocation puller runs for the rest of the visit. revocation_peer_contract reads two fields off ClusterReplicationHeadsView and returns one of two variants.

pub(crate) fn revocation_peer_contract(
    replication: &ClusterReplicationHeadsView,
) -> Result<RevocationPeerContract, PeerProtocolError> {
    match (
        replication.revocation_cursor_version,
        replication.revocation_stream_id.as_deref(),
    ) {
        (None, None) => Ok(RevocationPeerContract::Legacy),
        (Some(version), Some(stream_id)) => {
            ensure_revocation_cursor_version(Some(version))?;
            if stream_id.is_empty() {
                return Err(PeerProtocolError::MissingRevocationStreamIdentity);
            }
            let head_seq = match replication.revocation_cursor.as_ref() {
                None => 0,
                Some(cursor)
                    if cursor.cursor_version == Some(version)
                        && cursor.stream_id.as_deref() == Some(stream_id)
                        && cursor.seq.is_some_and(|seq| seq > 0) =>
                {
                    cursor.seq.unwrap_or(0)
                }
                Some(_) => {
                    return Err(PeerProtocolError::IncompleteRevocationStreamContract);
                }
            };
            Ok(RevocationPeerContract::Current {
                stream_id: stream_id.to_string(),
                head_seq,
            })
        }
        _ => Err(PeerProtocolError::IncompleteRevocationStreamContract),
    }
}

cluster_replication_heads fills both fields with REVOCATION_SEQUENCE_CURSOR_VERSION and the store’s stream id whenever a revocation store is open, and does so even when the stream is empty, so any peer serving a revocation store selects Current. Advertising the pair unconditionally is what makes an empty current stream distinguishable from a peer with no stream-epoch support at all; the field’s own doc comment says so. Legacy is reachable only against a peer that advertises neither, and half a pair is IncompleteRevocationStreamContract, which marks the peer Unhealthy and ends the visit before any lane runs.

The version is 4, and ensure_revocation_cursor_version is a three-way check rather than a comparison: 4 passes, an absent version is LegacyRevocationCursorUnsupported, and any other value is UnsupportedRevocationCursorVersion carrying the number the peer sent. The constant’s doc comment names what the two retired versions could not do: version 2 exposed the mutable projection sequence, and version 3 carried no durable stream-instance identity and so could not detect database replacement or rollback.

Detecting that is the whole point of the stream id. current_revocation_cursor_requires_snapshot compares the cursor this node cached for the peer against the identity and head the peer just advertised, and request_peer_snapshot_recovery flags a full resync with revocation stream identity or advertised head changed; snapshot required when the cached cursor is not stamped at version 4, names a different stream id, carries no seq, or sits above the advertised head. That is a peer whose database was replaced or rolled back, so the flag keeps the peer Healthy and routes it to a snapshot rather than demoting it. On the other arm the cached cursor is thrown away instead: clear_peer_revocation_cursor runs whenever a peer that now reads Legacy left behind a cursor carrying a version, a stream id, or a seq, so a sequence cursor is never carried across a downgrade.

The negotiation repeats on the snapshot path rather than being trusted from the status call. sync_peer re-reads the contract off the snapshot body and runs revocation_snapshot_contract_is_compatible against the one the status advertised; a mismatch is peer changed or regressed its revocation stream contract between status and snapshot and demotes the peer before the body is applied.

Clamping down early, raising at finalize

Two orderings inside the visit are load-bearing. First, a peer’s freshly advertised ack heads are clamped down at the top of the round, before any witness-visible pulling: clamp_down_peer_budget_acks sets each recorded head to min(recorded, advertised) and drops an origin the peer no longer advertises at all. A peer that restored an older budget database has disavowed those writes, and the stale-high value must stop counting immediately, not at the end of the round.

Second, an increase is not applied there. It waits for finalize_peer_sync_round, which runs only after the pull round completed without demotion and without a pending snapshot. Recording a high advertised head before the pullers ran, then waking parked writers, would let a budget write commit on an ack from a peer that the same round was about to remove from the witness set. Both directions are covered by regressed_ack_head_is_cleared_before_validation_not_witnessed_at_old_high.

Stopping between peers

The loop stops between peers on shutdown. A visit already in flight finishes, since a blocking peer call cannot be interrupted, but no further peer is dialed (peer_round_stops_visiting_when_shutdown_fires_midround). That bounds the remaining work to one visit, not one HTTP call: a full visit is three blocking stages at CONTROL_HTTP_TIMEOUT (15s) plus two 20s delta rounds, so a stop signal can still cost roughly 85 seconds before the loop exits.

Contiguity where the stream is dense, forward progress where it is not

Budget mutation events are a single store-wide, dense event_seq stream: every allocation yields exactly one event, so each authority’s events are a sparse subsequence of one global sequence and the pull cursor is a single global seq. That stream gets the strict guard.

pub(crate) fn require_contiguous_page(
    expected_next_seq: u64,
    seqs: &[u64],
) -> Result<(), PeerProtocolError> {
    let mut expected = expected_next_seq;
    for &seq in seqs {
        if seq != expected {
            return Err(PeerProtocolError::NonContiguousPage {
                expected_seq: expected,
                found_seq: seq,
            });
        }
        expected = expected.saturating_add(1);
    }
    Ok(())
}

A max-advance-only check is not enough, and the test says why: a peer at cursor 10 returning the page {110, 111} would pass it, import, advance to 111, and permanently omit rows 11 through 109. Ordering is checked twice, in two places. import_budget_delta_response walks the event seqs in their received order and rejects any that is not strictly above its predecessor, so a release-before-authorize reorder is a violation rather than a wrong import. It then sorts the union of those events and the page’s abandoned slots and runs the contiguity guard over that union. There is no compaction floor that authorizes a jump: a per-origin floor cannot license advancing a cursor that spans all origins, so a gap is a protocol violation and the peer heals through the snapshot path instead.

The other three delta streams are not dense. Receipt and lineage seqs come from INTEGER PRIMARY KEY AUTOINCREMENT columns written with ON CONFLICT DO NOTHING and are subject to retention deletes, so a store legitimately holds rows 1 and 3 with no row 2. Applying the dense guard there would demote honest peers and break the stream permanently. They get require_forward_progress instead: the page is sorted locally, its lowest seq must sit strictly above the cursor, and no seq may repeat. A gap above the cursor is accepted.

That is a real weakening, and the module doc names it: on a non-dense stream a malicious mid-stream skip is not reliably client-detectable. The periodic full snapshot is the backstop that bounds a lying peer, not incremental convergence.

Revocations are the second dense stream, and they take the same guard. The lane pages a revocation_delta_log table with its own seq column rather than the mutable revoked-capability projection, so a page has a density to argue from. ensure_revocation_page_ascending checks the response’s cursor version and stream identity against the ones the peer advertised, then ensure_sequence_revocation_page_ascending collects the page’s sequence numbers, refuses a record carrying none with MissingRevocationSequence, and hands them to require_contiguous_page anchored at after_seq + 1. The cursor moves to the last record’s seq, which is the page maximum by construction because the page is consecutive. Revocation Propagation owns that lane in full.

Against a peer that advertises no stream identity the same lane falls back to the second bucket. Its pages come from the projection, ordered by the composite tuple (revoked_at, capability_id), and ensure_legacy_revocation_page_ascending compares each row against its predecessor and returns NonAdvancingPage on the first that fails to advance. It also refuses a cursor or a record carrying any sequence field, as IncompleteRevocationStreamContract and UnexpectedRevocationSequence, so the two contracts cannot be mixed inside one page. What it cannot reject is an omission, which is why the puller clears its cursor on the first empty page and replays the projection from genesis.

The abandoned-seq trust boundary

A rolled-back-then-re-appended budget write leaves a permanent hole in the global seq stream. To keep contiguity satisfiable, a budget delta page may carry abandoned_seqs, and the importer treats the union of imported events and those slots as gap-free. The follower validates that each abandoned seq is in range, strictly ascending, and does not collide with a live event on the same page, then trusts the claim.

Falling behind, and the snapshot backstop

A budget delta page is capped at BUDGET_DELTA_MAX_RECORDS, which is MAX_LIST_LIMIT * 2, so 400 records counting events, paired usage projections, and abandoned slots together. A page can breach that cap merely because the puller asked for a full 200-event window: 200 events plus 200 usages plus one abandoned seq is 401. Routing that straight to a full snapshot would be wasteful, and on a large ledger whose snapshot itself exceeds the 64 MiB response cap it would be a permanent stall.

So drain_budget_delta_pages shrinks before it snapshots. On an over-cap page carrying more than one event it refetches the same cursor with half the event limit, and reopens the limit once a normal window lands. Only a minimal one-event page that still overflows is genuinely unpageable: a dense rollback burst can pack more abandoned seqs than the cap before the next live event, so no cursor-anchored page can make progress. Both halves are tested: oversized_budget_page_retries_smaller_before_snapshotting asserts the smaller refetch happens before any snapshot, and unpageable_single_event_budget_page_still_force_snapshots asserts the unpageable case force-snapshots after exactly one fetch with no shrink-retry loop.

Two limits sit under that. The serving side caps its abandoned list at BUDGET_DELTA_MAX_RECORDS + 1 rows, so a truncated list always trips the puller’s over-cap check before the contiguity guard can see the truncation and demote an honest peer. And an over-cap page is rejected before charge_page runs, so a shrink-retry costs a real HTTP fetch that the 64-page count never sees; only the 20s wall clock bounds that retry chain.

Snapshot recovery is also routine, not only exceptional. CLUSTER_SNAPSHOT_RECORD_THRESHOLD is 8: once a peer has contributed that many delta records since its last snapshot, peer_should_force_snapshot returns true and the next visit begins with a full cluster_snapshot. Applying one resets every cursor to the snapshot heads, records the abandoned seq ranges (range-encoded, so a rollback storm stays a few pairs rather than millions of integers), and clears the peer’s cached ack map in the same update that clears force_snapshot. Clearing the flag while leaving a stale-high ack map would let any later early return leave a healthy peer witnessing at a head this round never validated.

The revocation lane raises the same flag from inside its pull loop as well, three ways, and none of them is a demotion. sync_current_peer_revocations returns PullError::ForceSnapshot when the cursor it holds no longer matches the stream identity the peer advertises, when the response’s headSeq has moved behind that cursor, and when the peer answers an empty page while the cursor still sits below the advertised head. Each of those is a database that was rolled back or replaced rather than a peer that lied, so the peer keeps its standing and takes a snapshot instead.

The snapshot is validated before it is applied, and only that path can hand a reusable revocation cursor back. validate_revocation_snapshot runs first inside apply_cluster_snapshot: it requires the body to be strictly tuple ordered, then returns the advertised cursor only when the peer named a version, a non-empty stream id, a head bound to that same identity, a dense seq on it, and a head row actually present in the projection it just sent. Anything less specific is a refusal with its own message, and a peer advertising neither version nor stream id yields None, which clears the cursor so the next pass starts at genesis.

Who may witness a budget write

A clustered budget write does not return until peers acknowledge it or the wait fails closed. The witness predicate is four conjuncts, and three of them are about standing rather than data.

let acked = peer_state
    .budget_import_acks
    .get(&write.origin_id)
    .is_some_and(|imported_seq| *imported_seq >= write.event_seq);
if peer_state.health.is_reachable()
    && !peer_state.partitioned
    && !peer_state.force_snapshot
    && acked
{
    witness_urls.insert(peer_url.clone());
}

The write is identified by a BudgetWriteToken of origin_id, event_seq, and budget_term, so a peer that holds an unrelated event at a higher magnitude cannot satisfy it. A peer advertises per-origin heads computed from its global contiguous import head: the store finds the largest seq with no hole beneath it, then reports each origin’s maximum event at or below that head. The RFC proposed per-origin gaps-and-islands runs; the shipped store rejects that shape, because an origin whose block starts mid-sequence after a leadership change looks like a gap and would stall its writes.

Quorum size is peers.div_ceil(2) + 1 over the peer list, which excludes self, and the writing node always counts itself. The wait parks on a watch channel and drives no sync of its own, so N concurrent writes share one background loop instead of launching N sync storms. Its timeout scales with the peer count, because a slow peer visited before the peer whose ack makes quorum must be waited out: the per-visit cost above, three 15s blocking stages plus two 20s delta rounds, multiplied by peer_count + 1 cycles, plus one sync interval, floored at 5 seconds and capped at 300. A three-peer cluster already saturates that cap. A genuine partition returns 503 without waiting it out, through the has-quorum check the loop re-runs on every progress tick.

Guarantees and limits

Next Steps

• Revocation Store · the node-local revoked set this stream imports into, and the durability gate over it
• Cluster Overview · membership, reachability, and the quorum arithmetic this page’s witness depends on
• Budgets Across Nodes · the write half: forwarding to the leader, and what the ack witness is asked to back
• Cluster Peer Auth · the digest, the allowlist, and the skew window every request on this page passes through
• Budget Store · authorize, capture, release, and reconcile as one process performs them, before any of it replicates
• Trust Control Plane · running the service: peer URLs, sync interval, failover, and key rotation
• Node State on Disk · the single-writer SQLite substrate every one of these streams is imported into