Replication & Convergence

Replication streams

Budget events must arrive in ascending order, and their union with valid abandoned slots must be contiguous from the cursor. The authority merge runs when the receiving node has an authority database and must succeed before either lane runs: if the receiver has --authority-db and the peer does not, the peer’s snapshot route returns 409 and the visit ends with no further pulls or acknowledgment increases. Acknowledgments already recorded stay as they were, apart from status clamping and any snapshot already applied. On a receiver without an authority database, sync_peer_authority does no work; Authority & Rotation separates replicated public state from signing custody.

Pull outcomes

A failed or rejected page never moves the cursor past missing data, and it never rolls back pages already imported in the visit. Read last_error as current peer state: update_peer_success clears it for an unpartitioned peer.

Work allowances

The deadline stops further fetches; an in-flight request finishes, so the last one can end after the 20 seconds. The authority merge and full cluster snapshot sit outside both allowances, and an oversized-page retry sends an HTTP request of its own and uses no charged-page slot. The server caps an abandoned-slot list at 401 entries, so truncation trips the receiver’s oversized-page check before its contiguity check.

Page validation by stream

• Budget mutation stream. Received events ascend, and events plus validated abandoned slots form a gap-free sequence from the cursor’s successor. The receiver checks that reported abandoned slots are ordered, in range, and clear of live events on the page, then accepts the source’s claim that those positions are empty.
• Revocation log. Version and stream identity match, every record carries a sequence, and the page runs consecutively from the cursor’s successor.
• Tool receipts, child receipts, lineage. The sorted page advances beyond the cursor and repeats no sequence.
• Tuple revocation contract. Each (revoked_at, capability_id) tuple advances and the sequence fields are absent. Contiguity cannot detect an omission here, and an empty page resets the cursor so the next pass replays from the beginning.

Budget pages also refuse usage records without mutation events, abandoned slots without anchoring events, and invalid abandoned sequences. The records-without-events check runs before the oversized-page check, so that shape is a protocol violation even on an oversized page.

Revocation stream contract

The status response selects the revocation contract once per visit. A peer serving a current revocation store advertises cursor version 4 and a stream identity, even with no records; advertising neither selects the tuple contract, and advertising half the pair is a protocol error. A peer read under the tuple contract inherits no sequence cursor: version, identity, and sequence are cleared before the pass begins.

For a sequenced stream, a cached cursor with a different identity, a missing sequence, the wrong version, or a sequence above the advertised head requires snapshot repair. Those conditions fit a replaced or rolled-back source, and the receiver resynchronizes without diagnosing which. A snapshot must name the same stream and a head at least as high as the status advertised; a changed or regressed contract is refused before the body applies. Revocation Propagation specifies page and snapshot validation.

pub(crate) fn revocation_peer_contract(
    replication: &ClusterReplicationHeadsView,
) -> Result<RevocationPeerContract, PeerProtocolError> {
    match (
        replication.revocation_cursor_version,
        replication.revocation_stream_id.as_deref(),
    ) {
        (None, None) => Ok(RevocationPeerContract::Legacy),
        (Some(version), Some(stream_id)) => {
            ensure_revocation_cursor_version(Some(version))?;
            if stream_id.is_empty() {
                return Err(PeerProtocolError::MissingRevocationStreamIdentity);
            }
            let head_seq = match replication.revocation_cursor.as_ref() {
                None => 0,
                Some(cursor)
                    if cursor.cursor_version == Some(version)
                        && cursor.stream_id.as_deref() == Some(stream_id)
                        && cursor.seq.is_some_and(|seq| seq > 0) =>
                {
                    cursor.seq.unwrap_or(0)
                }
                Some(_) => {
                    return Err(PeerProtocolError::IncompleteRevocationStreamContract);
                }
            };
            Ok(RevocationPeerContract::Current {
                stream_id: stream_id.to_string(),
                head_seq,
            })
        }
        _ => Err(PeerProtocolError::IncompleteRevocationStreamContract),
    }
}

Acknowledgment ordering

After the status and contract checks, clamp_down_peer_budget_acks sets each recorded head to min(recorded, advertised) and removes origins the peer no longer advertises, so a stale high acknowledgment stops counting before new pull work begins. Increases wait for finalize_peer_sync_round, which checks for demotion or a pending snapshot before it records the advertised heads, updates peer success, and wakes waiting writers. That gate reads peer standing: a transient pull error preserves eligibility and still reaches it.

Waiting writes subscribe to shared progress notifications and start no sync loop of their own. The wait grows with the configured peer count, is capped at 300 seconds, and fails early when the node detects it has lost local quorum. The timeout is the caller’s wait policy.

Shutdown between peers

The sync loop checks for shutdown before it starts the next peer. A visit in progress completes its blocking work, so at most one visit remains: status, a snapshot, the authority merge, and both lanes. The budget-write timeout models a visit as three 15-second HTTP stages and two 20-second allowances; that 85-second figure is arithmetic for the timeout, since pull deadlines leave in-flight fetches running and local import takes time of its own.

Source tests

• Dense sequence validation. require_contiguous_page_rejects_cursor_jump_and_interior_gap, sequence_revocation_page_must_be_cursor_anchored_and_dense.
• Allowance exhaustion. charge_page_reports_round_limits_not_protocol_errors, is_exhausted_flags_spent_budget_before_the_next_fetch.
• Acknowledgment ordering. regressed_ack_head_is_cleared_before_validation_not_witnessed_at_old_high, with the gate in finalize_peer_sync_round.
• Same-origin witnessing. prop_witness_never_overclaims_durability, witness_requires_same_origin_ack.
• Oversized-page repair. oversized_budget_page_retries_smaller_before_snapshotting, unpageable_single_event_budget_page_still_force_snapshots.

The visit order lives in crates/platform/chio-control-plane/src/trust_control/cluster/deltas.rs.