Revocation Propagation

Lane two, on its own budget

A clustered trust-control node visits each configured peer in turn. Every visit runs two independent pull lanes, and revocations are the second one, deliberately outside the shared round the other four delta streams compete inside. This is shipped and unconditional for any clustered node started with --revocation-db. Inside the cluster boundary the lane is pull-only and driven by the background sync loop: nothing pushes, nothing gossips, and there is no separate revocation protocol.

// Lane 2: revocations replicate on their OWN round budget, INDEPENDENT of the
// shared budget/receipt round, so a sustained budget backlog (shared-round
// exhaustion) or a broken/slow budget-delta endpoint can never starve
// security-critical revocation propagation. It is skipped ONLY when the peer was
// demoted by lane 1 (fail-closed: a peer that violated the wire contract is
// untrusted, so we pull nothing more from it).
if !peer_was_demoted(state, peer_url) {
    let mut revocation_round = PullRoundBudget::new();
    let _ = route_pull(
        state,
        peer_url,
        sync_peer_revocations(
            state,
            &client,
            peer_url,
            &revocation_contract,
            &mut revocation_round,
        ),
        &mut delta_records,
    );
}

The isolation is asserted, not assumed: revocations_are_not_in_the_shared_budget_round compares function pointers against peer_pullers() and fails if anyone ever adds sync_peer_revocations to the shared list. The lane’s own guard has exactly one condition, a peer demoted earlier in the same visit for violating the wire contract, which is a fail-closed refusal to keep reading from a peer that already misbehaved. Four earlier stages of sync_peer can return before the guard is reached at all: a peer marked partitioned, a failed cluster_status, a failed force-snapshot fetch, and a failed authority sync. Independent of the shared round is not independent of the visit.

Read the let _ carefully. The result is discarded. Revocations are the last pull a visit does, so an error here short-circuits nothing, and unless the error was a protocol violation the peer finishes the round with full standing. All three PullError variants can come out of this lane, and the split is the point: Protocol from a page guard demotes the peer, Transient from the transport or the local store leaves its standing alone, and ForceSnapshot routes it to a full resync without calling it misbehavior. The sequence contract raises the third one on a cursor whose stream identity no longer matches, a stream head that has moved behind the local cursor, and an empty page while the cursor still sits below the advertised head. Each of those is a database that was rolled back or replaced, which is a recoverable state rather than a lie.

The parts

What the lane pages is not the revoked-capability set. It is a separate append-only log beside it. upsert_revocation_if_newer writes the projection row and then appends to revocation_delta_log, which carries a dense seq per applied revocation, and list_revocations_after_seq_with_head reads that log rather than the projection:

SELECT seq, capability_id, revoked_at
FROM revocation_delta_log
WHERE seq > ?1
ORDER BY seq ASC
LIMIT ?2

The same call returns next_seq from revocation_delta_meta as the stream head, so a page and the head it was cut from arrive together. A second singleton table, revocation_stream_meta, holds a stream_id that identifies the serving epoch. A replication source rotates that id at activation, before it advertises status, so a database rolled back to an earlier file cannot reuse sequence numbers under an identity a peer already cached. A cursor is valid only for the identity that issued it, which is why the cursor carries the id alongside the number. list_limit clamps limit into 1 through MAX_LIST_LIMIT (200), and the puller always asks for the ceiling.

The older contract is still served, and a peer that advertises neither a cursor version nor a stream id in its replication heads is RevocationPeerContract::Legacy. Its pages come from the mutable projection, paginated by the composite tuple:

SELECT capability_id, revoked_at
FROM revoked_capabilities
WHERE (
    ?1 IS NULL
    OR revoked_at > ?1
    OR (revoked_at = ?1 AND ?2 IS NOT NULL AND capability_id > ?2)
)
ORDER BY revoked_at ASC, capability_id ASC
LIMIT ?3

revoked_at is whole Unix seconds, so ties are common and the capability id is what breaks them. The puller always sends both cursor fields or neither, which is the only combination the tie-break clause handles correctly. That tuple is a pagination token for one pass and nothing more: sync_legacy_peer_revocations calls clear_peer_revocation_cursor on the first empty page, so the next round replays the whole projection from genesis and catches a same-second row that sorts lexically before the previous pass head.

One revocation, end to end

The write is an ordinary cluster-aware mutation. handle_revoke_capability validates the service token, calls forward_post_to_leader, and only then touches a store, so a revoke that lands on a follower is applied once on the leader (Leader & Failover owns that hop). revoke stamps revoked_at with the current whole second and returns whether the row was new; a repeat revoke of the same id rolls back and returns false. Only a genuinely new revoke observes the lag family, and that sample is not measured against the row it just wrote: the handler passes a fresh unix_timestamp_now() rather than the stored revoked_at, so a local sample is zero by construction, up to a second boundary crossed between the two clock reads. respond_after_leader_visible_write then re-reads is_revoked and answers on every successful revoke, new or repeat; only the observation is gated on newness.

Serving is four steps and no state. handle_internal_revocations_delta runs validate_cluster_peer_auth (see Cluster Peer Auth), opens the store, runs the query above, and maps rows into the two-field view. A node without --revocation-db answers 409 with trust control service requires --revocation-db rather than an empty page, so a misconfigured peer is loud on the wire. It is not loud in the health view: that 409 becomes a transient pull error, which writes the body into the peer’s last_error (visible on the internal status route) and forces the health label back to Healthy. Read the error string, not the label.

The pulling side is a bounded loop, and the order of its steps is the whole safety argument:

fn sync_current_peer_revocations(
    state: &TrustServiceState,
    client: &TrustControlClient,
    peer_url: &str,
    store: &SqliteRevocationStore,
    stream_id: &str,
    advertised_head_seq: u64,
    round: &mut PullRoundBudget,
) -> Result<u64, PullError> {
    let mut applied = 0u64;
    loop {
        // Check the shared round budget BEFORE the next blocking fetch so an
        // exhausted stream stops without one more peer request.
        if round.is_exhausted() {
            break;
        }
        let cursor = peer_revocation_cursor(state, peer_url);
        ensure_current_revocation_cursor(cursor.as_ref(), stream_id).map_err(|error| {
            PullError::ForceSnapshot(CliError::cli_other_error(format!(
                "revocation cursor stream epoch changed; snapshot required: {error}"
            )))
        })?;
        let after_seq = cursor.as_ref().and_then(|value| value.seq).unwrap_or(0);
        let response = client.revocation_deltas(&RevocationDeltaQuery {
            cursor_version: Some(REVOCATION_SEQUENCE_CURSOR_VERSION),
            stream_id: Some(stream_id.to_string()),
            after_seq: Some(after_seq),
            after_revoked_at: cursor.as_ref().map(|value| value.revoked_at),
            after_capability_id: cursor.as_ref().map(|value| value.capability_id.clone()),
            limit: Some(MAX_LIST_LIMIT),
        })?;
        ensure_revocation_cursor_version(response.cursor_version)?;
        if response.stream_id.as_deref() != Some(stream_id) {
            return Err(PeerProtocolError::RevocationStreamIdentityMismatch.into());
        }
        let response_head_seq = response
            .head_seq
            .ok_or(PeerProtocolError::MissingRevocationSequence)?;
        if response_head_seq < after_seq {
            return Err(PullError::ForceSnapshot(CliError::cli_other_error(
                "revocation stream head moved behind the local cursor; snapshot required"
                    .to_string(),
            )));
        }
        if response.records.is_empty() {
            if after_seq < advertised_head_seq || after_seq < response_head_seq {
                return Err(PullError::ForceSnapshot(CliError::cli_other_error(
                    "revocation stream ended below its advertised head; snapshot required"
                        .to_string(),
                )));
            }
            break;
        }
        // Stop the round (not demote) when the local per-round pull cap is hit:
        // a large well-ordered backlog resumes next sync round.
        if round.charge_page(response.records.len() as u64).is_err() {
            break;
        }
        // Version 4 pages advance through a dense append-only revocation log
        // bound to one durable stream identity.
        // The validator rejects a missing cursor successor, an interior gap, or
        // a legacy response before the local cursor can advance past omitted
        // revocations.
        let page_head = ensure_revocation_page_ascending(
            cursor.as_ref(),
            response.cursor_version,
            response.stream_id.as_deref(),
            stream_id,
            &response.records,
        )?;
        if page_head.seq.unwrap_or(0) > response_head_seq {
            return Err(PeerProtocolError::IncompleteRevocationStreamContract.into());
        }
        applied = applied.saturating_add(apply_revocation_page(store, &response.records)?);
        update_peer_revocation_cursor(state, peer_url, page_head);
    }
    Ok(applied)
}

The response checks between the fetch and the round charge are where a stale or swapped database is caught. A response whose cursorVersion is not 4 is UnsupportedRevocationCursorVersion; a response whose streamId differs from the one the peer advertised is RevocationStreamIdentityMismatch; a headSeq that has moved behind the local cursor, and an empty page while the cursor still sits below the advertised head, are both PullError::ForceSnapshot rather than a demotion, because a rolled-back peer is not the same thing as a lying one. An empty page at or above the head is caught up and ends the lane without touching the cursor.

The round charge comes before the guard and before any import, so a page that trips the local cap is dropped whole and refetched next round rather than half-applied. Because pages are 200 rows and the page cap is 64, the binding limit for this lane is 12,800 rows per peer per round, or the 20 second wall clock, whichever arrives first; the 200,000 record cap never binds here. The cursor moves exactly once per page, after every row in it is durable, and it moves to the page head the guard returned rather than to anything the peer nominated.

The 20 seconds is a between-fetch check, not a cancellation. Both is_exhausted and charge_page compare against the deadline, but neither can interrupt a request already in flight, and the peer client is built with a CONTROL_HTTP_TIMEOUT of 15 seconds. A lane that starts its last fetch one millisecond inside the deadline can therefore run to roughly 35 seconds. The budget-write quorum timeout is sized for exactly that: it budgets two full PEER_ROUND_WALL_CLOCK_BUDGET rounds plus three HTTP timeouts per preceding peer.

Cursor-anchored and gap-free, not merely ascending

The log the sequence contract pages is dense, and the puller persists the page head. Those two facts have to be checked against each other, because a page that only ascends can still strand rows. A peer at cursor 10 that answers {110, 111} passes a max-advance test, gets imported, and moves the cursor to 111; rows 11 through 109 are then permanently below the cursor and only a full snapshot delivers them. So the guard requires the page to begin at the cursor's successor and to run consecutively from there:

let mut expected = expected_next_seq;
for &seq in seqs {
    if seq != expected {
        return Err(PeerProtocolError::NonContiguousPage {
            expected_seq: expected,
            found_seq: seq,
        });
    }
    expected = expected.saturating_add(1);
}
Ok(())

The page is checked in the order it arrived, not sorted first. Sorting would let a reordered page through the guard and then import wrong, which surfaces as a retryable store error instead of demoting the peer that sent it. ensure_revocation_page_ascending runs three checks before it reaches that loop, then ensure_sequence_revocation_page_ascending collects the page's sequence numbers, refuses a record that carries none with MissingRevocationSequence, and anchors the check at after_seq + 1. The head it returns is the last record's seq, which is the page maximum by construction because the page is consecutive.

The tuple projection gets a weaker guard because it can only support a weaker claim. ensure_legacy_revocation_page_ascending walks the page comparing (revoked_at, capability_id) against the previous row and returns NonAdvancingPage the moment one fails to advance, which rejects an interior reorder and a rewind past the cursor. What it cannot reject is an omission: a projection has no density to argue from, so a peer that silently drops a row passes. That is why the legacy puller throws its cursor away at the end of every pass, and why the periodic snapshot rather than incremental convergence is the backstop on that contract. legacy_projection_cursor_is_safe_only_within_a_full_pass pins that shape directly: an ascending same-second page returns its last row as a head carrying no version, no stream id and no seq; a same-second row that sorts lexically before that head is NonAdvancingPage; and clearing the cursor makes the same backfill acceptable on the next genesis replay.

A violation is PullError::Protocol, which demotes the peer to Unhealthy and flags it for a full snapshot. That removes it from this node’s consensus candidate set and from the budget witness set, and it pins the revocation cursor where it was.

The demotion is not sticky. The next sync round opens with cluster_status, and a successful answer runs update_peer_reachable, which puts health straight back to Healthy before the snapshot probe even runs. What actually persists is the force_snapshot flag: budget_write_quorum_commit_view_locked excludes a flagged peer from the witness set regardless of health, and only a completed snapshot clears it. Consensus candidacy, by contrast, returns after one status call.

The two guards print different lines, and one of them is misleading. A sequence violation is peer returned a page that is not cursor-anchored or has a gap: expected next seq X, found Y, and both numbers are real sequence numbers. A tuple violation reuses NonAdvancingPage, whose two fields are named after_seq and page_max_seq after the integer-sequence streams they were written for. On the tuple contract both labels are wrong. after_seq carries the previous row’s revoked_at, which is the cursor only when the first row is the one that failed, and page_max_seq carries the offending row rather than the page maximum. Both are clamped at zero and cast to u64, and the capability id that actually broke the ordering never appears in the message. Grep for peer returned a non-empty page whose max seq, then read the two numbers it prints as adjacent revoke instants rather than as a maximum and a cursor.

Eight records, then a full snapshot

The delta lane is not the steady state on a busy cluster. Records applied across every lane in one visit are folded into delta_records_since_snapshot at the end of it, by finalize_peer_sync_round, and peer_should_force_snapshot returns true when the explicit flag is set or that counter reaches CLUSTER_SNAPSHOT_RECORD_THRESHOLD, which is 8. The next visit to that peer therefore opens with GET /v1/internal/cluster/snapshot, and apply_cluster_snapshot upserts every revocation in the body, sets the revocation cursor to the head the peer advertised, and zeroes the counter. That head survives only on the sequence contract: validate_revocation_snapshot checks the body is strictly tuple ordered, then returns the advertised cursor only when the peer named a version, a stream id, a dense seq, and a head that is present in the body it just sent. A legacy peer, or an empty set, yields None and the cursor is cleared, so the next pass starts at genesis. The counter is what carries the cadence, not the flag: update_peer_success clears the flag at the end of the same round that set it, and the threshold check still fires on the next visit. Note the fold is conditional. finalize_peer_sync_round returns early for a peer that was demoted this round or is already pending a snapshot, so neither the counter nor the ack heads move on those rounds.

That backstop is load-bearing on the tuple contract for exactly one reason: a composite value cursor is not an insertion order. Node C, holding a cursor of (200, cap-x) against node A, will never see a row A imported afterwards from node B that carries revoked_at = 150, because that row sorts below the cursor and the query filters it out. Clock skew produces the same hole more cheaply and more often: every node stamps revoked_at from its own SystemTime, nothing reconciles those clocks, and a revoke written on a lagging node lands below cursors that other nodes already hold. Incremental pulling cannot repair a backfill below the cursor. The full snapshot can, because it carries the whole set with no cursor at all, and every node pulls every configured peer directly rather than through a spanning tree.

The snapshot is a cadence, not a timer, and the cadence is driven by traffic. update_peer_delta_records returns immediately on a zero count, so a peer pair that has gone quiet never advances the counter and never schedules the repairing snapshot. A hole opened below the cursor on an otherwise idle pair persists until eight further records replicate from that peer, or until something else sets force_snapshot: a protocol violation, a budget window the delta stream cannot page, or a partition injected or healed through POST /v1/internal/cluster/partition.

Propagation lag is exported as a histogram

Lag is not inferred from logs. It is observed at the two capability revoke paths, computed the same way at both, and exported as a real histogram family.

pub(crate) fn observe_capability_revocation_lag(revoked_at: i64) {
    let now = std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .map(|elapsed| elapsed.as_secs() as i64)
        .unwrap_or(0);
    let lag_seconds = now.saturating_sub(revoked_at).max(0) as f64;
    chio_metrics_spec::runtime::families::CAPABILITY_REVOCATION_LAG
        .observe(&["control_plane"], lag_seconds);
}

Four things follow from that helper that a dashboard will not tell you. Local revokes and propagated ones share one series, so on a cluster where most revokes originate locally the near-zero samples pull p95 down. The clock is whole seconds on both ends, so sub-second propagation is indistinguishable from instantaneous. The observation is conditional: apply_revocation_page skips it whenever upsert_revocation_if_newer returns false, so the node that originated a revoke and then re-fetches its own row from a peer records nothing, and a row that arrives from two peers is sampled once rather than twice. And the snapshot path applies revocations without observing anything, so the moment a peer resyncs, its catch-up is invisible to the service level objective. The alert aggregates sum by (le), so all of that lands in one p95 with no way to separate the populations.

Guarantees and limits

Next Steps

• Revocation Store · the set this lane writes into, and the durability gate that denies dispatch without one
• Replication & Convergence · the round this lane rides in, its four peer-standing outcomes, and the snapshot backstop in general
• Revocation Oracle · the signed epoch-root path this page excludes, and the freshness window it is gated on
• Cluster Peer Auth · who is allowed to ask for a delta page, and what the digest covers
• Leader & Failover · the forward that decides which node performs the revoke in the first place