Revocation Propagation

Each node keeps its own log and a cursor for every source

The local projection answers whether a capability is revoked. A dense, append-only log beside it records each change, and a cursor names a source stream and a position in that stream.

An accepted revocation updates the projection and appends one local log event in the same SQLite transaction. In the import drawn here, node A serves records 101 and 102 for two capabilities that are new to node B. Node B appends them at its own positions 41 and 42 and records its cursor for stream A at 102 once the whole page has applied.

A source rotates its stream identity each time a database is activated as a replication source, before it advertises status. A restored database therefore never reuses a serving identity it held before, and the receiver keeps each source cursor apart from its own positions.

Read more in Revocation Store, which covers local durability and kernel consultation.

A revocation is written at the leader and pulled by each peer

A service-authenticated POST /v1/revocations follows the locally selected leader to the store, rechecks local visibility, and returns the leader-visible result. Other nodes import the record on their next peer visit.

The receiver runs the page guard, then commits each changed row's projection update and log append in that row's own transaction. The source cursor advances after the complete page applies, so a failure on a later row leaves the preceding rows applied under the old cursor.

fn sync_current_peer_revocations(
    state: &TrustServiceState,
    client: &TrustControlClient,
    peer_url: &str,
    store: &SqliteRevocationStore,
    stream_id: &str,
    advertised_head_seq: u64,
    round: &mut PullRoundBudget,
) -> Result<u64, PullError> {
    let mut applied = 0u64;
    loop {
        // Check the shared round budget BEFORE the next blocking fetch so an
        // exhausted stream stops without one more peer request.
        if round.is_exhausted() {
            break;
        }
        let cursor = peer_revocation_cursor(state, peer_url);
        ensure_current_revocation_cursor(cursor.as_ref(), stream_id).map_err(|error| {
            PullError::ForceSnapshot(CliError::cli_other_error(format!(
                "revocation cursor stream epoch changed; snapshot required: {error}"
            )))
        })?;
        let after_seq = cursor.as_ref().and_then(|value| value.seq).unwrap_or(0);
        let response = client.revocation_deltas(&RevocationDeltaQuery {
            cursor_version: Some(REVOCATION_SEQUENCE_CURSOR_VERSION),
            stream_id: Some(stream_id.to_string()),
            after_seq: Some(after_seq),
            after_revoked_at: cursor.as_ref().map(|value| value.revoked_at),
            after_capability_id: cursor.as_ref().map(|value| value.capability_id.clone()),
            limit: Some(MAX_LIST_LIMIT),
        })?;
        ensure_revocation_cursor_version(response.cursor_version)?;
        if response.stream_id.as_deref() != Some(stream_id) {
            return Err(PeerProtocolError::RevocationStreamIdentityMismatch.into());
        }
        let response_head_seq = response
            .head_seq
            .ok_or(PeerProtocolError::MissingRevocationSequence)?;
        if response_head_seq < after_seq {
            return Err(PullError::ForceSnapshot(CliError::cli_other_error(
                "revocation stream head moved behind the local cursor; snapshot required"
                    .to_string(),
            )));
        }
        if response.records.is_empty() {
            if after_seq < advertised_head_seq || after_seq < response_head_seq {
                return Err(PullError::ForceSnapshot(CliError::cli_other_error(
                    "revocation stream ended below its advertised head; snapshot required"
                        .to_string(),
                )));
            }
            break;
        }
        // Stop the round (not demote) when the local per-round pull cap is hit:
        // a large well-ordered backlog resumes next sync round.
        if round.charge_page(response.records.len() as u64).is_err() {
            break;
        }
        // Version 4 pages advance through a dense append-only revocation log
        // bound to one durable stream identity.
        // The validator rejects a missing cursor successor, an interior gap, or
        // a legacy response before the local cursor can advance past omitted
        // revocations.
        let page_head = ensure_revocation_page_ascending(
            cursor.as_ref(),
            response.cursor_version,
            response.stream_id.as_deref(),
            stream_id,
            &response.records,
        )?;
        if page_head.seq.unwrap_or(0) > response_head_seq {
            return Err(PeerProtocolError::IncompleteRevocationStreamContract.into());
        }
        applied = applied.saturating_add(apply_revocation_page(store, &response.records)?);
        update_peer_revocation_cursor(state, peer_url, page_head);
    }
    Ok(applied)
}

On a retry, a revokedAt at or below the stored one is ignored, and a strictly newer one advances the projection with another local log event. The timestamp comes from the peer and is unbounded against the receiver's clock: a future value can dominate later updates and zero the lag reading, while the capability stays revoked.

A page must start at the cursor’s successor and carry no gap

At source cursor 100, the next accepted page begins at 101 and continues consecutively; a page that begins at 110 fails even when its records ascend. With no cursor, the puller begins at zero and requires sequence 1. Every record carries a sequence, and the page's last sequence stays at or below the advertised head.

1. Protocol error. A version or stream identity outside the negotiated contract, a missing sequence, a gap, or a rewind demotes the peer and requires a snapshot. The failing page leaves the cursor in place.
2. Snapshot recovery. A cached cursor from another stream, a head that regresses below the cursor, or an empty page below an advertised head requests a snapshot without demotion. The snapshot repairs the receiver; the source inconsistency still needs an operator to trace it.
3. Allowance spent. Pulling stops for the visit and previous progress stands. A backlog is never treated as a protocol violation.
4. Transient error. A request or local store error keeps the peer healthy. Finalization can still run in the same visit when no snapshot is pending, and can clear the error there.

Revocations draw a fresh allowance after the shared budget, receipt, and lineage lane, so a spending backlog cannot consume their pages: 64 pages, 200,000 records, and a 20-second deadline checked around blocking fetches. The lanes run in sequence, so status, snapshot, or authority preparation can still end a visit before revocation pulling begins.

A required snapshot also removes the peer from the budget witness set and holds back its acknowledgment finalization. A demoted peer can report healthy on a later probe while its snapshot repair is still pending.

Read more in Replication & Convergence, which shows the scheduler that runs each visit.

The lag metric samples a record’s age when a row changes

chio_capability_revocation_lag_seconds observes the gap between the receiver's current Unix second and the record's revoke timestamp. A newly originated revoke samples near zero; a replicated row samples only when its upsert changes the projection.

The ChioRevocationLagHigh alert fires when the aggregate p95 stays above 30 seconds for 10 minutes over a five-minute rate window. It sums buckets with sum by (le), so local and replicated samples share one percentile, and near-zero local samples pull it down.

Snapshot catch-up records no sample. Whole-second timestamps hide sub-second differences, independent clocks add skew, and a future timestamp clamps to zero. Read the metric beside each peer's cursor, its snapshot progress, and the revocation records a workflow depends on.

Propagation is row replication inside one operator’s trust domain

A replicated row shows that the receiving store holds the revocation, and the Revocation Store integration decides when a kernel admission consults it. The interface has no un-revoke operation and trusts authenticated peers inside the configured administrative boundary.

The cluster offers each node's own view, with no linearizable fleet-wide view and no fixed propagation deadline. Signed cross-operator revocation evidence comes from the Revocation Oracle, and swarm revocation epochs carry their own contract and consumers.

Revocation delta, snapshot, and metric

Next: collect the receipts those nodes sign