Quarantine and recovery

Each response removes only the restriction it owns

One shared quarantined flag would let the first incident to close clear the other incident's protection. Owned contributions name the reason for each restriction: a response adds its bounded effect, later removes that effect through the same retained identity, and the resource stays restricted while any other contribution applies.

Response A restricts the session's egress for five minutes and response B for ten; the durations are illustrative. When A expires, the protected runtime keeps enforcing B through A's whole rollback. When B later restores, the session's other policy and controls still decide what work is permitted, and both response histories stay inspectable throughout.

If A's removal was requested and its adapter result is missing, A stays unresolved under its original response and effect identities. A process exit or an expired timer says nothing about the adapter. The two cases to rehearse are B expiring first and a restart between A's rollback request and its result: in neither may B's protection lapse or A apply again.

A response applies its authorized plan once per effect

Before a response applies, its finding, tenant, affected-set commitment, policy, operator capability, approval, duration, and effect list are verified, and the exact authorized plan is kept with its artifact reference. Changing an affected identity, a contribution, or the duration needs that authorization checked again.

One plan can throttle a session, restrict egress, and suspend a capability set, and each effect succeeds or fails on its own. The worker persists the effect request before it calls the adapter, then records the applied or failed result under the same effect identity. Lease owners and fences stop competing workers from applying one contribution twice or changing work that another live executor owns.

These names carry operational meaning and stay visible in receipts and incident tooling. A partial state is the record of exactly which controls are active, which failed, and which removals still need attention.

pub enum ResponseState {
    Planned,
    AwaitingApproval,
    Applying,
    Active,
    ApplyPartial,
    Expiring,
    RollingBack,
    RollbackPartial,
    Cancelled,
    Expired,
    Failed,
    Lifted,
}

Expiry starts removal; a restored result ends it

A timer decides when removal is due, and the adapter's retained result decides whether removal happened. When the duration ends, the response passes through Expiring into RollingBack, and each rollback record carries one of three outcomes:

pub enum ResponseRollbackOutcome {
    Requested,
    Restored { resulting_version_hash: Digest32 },
    Failed { error_code: ErrorCode },
}

Requested records the intent to restore an effect, Restored carries the resulting version hash, and Failed carries an error code. A conflict, a stale fence, a missing store, or a failed adapter keeps the response in a partial or failed recovery state with that evidence. An alert escalation has no reverse, since a sent alert stays sent, so the plan names which effects need restoration before the response can read Lifted.

Each record also retains the response generation, effect generation, scheduler lease owner and fencing token, effect transition, and prior receipt. Those fields separate the original removal from a new request that happens to name the same target. On a stale-fence or version conflict, compare the effect generation and transition references: a reacquired lease continues the original contribution and never rewrites it. Retrying the whole plan as a new response would hide which original contributions remain and could duplicate a mutation.

pub struct ResponseRollbackRecord {
    pub transition_id: RecordId,
    pub generation: u64,
    pub effect_id: EffectId,
    pub effect_generation: u64,
    pub outcome: ResponseRollbackOutcome,
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub scheduler_lease_owner_id: Option<LeaseOwnerId>,
    pub scheduler_fencing_token: u64,
    pub effect_transition_id: Option<RecordId>,
    pub prior_receipt_id: OpaqueReceiptRef,
    pub occurred_at_unix_ms: u64,
}

A replacement worker continues the original response

The configured host supplies the durable store, the authorized executor, and the health sink, and the scheduler continues fixer-egress-A as the same response. Its plan, affected identities, approval, and duration were retained through response authorization. Each poll passes A's action_id; tenant and worker identities, the claim identity, and time come from the trusted host context.

use crate::inspect_response::inspect_response;
use crate::response_outcomes::inspect_tick;
use crate::response_worker::process_due;
use chio_quarantine::{
    ResponseScheduler, ScheduledResponseExecutor, SchedulerError, SchedulerWorkOutcome,
    StateMachineError,
};
use chio_security_types::ports::{
    ActionId, LeaseOwnerId, RecordId, ResponseSchedulerStore, ResponseStore, SchedulerHealthPort,
    TenantId,
};
use chio_security_types::ResponseSnapshot;

pub struct ResponseObservation {
    pub tick: Result<Vec<SchedulerWorkOutcome>, SchedulerError>,
    pub before: Option<ResponseSnapshot>,
    pub after: Option<ResponseSnapshot>,
}

// The action belongs to an already authorized, retained response. The worker
// polls a batch; the two snapshots inspect this particular original action.
pub fn poll_response<S, X, H>(
    worker: &ResponseScheduler<S, X, H>,
    response_store: &S,
    tenant_id: TenantId,
    action_id: ActionId,
    claim_id: RecordId,
    lease_owner_id: LeaseOwnerId,
    now_unix_ms: u64,
) -> Result<ResponseObservation, StateMachineError>
where
    S: ResponseSchedulerStore + ResponseStore + ?Sized,
    X: ScheduledResponseExecutor + ?Sized,
    H: SchedulerHealthPort + ?Sized,
{
    let before = inspect_response(response_store, tenant_id.clone(), action_id.clone())?;
    let tick = process_due(
        worker,
        tenant_id.clone(),
        claim_id,
        lease_owner_id,
        now_unix_ms,
    );
    match &tick {
        Ok(outcomes) => inspect_tick(outcomes),
        Err(error) => eprintln!("tick failed: {error}; earlier batch work may have run"),
    }
    let after = inspect_response(response_store, tenant_id, action_id)?;
    Ok(ResponseObservation {
        tick,
        before,
        after,
    })
}

A tick can process other due responses, so the function inspects every returned item and then reloads A. On A's expiry the executor removes egress-A, and a failed removal leaves A partial with a retry scheduled. After a worker restart, reopen the same durable stores, acquire a new valid worker owner, and run the same function with A's original action. A's retained history then reads:

1. Active, egress-A: Applied. Investigate within A's five-minute window.
2. RollbackPartial, egress-A: RollbackFailed. Preserve the effect's generation, failed result, and receipt, then inspect the adapter.
3. The same action and effect, in a replacement worker. Reconcile the requested removal with the adapter's durable state under current ownership before any new mutation.
4. Lifted, egress-A: Restored. Confirm the control itself and every other response before resuming.

The progression illustrates retained records, not a captured egress run. A failed result leaves open whether the adapter changed the resource before its acknowledgement was lost, which is why reconciliation reads the adapter's durable state, and a later reconciliation records restoration without a new contribution. Restoring A says nothing about B.

use chio_quarantine::{
    ResponseScheduler, ScheduledResponseExecutor, SchedulerError, SchedulerPolicy,
    SchedulerTickRequest, SchedulerWorkOutcome,
};
use chio_security_types::ports::{
    LeaseOwnerId, RecordId, ResponseSchedulerStore, SchedulerHealthPort, TenantId,
};
use std::sync::Arc;

// The embedding supplies durable stores, an authorized effect executor and
// a health sink. Construct this once for a worker, not once per polling tick.
pub fn response_worker<S, X, H>(
    store: Arc<S>,
    executor: Arc<X>,
    health: Arc<H>,
) -> Result<ResponseScheduler<S, X, H>, SchedulerError>
where
    S: ResponseSchedulerStore + ?Sized,
    X: ScheduledResponseExecutor + ?Sized,
    H: SchedulerHealthPort + ?Sized,
{
    ResponseScheduler::new(
        store,
        executor,
        health,
        SchedulerPolicy {
            lease_duration_ms: 30_000,
            base_backoff_ms: 1_000,
            max_backoff_ms: 10_000,
            operator_page_threshold_ms: 60_000,
            max_claims: 8,
        },
    )
}

// Identity, tenant and time come from the trusted worker context.
pub fn process_due<S, X, H>(
    worker: &ResponseScheduler<S, X, H>,
    tenant_id: TenantId,
    claim_id: RecordId,
    lease_owner_id: LeaseOwnerId,
    now_unix_ms: u64,
) -> Result<Vec<SchedulerWorkOutcome>, SchedulerError>
where
    S: ResponseSchedulerStore + ?Sized,
    X: ScheduledResponseExecutor + ?Sized,
    H: SchedulerHealthPort + ?Sized,
{
    worker.tick(&SchedulerTickRequest {
        tenant_id,
        claim_id,
        lease_owner_id,
        now_unix_ms,
    })
}

Construct the scheduler once per worker. The example leases claimed work for 30 seconds, claims at most 8 items per tick, bounds retry backoff between 1 and 10 seconds, and pages an operator at 60 seconds, an example policy rather than a universal deadline. A tick grants no operator new scope and lets no tool write a plan: response authorization binds the finding, affected set, and exact effects, and the response authority service installs that decision for the executor.

Every claimed item reports its own outcome

A tick returns one outcome per claimed item, and its outer Result reports a tick that failed partway through its batch.

use chio_quarantine::SchedulerWorkOutcome;

pub fn inspect_tick(outcomes: &[SchedulerWorkOutcome]) {
    for outcome in outcomes {
        match outcome {
            SchedulerWorkOutcome::Completed { action_id, state } => {
                println!("{}: completed work, state={state:?}", action_id.as_str());
            }
            SchedulerWorkOutcome::RetryScheduled {
                action_id,
                not_before_unix_ms,
                error_code,
                ..
            } => {
                println!(
                    "{}: retry after {not_before_unix_ms}, error={error_code:?}",
                    action_id.as_str(),
                );
            }
            SchedulerWorkOutcome::LeaseLost { action_id } => {
                println!(
                    "{}: lease lost; this worker cannot mutate",
                    action_id.as_str()
                );
            }
            SchedulerWorkOutcome::ProcessingFailed {
                action_id,
                error_code,
            } => {
                println!(
                    "{}: inspect processing failure {error_code:?}",
                    action_id.as_str()
                );
            }
        }
    }
}

Completed names a response in Active, Cancelled, Expired, Failed, or Lifted; here Active is a finished scheduling result, never permission to reopen the session. A RollbackPartial response returns RetryScheduled with response.rollback_partial, and other unfinished states, ApplyPartial and RollingBack among them, retry with response.execution_incomplete. Each retry names the original action, the earliest next time, and the error code.

LeaseLost stops a stale worker from mutating, and ProcessingFailed reports a per-item failure on its own. A tick that returns an error may already have run the items ahead of the failure, so keep the original identities and inspect the retained records after it. When the store itself is unavailable, hold the affected work until its original state can be read and reconciled.

Work resumes against the retained effects and current checks

Load the original response by its tenant and action. decode_response_record checks that the canonical snapshot agrees with the stored record, and the function prints progress for each retained effect, returning a missing record apart from a decoding or store failure.

use chio_quarantine::{decode_response_record, StateMachineError};
use chio_security_types::ports::{ActionId, ResponsePlanKey, ResponseStore, TenantId};
use chio_security_types::ResponseSnapshot;

pub fn inspect_response<S: ResponseStore + ?Sized>(
    store: &S,
    tenant_id: TenantId,
    action_id: ActionId,
) -> Result<Option<ResponseSnapshot>, StateMachineError> {
    let key = ResponsePlanKey {
        tenant_id,
        action_id,
    };
    let Some(record) = store.load_plan(&key)? else {
        return Ok(None);
    };

    // Decode verifies that the canonical snapshot agrees with the record.
    let snapshot = decode_response_record(&record)?;
    println!(
        "state={:?} generation={}",
        snapshot.state, snapshot.generation
    );
    for effect in snapshot.plan.effects.as_slice() {
        println!(
            "effect={} progress={:?}",
            effect.effect_id.as_str(),
            snapshot.effect_progress(&effect.effect_id),
        );
    }
    Ok(Some(snapshot))
}

An effect that already reads Restored stays restored while the worker repairs one that reads RollbackFailed. Resume once every required reversible effect is restored and the original task passes current authority and data-flow checks. A Lifted record describes one response: revocations stay in force, learned information stays learned, and no captured operation gains a second run. Permanent revocation remains a separate, manual decision.

The retained incident decision holds the triggering evidence, authorized scope, applied effects, each removal result, and the checks that permitted resumption. The rehearsal set covers successful application and expiry, a failure after the first effect, process death during rollback, overlapping responses, stale ownership, and a store outage, each read against the controls and receipts together. A recovered response is distinct from an external operation whose outcome stays unknown.

Read more in Investigation.

Next: promote these controls for a cohort