Detect, Contain, Recover

One incident, from contact to revocation

The tester follows the comment. Its plan calls the honey tool to read the deployment settings, then tries an upload. Until the response applies, the rest of the lineage keeps working.

pub(crate) fn check_revocation(&self, cap: &CapabilityToken) -> Result<(), KernelError> {
    if self.with_revocation_store(|store| Ok(store.is_revoked(&cap.id)?))? {
        return Err(KernelError::CapabilityRevoked(cap.id.clone()));
    }
    for link in &cap.delegation_chain {
        if self.with_revocation_store(|store| Ok(store.is_revoked(&link.capability_id)?))? {
            return Err(KernelError::DelegationChainRevoked(
                link.capability_id.clone(),
            ));
        }
    }
    Ok(())
}

Calls admitted before the response keep their effects and receipts. The fixer stays untouched because the affected set comes from the lineage walk, and sharing a project puts no capability in it. Every refusal happens at the kernel’s next check of a grant, so containment needs no access to the workers’ processes.

Decoys turn misuse into evidence

A decoy is a tripwire the operator plants on purpose: a canary capability, a honey tool, a credential-shaped file, or a signed watermark. Its registry is private and bound to a tenant, so a string that resembles a decoy cannot authenticate itself. The decoy in this example is a honey tool: you list it in the workers’ grants beside the real tools, and no review task calls it.

The kernel’s tripwire guard runs first in its guard pipeline. Before dispatch it refuses a call that names a registered honey tool or presents a registered canary capability, and it publishes the match as a signed security event. Its output hook blocks a result that carries an active signed watermark. The guard sees only calls that pass the scope check, which is why the honey tool sits in the grant: a call outside a grant’s scope is refused there, and no event is published.

Placement decides what contact means. A credential-shaped file among the fixtures asks a different question, because the tester reads the fixtures as ordinary work. The decoy service classifies each contact it is given: a direct presentation is an ActiveMatch that requires an immediate deny, other contact is an ActiveMatch without one, and contact after expiry is an InactiveObservation. No class sets malice_proven; a decoy shows that something reached it, and intent needs separate investigation. Decoys and Tripwires covers placement, lifecycle, and cleanup of the exact file the materializer created.

Observations become signed events

A detector’s observation reaches correlation as a security event: a signed body that binds the tenant, subject, session, capability lineage, source receipt, and producer. Ingress verifies the producer against independently configured keys and refuses an event outside its freshness bounds, so JSON a caller writes in the shape of an event never enters correlation as verified input.

Here the tripwire guard publishes the honey-tool call as a tripwire_observation event, and a detector the host registers signs the refused upload as egress_attempt. Both are among the nine kinds SecurityEventKind names. The producer’s trust class decides whether an event may drive a response. An InternalDetector event may, and the tripwire guard publishes as one. A VerifiedReceipt event, projected from a kernel receipt, may when the correlation policy sets allow_verified_receipt. An event the policy does not permit is reported as AdvisoryOnly: available to an investigator and unable to start a response.

A rule correlates the sequence

An outbound request can be ordinary. A tripwire match followed by an egress attempt in the same lineage within seconds is a different observation: the agent called a decoy, then tried to send data out. A temporal rule states that relationship exactly: the stages in order, the identity that groups them, the window between them, and the state the detector may retain while it waits.

{
  "rule_id": "tripwire-then-egress",
  "policy_version": "larkfield-review-v1",
  "group_by": "lineage_seed",
  "max_groups": 128,
  "max_partial_matches_per_group": 16,
  "allow_event_reuse": false,
  "stages": [
    {
      "name": "tripwire",
      "event_kind": "tripwire_observation",
      "minimum_severity": "low"
    },
    {
      "name": "outbound",
      "event_kind": "egress_attempt",
      "minimum_severity": "low",
      "after": "tripwire",
      "within_ms": 10000
    }
  ]
}

Grouping by lineage_seed follows delegated work. Each event’s subject carries the seed of its grant’s delegation tree, so the rule matches whether the tester makes both calls or the tester makes one and the reviewer the other. Grouping by session_id can miss the second case. The ten-second window, the group budget, and the event-reuse setting are choices for this workload; Temporal Detection tests the boundaries.

A match becomes a correlated finding. It records the rule’s version hash, the ordered event ids, their evidence digests, their source receipts, and the time interval, so an investigator can reconstruct the match instead of trusting a severity score.

A quiet detector and a blind one both report zero findings, and Chio keeps them apart. A detector that cannot parse its input, trust or reach its retained state, finish a scan, or stay within its state budget records typed health evidence, signs it as a receipt, and suppresses automatic response for the affected scope. Repairing the detector restores its ability to decide; it lifts no restriction a response already owns. Detector Health lists the six failure kinds.

A bounded walk names who is affected

A response needs a defensible answer to who is affected. The tenant is too broad: the fixer belongs to the same organization and has nothing to do with the pull request. The set of running workers is too narrow, because a delegated grant can outlive the process that received it.

The blast-radius resolver walks the recorded causal lineage from the correlated finding’s lineage seed, within explicit bounds of at most depth 64, 4,096 nodes, and 8,192 edges, and keeps only capability nodes. An Exact result names the sorted capability ids and commits to them with an affected-set hash and a graph-slice hash. In the example it returns the reviewer, the tester, and triage.

pub enum BlastRadiusResult {
    Exact {
        metadata: BlastRadiusSnapshotMetadata,
        sorted_affected_ids: RecordIdSet,
        affected_set_hash: Digest32,
        graph_slice_hash: Digest32,
    },
    Incomplete {
        metadata: BlastRadiusSnapshotMetadata,
        reason: BlastRadiusIncompleteReason,
    },
}

The same graph answers an operator directly. Walk forward from the reviewer's grant and keep the capability nodes, as the resolver does. The three grants it names come back, and the fixer's grant is absent because no edge reaches it:

chio lineage query --live --seed cap-01f0…3a71 --direction forward --json \
  | jq -r '.nodes[] | select(.kind == "capability") | "\(.id)  \(.subject)"'

cap-01f0…3a71  reviewer
cap-01f0…8d02  tester
cap-01f0…c4e9  triage

An incomplete walk returns no affected set. Truncation, replica lag, or a missing completeness watermark yields Incomplete with a reason and nothing to bind, so no lineage-scoped plan, automatic or governed, can proceed from it.

The response authority admits one exact plan

A correlated finding is evidence, and it grants nothing. A response plan turns it into reviewable data: the triggering finding, the tenant, the affected set and its hash, each effect with its target and contribution, the duration, the operator capability that applies it, and the approval requirement. Changing any binding, even the duration by one millisecond, changes the plan commitment, and the new plan needs its own authorization.

The approval requirement is Automatic or Governed. A governed plan waits for a threshold approval before its dispatch can commit, which suits heavier effects such as suspending a capability set. The response authority runs as its own daemon, accepts requests only from pinned processes, and serves policy from a reviewed snapshot; it mints no policy, capability, or approval online.

A plan that suspends a capability set must begin with FreezeIssuance, which holds new grants out of the lineage while the rest of the plan applies. Under that fence the resolver walks the lineage again and invalidates the approval if the set, its commit, or its hashes moved, so the set being contained is the set that was approved. The kernel’s context-bound issuance path checks the issuance freeze; the POST /v1/capabilities/issue route does not.

Containment takes effect at the next call

Once the plan applies, every capability in the set meets the suspension at admission. The chio-capability-set-suspension guard runs before dispatch, looks up the presented capability in the active suspended sets, and denies a member with guard evidence on its signed receipt. A failed lookup also denies.

Suspension lasts for the plan’s duration and rolls back when it expires. Revocation is the operator’s permanent decision, made separately. Revoking the reviewer’s grant writes one id to the revocation store. On each later call the kernel checks the presented grant’s id and then every ancestor id in its delegation chain, so the reviewer is refused with CapabilityRevoked, and the tester and triage with DelegationChainRevoked naming the reviewer.

Autonomy with Authority draws that walk over a deeper tree. In the security swarm example, revoking the root refuses all six planned descendant calls, and the mailbox they would have written stays empty.

Revocation acts at the next check. A call that completed earlier keeps its effect and its receipt, and a call already in flight meets the dispatch and release checks that Revocation describes.

An emergency stop is the coarse alternative. It refuses new progress for every agent in one kernel process, lives in memory, clears on restart, and revokes nothing, so pair it with revocation when authority has to end for good. On a process host, Contain a Compromised Agent applies three separate controls: chio process revoke-capability revokes a process’s capability and its descendants, chio process revoke withdraws its bearer credentials, and chio process cancel stops new admissions for its process tree.

Recover from the signed record

Chio retains each response effect as its own contribution, with requested, applied, failed, and restored outcomes. When the suspension expires, the scheduler removes only this response’s contribution; a failed removal leaves the response in RollbackPartial until it is reconciled. A Lifted response does not rewind the revocation, erase what the agents read, or authorize a captured operation to run again. Quarantine and Recovery follows overlapping responses through expiry.

The receipts reconstruct the session, and each kind answers a different question.

The kernel receipts show the containment working. After you revoke the reviewer's grant, the tester's next call is refused with the reviewer named as the revoked ancestor:

chio receipt list --agent tester --outcome deny --order newest --limit 1 \
  | jq -c '{id, capability: .capability_id, tool: .tool_name, reason: .decision.reason}'

{"id":"rcpt-01f0…6e93","capability":"cap-01f0…8d02","tool":"run_command","reason":"delegation chain revoked at ancestor cap-01f0…3a71"}

chio receipt list exports the kernel’s receipts, and chio receipt verify checks their signatures against a kernel key you retain independently. To keep the record outside the host, chio-wall siem-export streams the receipt log through chio-siem to Splunk, Elasticsearch, or another configured sink. Investigate a Session shows how to build a timeline from those records that keeps each observation apart from what an investigator inferred.

Go deeper