Investigate a Session

An investigation ends in a bounded statement about one operation: what happened on the observed path, which signed evidence supports it, what remains unknown, and which owner repairs it.

A caller's error, a signed decision, and a resource observation describe different parts of the same attempt. Preserve all three, then join them through the original operation's identities before you call it refused, completed, or unresolved.

1. Name the original operation

Record the request, authenticated caller, session, tool, destination, and approximate time. Retain the original operation key and request commitment where the execution path provides them, and locate the host that admitted the work and the resource owner that could have executed it.

Follow the operation through admission, dispatch, output, and recovery. A denied call, an evaluator failure, and a lost provider response look alike to the caller and need different repairs, so find the boundary that produced the symptom first.

2. Export the original receipts

Use the installed chio and jq, with administrative read access to the deployment's receipt database. Set the original store and the tool-server ID under investigation, and export into a new private directory, which keeps evidence already gathered intact.

set -eu
umask 077
: "${CHIO_RECEIPT_DB:?Set the original deployment receipt database path}"
: "${CHIO_TOOL_SERVER:?Set the tool-server ID under investigation}"
: "${CHIO_INCIDENT:?Set a new private incident directory}"
test ! -e "$CHIO_INCIDENT"
mkdir -m 0700 "$CHIO_INCIDENT"
chio --receipt-db "$CHIO_RECEIPT_DB" receipt list \
  --admin-all --tool-server "$CHIO_TOOL_SERVER" --limit 100 \
  > "$CHIO_INCIDENT/receipts.jsonl" 2> "$CHIO_INCIDENT/selection.stderr"

You should see one receipt per line in receipts.jsonl. When selection.stderr prints next_cursor=<seq> total_count=<n>, repeat the query with --cursor set to that value under the same database, scope, filter, and limit, and save each page and its stderr separately. Continue until no next cursor remains, as receipt pagination describes: a partial export cannot show that an absent receipt never existed.

3. Verify the export under an independent key

Take the signer from trusted configuration, never from a key copied out of the suspect artifact. The host owner supplies the original kernel public-key file from independent configuration.

set -eu
: "${CHIO_KERNEL_PIN:?Set the independently retained original kernel public-key file}"
chio receipt verify --json --input "$CHIO_INCIDENT/receipts.jsonl" \
  --trusted-kernel-pubkey "$CHIO_KERNEL_PIN" \
  > "$CHIO_INCIDENT/verification.json" 2> "$CHIO_INCIDENT/verification.stderr"

You should see verification.json. Keep the input and any error: a failure is incident evidence. chio receipt verify checks original signatures, the signer pin, and action hashes offline; policy re-evaluation, log inclusion, and the external provider are separate questions.

A valid signature and relevance to this operation are two checks, as verify security evidence explains. Check tenant, agent, session anchor, authentication epoch, capability, request commitment, and tool identity. For caller delivery, add the executor pin and original delivery binding; for a confined tool, compare the launch's manifest, executable, and descriptor commitments.

Once signatures and bindings pass, write a separate reading aid. The signed JSONL stays unchanged, so each projected entry traces to its envelope.

jq -c '{id, tool_server, tool_name, decision, receipt_kind}' \
  "$CHIO_INCIDENT/receipts.jsonl" > "$CHIO_INCIDENT/summary.jsonl"

jq -s '
  group_by([.tool_server, .tool_name, .decision])
  | map({
      tool_server: .[0].tool_server,
      tool_name: .[0].tool_name,
      decision: .[0].decision,
      count: length,
      receipt_ids: map(.id)
    })
' incident/receipts.jsonl > incident/by-tool-and-decision.json

4. Order the observations by what each owner can establish

The kernel records admission and capture, the executor reports its own work, and the provider or resource exposes the resulting state. Keep a working timeline beside the original artifacts, one entry per observation:

1. Operation. The original operation identity the observation belongs to.
2. Times. When the event occurred, if authenticated; when another component learned of it; when you collected the record.
3. Source. The producing component and the original artifact reference.
4. Fact. What that artifact establishes.
5. Inference. Your hypothesis, kept in its own field so it stays distinct from signed observations as the incident changes hands.

Sort by cause as well as by timestamp. A receipt that references a prior operation or effect fixes an order that wall clocks miss; a late receipt can describe an effect that completed first, and an early timeout can precede a provider response that never reached the caller. Join observations through the original request and retained identities before you treat them as one sequence.

A provider object beside an unknown kernel outcome marks a missing reconciliation step; a denied admission beside a changed resource points to another effect path or a miscorrelated record. Keep both disagreeing observations, and investigate the boundary between them.

5. Classify the outcome by the boundary it reached

The last boundary the operation crossed decides what the observation means and where to look next.

An evaluator failure stays distinct from a policy denial even when no output reached the client. The signed session report binds the failure observation and retains execution_outcome=unknown.

For an active-defense finding, inspect the versioned rule, group commitment, ordered event IDs, evidence digests, source receipts, and time interval. Compare event time with ingest time, and read detector-health records for lateness, eviction, or store failure. Follow the finding to its authorized plan and each requested, applied, failed, and rolled-back effect, and verify the affected-set commitment; a session-wide status can hide one remaining contribution. A failed publication walks these boundaries in order.

6. Trace one message across a cancellation

Use the stopped host from the containment rehearsal. The trace establishes whether worker sent cancel-effect-1 before cancellation and whether later refused work changed the mailbox. The native server is chio-ipc, its tool is send_effects, and the receipt's process metadata carries the original operation key.

Run the query from the extracted rehearsal directory. Supply the kernel public-key file the host owner retained, never a pin taken from the suspect receipt's kernel_key; a verification error stops the projection and keeps the export and its stderr.

set -eu
umask 077
CHIO_STATE="$PWD/.chio/cancellation/host"
CHIO_INCIDENT="$PWD/incident-cancellation"
test ! -e "$CHIO_INCIDENT"
mkdir -m 0700 "$CHIO_INCIDENT"
chio receipt list --receipt-db "$CHIO_STATE/receipts.db" \
  --admin-all --tool-server chio-ipc --limit 100 \
  > "$CHIO_INCIDENT/original-receipts.jsonl" \
  2> "$CHIO_INCIDENT/selection.stderr"
# The owner supplies the original trusted key, independently of the export.
: "${CHIO_KERNEL_PIN:?Set the path to the independently retained original kernel public-key file}"
chio receipt verify --json --input "$CHIO_INCIDENT/original-receipts.jsonl" \
  --trusted-kernel-pubkey "$CHIO_KERNEL_PIN" \
  > "$CHIO_INCIDENT/verification.json" 2> "$CHIO_INCIDENT/verification.stderr"
jq -c 'select(.metadata.chio_process.operation_key == "cancel-effect-1") |
  {tool_server, tool_name, decision: .decision.verdict,
   operation_key: .metadata.chio_process.operation_key,
   dispatch_state: .metadata.admission_operation.projected_dispatch_state,
   state: .metadata.admission_operation.projected_state}' \
  "$CHIO_INCIDENT/original-receipts.jsonl"

{"tool_server":"chio-ipc","tool_name":"send_effects","decision":"allow","operation_key":"cancel-effect-1","dispatch_state":"terminal","state":"completed"}

You should see one allow receipt, dispatch_state terminal and state completed, for the operation before cancellation. The projection omits receipt IDs and timestamps, which change per run; the original NDJSON stays beside it.

The refused call left no receipt, because authentication and cancellation refusals happen before kernel evaluation. Its absence from the query therefore shows nothing about whether a request reached the host. Confirm the rest outside the receipt store: the stopped mailbox store should hold the cancel-effect-1 row alone, and the later cancelled wire response belongs with the original evidence.

7. Route the repair and close with a bounded statement

Each missing fact belongs to the owner that can establish it; a general retry cannot create it.

• An identity mismatch goes to the authenticated host and its authority configuration.
• A captured effect with an unknown outcome goes to the original execution path, keeping the original attempt and holds while you reconcile that operation.
• A missing release checkpoint goes to the output and release owner.
• A partial response goes to the adapter and worker that own the outstanding contribution. For credential access followed by egress, join the rule's ordered event IDs and evidence digests to its finding, plan, and individual effect and removal records with the correlation and action inspector.

Keep captured authority attached to the original operation until its effect is reconciled. A fresh credential attempt, a refunded captured budget, or a resend hides the incident without resolving it; repair custody, authority, or store availability first, under valid serving fences. Quarantine recovery and key transparency take over when the failing boundary is theirs.

Before repair, restrict access to provider responses, decoy records, and sensitive request bodies; routine summaries cite commitments and artifact locations. Close with the bounded statement from the opening, naming the new work it authorizes and every alternate path outside your observation, so the next operator reads an incomplete history as incomplete.

Evidence by boundary

The identities and records each boundary of an operation contributes, for lookup during an incident.

Next: act on what the investigation established