Recover an Operation

Bring a stopped host back on its original state and resolve the calls it was serving without repeating their effects. A completed read replays its retained response and receipt; an interrupted read stays OutcomeUnknownAfterDispatch.

Recovery continues work that already has an identity: the replacement host reopens the original stores, takes serving ownership, and finds which transition survived.

1. Resolve the read and keep a baseline

Start inside the built security-execution/ directory of the Execution application, with its host serving .chio and repository. Resolve the read, then copy the original evidence for the comparisons in step 7.

review read-pricing-v1 src/pricing.py read-pricing.receipt.json \
  < .chio/connection.json > original-result.json
cp read-pricing.receipt.json before-restart.receipt.json
cp .chio/resource-reads.jsonl before-restart.reads.jsonl
cp .chio/kernel.pub before-restart.kernel.pub

You should see the guarded value in original-result.json and the signed receipt in read-pricing.receipt.json, with copies of the receipt, read journal, and kernel key.

2. Restart the host on its original state

Stop the serving host with Ctrl-C in its terminal, then start the replacement with the same state and repository paths.

security-execution-host .chio repository

Wait for "ready":true. Startup acquires the exclusive host lock, reopens the original keys and capability, obtains the authority store's serving fence, and reconciles durable admissions. It then replaces the connection credential for the retained process, so read the new .chio/connection.json before the next call.

Startup refuses a different repository binding, a competing owner, and an expired original grant. Diagnose those errors with the state directory and its authority intact, because recovery reads that history: the admitted request, the transitions that survived, and the authority to continue them.

3. Interrupt a read and keep its outcome unknown

Now stop the host after the connector has read the file and before it returns the value to the kernel. The host knows dispatch committed, and its reopened stores hold no response to return.

An operator-controlled pause-after-read marker holds that boundary: the connector syncs its resource-journal entry, then waits before returning, and only the operator sets the marker, never the worker through tool arguments. The exercise driver starts its own host, waits for the read observation, kills that host process, removes the marker, and reopens the original state.

python3 exercise.py \
  --host "$(command -v security-execution-host)" \
  --client "$(command -v review)" \
  --chio "$(command -v chio)" \
  --output /tmp/chio-recovery-run-1

Use a new output directory for each campaign. The driver keeps logs, original receipts, operation observations, and the resource journal under evidence/, beside private host state. You should see two separate operations:

The campaign records two resource reads, and the second belongs to the interrupted operation. Its denied recovery response sits beside a read the independent journal recorded, so keep the original unknown state; a new key is a new read. This connector is local and read-only. A mutating provider completes only on the original attempt's qualified status or executor evidence, as Unknown outcomes describes.

4. Choose the next action from what survived

Each durable transition the operation reached leaves one supported action to the current owner. In an incident, find the original operation and serving epoch, whether capture committed, and any authenticated executor outcome or exact release checkpoint.

• A reservation that never captured. Reconcile that same hold through its supported reversal and keep its history; releasing it takes the hold's owner, never an observation alone.
• A dispatch commitment with no proved outcome. Keep the capture and reconcile the original provider attempt or authenticated caller report; a lost acknowledgement permits no second effect.
• A raw result and evaluation with release unfinished. Continue the original post-return checks and release path; stored raw bytes become caller output only through that release.
• An exact guarded release checkpoint. Return that response through supported replay, without invoking the tool, whether or not the first caller received it.
• A missing operation, an unavailable store, or too little evidence. Keep the evidence and restore the right store and owner before deciding the outcome.

On a caller-owned path, submit the original signed executor report to authenticated reconciliation. On a native path, the qualified host owns dispatch recovery. Process-host recovery workflows answer to chio recovery inspect, explain, and resume, and resume rechecks current facts before dispatch.

Inspect reads the state the host recorded for an operation and the one action it permits. For the interrupted read, it reports a committed dispatch with no retained response:

chio recovery inspect --state .chio --capability owner.cap.json --workflow-id read-crash-v1

workflow   read-crash-v1
state      CommitUnknown
dispatch   committed in serving epoch 4; no response retained
evidence   resource journal records 1 read
preview    no dispatch; resume refuses until the original outcome is resolved
offers     reconcile with authenticated executor evidence

The preview is the guarantee in one line. A workflow in CommitUnknown never dispatches again; chio recovery resume proceeds only after reconciliation moves it to a resolved state, and the kernel rechecks current facts before it does.

5. Run recovery when your own host starts

An embedding host calls reconcile_durable_admission_startup() once per serving runtime, after it opens the original durable stores and before it accepts work, and inspects the original operation before and after the sweep. The serving owner then schedules reconcile_recoverable_admissions().

use chio_kernel::{ChioKernel, KernelError};
use chio_kernel::admission_operation::{AdmissionOperationId, AdmissionOperationStore,
    AdmissionOperationState, AdmissionDispatchState, AdmissionOperationStoreError};

pub type OriginalState = Option<(AdmissionOperationState, AdmissionDispatchState, bool)>;
fn inspect(store: &dyn AdmissionOperationStore, id: &AdmissionOperationId)
    -> Result<OriginalState, AdmissionOperationStoreError> {
    Ok(store.load_by_operation_id(id)?.map(|operation| (
        operation.state(), operation.dispatch_state(), operation.dispatch_commit().is_some(),
    )))
}

/// The serving host has opened the original durable runtime under its new fence.
/// Startup reconciliation runs once per serving runtime, not once per request.
pub fn recover_original_operation(
    kernel: &ChioKernel, store: &dyn AdmissionOperationStore,
    original_id: &AdmissionOperationId,
) -> (
    Result<OriginalState, AdmissionOperationStoreError>,
    Result<usize, KernelError>,
    Result<OriginalState, AdmissionOperationStoreError>,
) {
    let before = inspect(store, original_id);
    let reconciled = kernel.reconcile_durable_admission_startup();
    let after = inspect(store, original_id);
    (before, reconciled, after)
    // The count covers the sweep, not just this ID. Zero does not prove completion.
    // An error may follow partial progress: still inspect the original state.
}

/// Scheduled by the serving owner after startup; never dispatch a new request
/// merely to learn whether the original provider attempt completed.
pub fn continue_recovery(kernel: &ChioKernel) -> Result<usize, KernelError> {
    kernel.reconcile_recoverable_admissions()
}

The returned count covers the whole sweep, supporting recovery work included, and a missing runtime or a repeated sweep after the one-time flag returns zero. An error can follow partial progress. Confirm the original operation and its release evidence instead of the count; learning whether the original attempt completed never takes a new request, and a lookup supplies no live mutation authority. Keep recovery workers fenced, and repair a composition that lacks its store or release authority before it serves new operations.

6. Rehearse every crash boundary on your host

The restart suite runs an exact inventory of process-recovery and live-ownership cases against the installed binaries, on the host that will serve the pricing project. It ships with the release, and its inventory count and digest are checked before any case executes.

chio security selftest restart --evidence ./evidence

suite            result
restart          pass   every required case passed once; inventory complete
1 suite passed; evidence in ./evidence

chio security selftest restart runs the native restart-safety inventory that ships with this release. A missing, ignored, renamed, or failed case fails the suite, which a filtered test run cannot stand in for. The command exits 0 only when each required case passed exactly once, and ./evidence holds the suite log, its exit status, and a SHA-256 manifest of both.

For each case, the log pairs the test's effect observation, the retained operation, and what recovery returned. A conservative unknown is right even where the test knows its connector never started: the test owns that connector, and a production provider offers a different observation contract.

The report describes test inventory; whether a deployment uses the tested store, kernel, adapter, and configuration is a separate check. Start a failure from its earliest failing assertion and cutpoint, and keep the full log unedited.

7. Verify the replayed response, its receipt, and the read count

Resolve the same read through the restarted host from step 2, compare it with the baseline from step 1, and verify the receipt under the key you kept before the restart.

review read-pricing-v1 src/pricing.py read-pricing.receipt.json \
  < .chio/connection.json > replay-result.json
cmp original-result.json replay-result.json
cmp before-restart.receipt.json read-pricing.receipt.json
cmp before-restart.reads.jsonl .chio/resource-reads.jsonl
chio receipt verify --input read-pricing.receipt.json \
  --trusted-kernel-pubkey before-restart.kernel.pub
jq '.metadata.admission_operation | {
  projected_state, projected_dispatch_state, projected_operation_version,
  retained_dispatch_commit
}' read-pricing.receipt.json

You should see all three comparisons exit 0 and print nothing: the same application result, the same signed receipt, and no new file read on this idle host. The worker checks receipt equality itself too. In the captured exercise, the source file changed from returning 42 to returning 43 while the host was stopped, and read-pricing-v1 still returned the original 34-byte file containing 42, with the same receipt and one journaled read. The key names the original work; choose a new key only for a fresh read.

The receipt verifies under the pre-restart key, and its admission projection reads projected_state: completed and projected_dispatch_state: terminal, with dispatch committed at version 6 and the completed projection at version 8. Those versions belong to this run. Keep the full signed receipt, which binds the original operation, serving fence, and provider attempt; the projection verifies nothing on its own.

Restart inventory cases

Each case in the restart suite names one crash cutpoint or ownership race and the invariant recovery keeps there.

Next: correlate the recovered operation with its evidence