Calls & Recovery

The request is saved before it is sent

Derive the key from the application task, never from a timestamp, a worker attempt, or a UUID generated on restart. Persist the whole request beside it: a checkpoint that says only “publishing” cannot say which report or which recovery policy went out.

def prepare_publication(client, report):
    cp = client.inspect()["checkpoint"]
    value = cp["value"] or {}
    request = {
        "key": "publish-report-42",
        "server": "reports",
        "tool": "publish",
        "arguments": {"report": report},
        "known_outcome_only": True,
    }
    saved = value.get("publication.v1")
    if saved is not None:
        if saved["request"] != request:
            raise ValueError(
                "Recovery input differs from the original publication"
            )
        return saved["request"]
    client.checkpoint(
        cp["revision"],
        {
            **value,
            "publication.v1": {
                "version": 1,
                "phase": "prepared",
                "request": request,
            },
        },
    )
    return request


# Pass the returned request to resolve_original below.
# Then validate the reports/publish output contract before marking it complete.

The helper prepares one publication slot and raises if re-entry brings a different task. A conflicting checkpoint stays as it is; the second writer rereads and decides which intent the application owns.

A repeated key returns the original outcome

After a lost response, the worker calls again with the saved request. This function takes an authenticated ProcessClient and that request; its caller owns reconnection and scheduling, and the function starts no second invocation after an exception.

from chio_process import WorkerError


def resolve_original(client, request):
    try:
        response = client.invoke(
            request["key"],
            request["server"],
            request["tool"],
            request["arguments"],
            known_outcome_only=request["known_outcome_only"],
        )
    except WorkerError as error:
        if error.code in {
            "transport_error",
            "truncated_response",
            "runtime_error",
        }:
            # Preserve request. Stop this attempt; reconcile/reconnect separately.
            raise RuntimeError(
                "Original operation requires recovery"
            ) from error
        # conflict, invalid_request, unauthenticated, limit_reached, etc.
        # need diagnosis; blindly retrying cannot repair their cause.
        raise
    if response["verdict"] != "allow":
        raise RuntimeError(
            f"Decision: {response['verdict']}: {response['reason']}"
        )
    if response["terminal_state"]["state"] != "completed":
        raise RuntimeError("Original outcome is not complete")
    if response["output"]["kind"] != "value":
        raise RuntimeError("Expected a materialized value")
    return response

A transport error can arrive after a publication went through. When the kernel retained the outcome, recovery returns the original result and receipt. When the effect may have happened and no outcome was retained, recovery returns an unknown outcome and the kernel refuses to dispatch that publication again. A fresh key creates a new intent and leaves the original unresolved.

• Process ID and operation key. Stable across recovery; they name the application's original logical operation.
• Frozen route, arguments, and recovery policy. Stable; they expose a changed intent behind a retained key.
• Kernel request ID. The same on a known replay, attempt-derived on an eligible read redispatch; it names the kernel attempt the receipt records.
• Original receipt_json. The same text on a known replay; verify it as returned, without reserializing it.
• OS attempt number. Changes on every launch; it names diagnostics and is never a tool idempotency key.

The observation decides whether the call can resume

Each result layer answers a different question. Read them in order before the worker acts.

A call the kernel never admitted is fixed at its cause: correct the refused request or authority, then make a new intent. A known completed outcome is replayed by its original key and checked against its original receipt. An unknown effectful outcome keeps its identity until the effect owner or the operator reconciles it; Recover an operation reopens the original host state and preserves an interrupted operation without repeating its effect.

A crash lands between two durable writes

Three writes order every call: the prepared request, the kernel's retained outcome, and the application's checkpoint. Where the crash falls decides what recovery can say.

Before dispatch, the prepared request can be submitted. After the resource commits but before the kernel records its outcome, recovery can be uncertain. After the outcome is retained but before the checkpoint, the worker replays that outcome and advances its own progress. A resource's idempotency can help reconcile the middle window, and the kernel's missing receipt stays missing.

Test that middle window at the effect owner. Terminate the host after the external service commits and before its answer is recorded, observe the service directly, reopen the same host state, and repeat the exact logical call. Assert that a known result replays or an unknown result refuses another delivery; a test that checks only the worker's exception misses duplicate effects. Durability describes the kernel's dispatch boundary, and Unknown outcomes the evidence retained around it.

Only a side-effect-free read is dispatched again

An unknown effectful outcome is never re-dispatched. A registered tool declared free of side effects is the one exception, and the kernel decides it from the selected grant and the request.

# A catalog observation can tolerate a later eligible read attempt.
# The kernel still decides eligibility from the actual tool and grant.
observation = client.invoke(
    "catalog-observation-42", "catalog", "read", {"item": "42"}
)

# A model decision must not be regenerated after an unknown outcome.
decision = client.invoke(
    "model-decision-42",
    "models",
    "complete",
    {
        "prompt": "Choose the report sections for task 42",
    },
    known_outcome_only=True,
)
# Persist these exact requests before either call. Resolve each original key
# on recovery; changing the policy behind a retained key is a conflict.

The catalog read can be dispatched again after an unknown outcome when its route and authority qualify. Calls under request-bound authorization, constrained grants, budgets, or other stateful authority are excluded, and a host cannot make an effectful tool eligible by setting a retry flag. An eligible attempt keeps the logical key and frozen request, gets an attempt-derived request ID, and counts against a bounded attempt ceiling. A saved guarded response is released exactly as retained.

known_outcome_only makes the stricter choice part of the original binding. It allows the first dispatch and the replay of a completed response, and refuses a fresh dispatch of an unknown outcome even for a declared read-only tool. Save it with the key, because changing it later is a conflict; retries inside a model provider or a tool are outside its reach. Python sets it with known_outcome_only=True, and the Rust client calls invoke_known_only.

The invocation helper checks a response against your context

For operator calls, the installed helper freezes the runtime, process, and capability identities, writes the request before dispatch, saves the original receipt text, and verifies the response binding. Run it against the manual host from the report project, with the connection descriptor and kernel public key chosen through the trusted initialization path.

{
  "operation_key": "send-report-42",
  "server_id": "chio-ipc",
  "tool_name": "send_jobs",
  "arguments": {
    "message_key": "report-42",
    "payload": {
      "items": [
        2,
        3,
        5
      ]
    }
  },
  "known_outcome_only": true
}

uv run python -m chio_process.invocation \
  --chio chio \
  --connection worker-connections/producer.json \
  --trusted-kernel-pubkey .chio-manual/kernel.pub \
  --request request.json \
  --output send-attempt-1

The output directory must be new. A successful check reports response_bound: true, binding the signed caller, operation, recovery policy, route, arguments, request ID, verdict, terminal state, and released output. A failed check preserves the evidence and never retries the operation.

The helper retains execution_nonce_json without checking it. Current resource ownership, application correctness, and complete receipt-log inclusion are facts to inspect at their owners.

To show a reviewer what became of the call, stop the manual host and attest the retained operation. The host signs its observation of the call and its continuation custody; the reviewer checks it against the request and context they kept, the kernel key, and the runtime ID, without opening host state.

chio process attest-call --state .chio-manual --request request.json \
  --context send-attempt-1/context.json --response send-attempt-1/response.json \
  --out send-report-42.call.json
chio process verify-call --artifact send-report-42.call.json \
  --request request.json --context send-attempt-1/context.json \
  --trusted-kernel-pubkey .chio-manual/kernel.pub --runtime-id 7a2d…e815

{
  "schema": "chio.process.call-verification.v1",
  "runtime_id": "7a2d…e815",
  "process_id": "producer",
  "request_id": "process:3b6f…c07a",
  "observed_operation_state": "completed",
  "custody_binding": "original_call_commitment_and_signed_store_readback",
  "artifact_schema": "chio.process.call-observation.v2",
  "checks": ["signer_pin", "runtime_pin", "original_request", "issued_capability", "worker_response", "retained_operation", "continuation_custody"],
  "unchecked": ["physical_effects", "aggregate_usage", "confinement"]
}

You should see observed_operation_state completed for send-report-42. An uncertain call reports its unknown state the same way, and a denied call with no retained operation reports no_custody_claim. Whether the effect happened in the world stays unchecked.

The same request holds in Python, TypeScript, and Rust

These recipes from the three-language recipe project run a self-send exercise whose input is frozen before launch. Each sends the same operation, checks every response layer, and resolves the original key after a transport failure, with no new key and no retry loop. They use the default recovery policy; when application input can change, persist the request first, as the prepared publication above does.

def send_report(client):
    # Freeze this key and payload before launching the worker.
    response = client.invoke(
        "report/send/v1",
        "chio-ipc",
        "send_jobs",
        {
            "message_key": "report-42",
            "payload": {"items": [2, 3, 5]},
        },
    )
    if completed(response)["status"] != "sent":
        raise RuntimeError("Mailbox did not accept the report")
    return response  # Keep receipt_json unchanged for independent verification.

export async function sendReport(client: ProcessClient): Promise<ToolResult> {
  // Freeze this key and payload before launching the worker.
  const response = await client.invoke(
    "report/send/v1",
    "chio-ipc",
    "send_jobs",
    {
      message_key: "report-42",
      payload: { items: [2, 3, 5] },
    },
  );
  assert.equal(completed(response).status, "sent");
  return response; // Keep receipt_json unchanged for independent verification.
}

async fn send_report(client: &ProcessClient) -> Result<ToolResult> {
    // Freeze this key and payload before launching the worker.
    let response = client
        .invoke(
            "report/send/v1",
            "chio-ipc",
            "send_jobs",
            json!({"message_key":"report-42","payload":{"items":[2,3,5]}}),
            None,
        )
        .await?;
    if completed(&response)?["status"] != "sent" {
        return Err("Mailbox did not accept report".into());
    }
    Ok(response) // Keep receipt_json unchanged for independent verification.
}

def completed(response):
    if response["verdict"] != "allow":
        raise RuntimeError("Decision requires attention: " + response["verdict"])
    if response["terminal_state"]["state"] != "completed":
        raise RuntimeError("Original operation has no complete outcome")
    if not response["output"] or response["output"]["kind"] != "value":
        raise RuntimeError("Expected a materialized tool response")
    value = response["output"]["value"]
    if value.get("isError"):
        raise RuntimeError("Tool returned a completed application error")
    return value

export function completed(response: ToolResult): Record<string, Json> {
  if (response.verdict !== "allow")
    throw new Error(`Decision requires attention: ${response.verdict}`);
  if (response.terminal_state.state !== "completed")
    throw new Error("Original operation has no complete outcome");
  if (response.output?.kind !== "value")
    throw new Error("Expected a materialized tool response");
  const value = object(response.output.value);
  if (value.isError)
    throw new Error("Tool returned a completed application error");
  return value;
}

fn completed(response: &ToolResult) -> Result<&Value> {
    // Checks allow + completed + value; denial and uncertainty cannot be success.
    let value = response
        .completed_value()
        .ok_or("Decision or terminal state requires attention")?;
    if value.get("isError").and_then(Value::as_bool) == Some(true) {
        return Err("Tool returned a completed application error".into());
    }
    Ok(value)
}

def recover_send(client):
    try:
        return send_report(client)  # Exact original key, arguments and intent.
    except WorkerError as error:
        if error.code in {"transport_error", "truncated_response", "runtime_error"}:
            raise RuntimeError(
                "Retain original request; resume after diagnosis"
            ) from error
        raise  # Authentication, CAS conflicts and limits need their own action.

export async function recoverSend(client: ProcessClient): Promise<ToolResult> {
  try {
    return await sendReport(client);
  } catch (error) {
    // Exact original key, arguments and intent.
    if (
      error instanceof WorkerError &&
      ["transport_error", "truncated_response", "runtime_error"].includes(
        error.code,
      )
    ) {
      throw new Error("Retain original request; resume after diagnosis", {
        cause: error,
      });
    }
    throw error; // Authentication, CAS conflicts and limits need their own action.
  }
}

async fn recover_send(client: &ProcessClient) -> Result<ToolResult> {
    match send_report(client).await {
        Ok(response) => Ok(response), // Same key, arguments and intent as before.
        Err(error) => {
            if matches!(
                error.downcast_ref::<Error>(),
                Some(Error::Transport | Error::Truncated)
            ) {
                eprintln!("Retain original request; resume after diagnosis");
            }
            Err(error) // No automatic rekeying or unbounded retry loop.
        }
    }
}

The policy takes effect only when the client sends the field on the wire. When an unknown outcome must never be redispatched, use a client call that sends known_outcome_only explicitly.

Next: keep the worker's own progress