Caller delivery

The executor performs the effect; the kernel decides whether it may

The kernel and the executor each keep a record of progress. The kernel records which operation it authorized; the executor records which commitment it claimed and what effect it observed.

A reservation alone cannot start the tool, and a signed start alone shows nothing about completion. A lost message between the two owners leaves each to retain the fact it holds, and the executor's authenticated report is what lets the kernel reconcile the original operation.

An executor therefore needs a pinned identity, a durable claim and replay policy, and a way to return its original observation after an interruption. The application decides how the protected resource prevents a duplicate change if the executor fails in the middle of its own mutation.

Each owner keeps its own keys and pins the other before work starts

The execution lab includes a review-note command that appends the reviewer's note on a pull request to the pricing project's review journal, implementing the complete Rust protocol in caller.rs. Each command is a new process that reopens the original stores. These are trusted operator commands under one demonstration Unix account; the operator keeps their private state and signing keys. The example covers the kernel and executor integration: Execution supplies the authenticated-worker path, and the HTTP transport below covers a configured sidecar.

kernel.set_durable_admission_store(
    Arc::new(authority.admission_operation_store()),
    Arc::new(authority.tool_outcome_store()),
    authority.mutation_fence(),
)?;
kernel.configure_durable_admission(DurableAdmissionMode::All, false)?;
let nonce = ExecutionNonceConfig {
    nonce_ttl_secs: 120,
    nonce_store_capacity: 16,
    require_nonce: true,
};
kernel.set_execution_nonce_store(
    nonce.clone(),
    Box::new(InMemoryExecutionNonceStore::from_config(&nonce)),
);
// Configure the original executor before startup recovery reads its profile.
kernel.set_caller_executor(read(&state.join("executor.json"))?)?;

The host installs persistent revocation, budget, and receipt stores, connects durable admission to the serving authority, and pins the executor before startup recovery reads any original operation. Strict nonce configuration sets the execution interval, and the durable admission store retains the operation-owned start and nonce state; the executor's replay ledger is a separate SQLite file, provisioned once at initialization and only opened afterward. An open failure never provisions a replacement, and the constructor refuses missing original keys or databases.

review-note init

Complete the example setup first and run the command once in security-execution/; it provisions both owners before any short-lived authorization is issued, and their state stays in .caller/. The root grant permits four review-notes/append invocations for one hour. The executor retains at most four claims, accepts notes of at most 4,096 UTF-8 bytes, and syncs each journal record before returning its observation.

Only a committed start lets the executor run

authorize retains the complete request, then asks the kernel to reserve it. The reservation returns the original nonce and a signed receipt, and the sidecar reports execution_authorized: false; the committed start is a separate response.

let reserved = kernel.reserve_caller_execution_blocking(&request)?;
retain(&work.join("reservation.receipt.json"), &reserved.receipt)?;
ensure!(
    reserved.verdict == Verdict::Allow,
    "reservation refused: {:?}",
    reserved.reason
);
let nonce = reserved
    .execution_nonce
    .context("reservation supplied no nonce")?;
retain(&work.join("nonce.json"), &nonce)?;
let authorization = match kernel.start_caller_execution_blocking(&nonce, &request.arguments)? {
    CallerStartResponse::Authorized(value) => *value,
    CallerStartResponse::Denied(value) => {
        retain(&work.join("start-refusal.receipt.json"), &value.receipt)?;
        anyhow::bail!("start refused: {:?}", value.reason)
    }
};

Only the Authorized branch yields a signed dispatch authorization. A denied start preserves its refusal receipt and exits. The host records the expected invocation separately for the executor, so an incoming authorization cannot choose its own expected request or trusted signer.

printf 'Reviewed PR #421: the fixture for #414 carries an injected comment; do not merge.\n' > review.txt
review-note authorize review.txt --operation note-1

Authorized: note-1

You should see Authorized: note-1. JSON mode adds dispatch_committed and the durable operation ID. The authorization binds the original request, arguments, capability, executor key epoch, and captured dispatch, so repeating authorize with changed text under the same key is refused. A fresh start opens a 120-second execution window, bounded by the grant's lifetime: run the executor promptly. Replaying a retained report reproduces its observation and never extends permission to begin a new effect.

The executor claims before it appends, and the kernel reconciles its report

The executor opens its original ledger and checks the authorization against its own kernel pin, executor identity, and expected invocation, including the note arguments. The application callback runs only after a durable claim.

// Open is deliberately separate from provision. Failure never creates history.
let ledger = SqliteCallerExecutionLedger::open(&state.join("ledger.db"), identity)?;
let report = ledger.execute_once(&authorization, &trusted, &expected, &signer, || {
    append_note(root, expected.operation_id.as_str(), note)
        .map_err(|error| KernelError::Internal(error.to_string()))
})?;

execute_once commits the claim before calling append_note, holds no database transaction across the resource operation, and retains the signed report in a later transaction. A retained duplicate returns the same report without entering the callback; a claim with no retained report stays unknown and is never appended again.

review-note execute note-1

Signed report retained: note-1

You should see the report retained, requests/note-1/report.json written with the original claim ID printed, and the PR #421 note in the journal. The signed report is the executor's attestation of its result, separate from any provider attestation. The host then reopens with the same pinned executor, runs startup recovery, and authenticates the report:

review-note report note-1

Completed: original report reconciled.

Reviewed PR #421: the fixture for #414 carries an injected comment; do not merge.

The command displays a guarded value only after an allowed verdict and a completed terminal state, and it preserves the original receipt bytes on every replay. A missing value or a reconciliation error leaves the operation available for diagnosis without invoking the executor again.

let response =
    kernel.reconcile_authenticated_caller_execution_blocking(&authorization, &report)?;
retain(&work.join("completed.receipt.json"), &response.receipt)?;
ensure!(
    response.verdict == Verdict::Allow
        && response.terminal_state == OperationTerminalState::Completed,
    "original report did not complete: {:?}",
    response.reason
);
let Some(ToolCallOutput::Value(value)) = response.output else {
    anyhow::bail!("completed response has no guarded value; retain original operation")
};
retain(&work.join("result.json"), &value)?;

chio receipt verify --input .caller/requests/note-1/completed.receipt.json \
  --trusted-kernel-pubkey .caller/host/kernel.pub
jq . .caller/requests/note-1/result.json

Verified 1 receipt signatures, signer pins and action hashes.
{
  "note": "Reviewed PR #421: the fixture for #414 carries an injected comment; do not merge.\n",
  "operation_id": "72aae3cbb00954fc23c6e270a3622e8232fc0402c75c812d5a42ea35f8210021",
  "sha256": "1237fdc278c34d32ab1e023f63d4b5e97922b13d18328e7f13783bb4d337e75f"
}

A lost acknowledgement replays the same report without appending twice

A second operation exercises the boundary where the note is appended and the signed report is durable, but the process exits before delivering its acknowledgement. The deliberate cutpoint exits 75 with the journal entry written and requests/note-2/report.json undelivered.

review-note authorize review.txt --operation note-2
review-note execute note-2 --drop-reply
review-note execute note-2
review-note report note-2

Signed report retained: note-2
Completed: original report reconciled.

Reviewed PR #421: the fixture for #414 carries an injected comment; do not merge.

You should see the original report retained and reconciled. The captured exercise recorded two notes in total, one per operation: reopening the executor and reporting again added none, both operations returned byte-identical signed reports after reopen, and repeated reconciliation returned byte-identical kernel receipts. A claim with no report is preserved as unknown while the original effect is investigated; a report the kernel accepted before its response was lost is reconciled again under the same authorization; an unavailable ledger or a changed identity is restored and diagnosed, never replaced by a new ledger.

When the host restarts after a committed start but before any report, the pinned-executor profile keeps the operation in AwaitingCallerReport, and the original report can still complete it. A native call whose completion is missing is terminally unknown instead. Neither path treats a past receipt as fresh execution authority.

cp .caller/requests/note-2/report.json original-report.json
cp .caller/requests/note-2/completed.receipt.json original-receipt.json
review-note execute note-2
review-note report note-2
cmp original-report.json .caller/requests/note-2/report.json
cmp original-receipt.json .caller/requests/note-2/completed.receipt.json
wc -l .caller/resource/review-notes.jsonl

python3 exercise_caller.py --binary target/debug/caller \
  --chio "$(command -v chio)" --output /tmp/chio-caller-check

use chio_core::{Keypair, sha256_hex, canonical_json_bytes};
use chio_kernel::{CallerExecutionReport, KernelError};
use chio_kernel::caller_delivery::{CallerDispatchAuthorizationBodyV1,
    SignedCallerDispatchAuthorizationV1, CALLER_DISPATCH_AUTHORIZATION_SCHEMA};
use chio_store_sqlite::caller_execution_ledger::{SqliteCallerExecutionLedger,
    CallerExecutionLedgerError};
use std::path::Path;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{SystemTime, UNIX_EPOCH};

/// Run only in a fresh private disposable directory. The authorizer below is
/// an explicit signed fixture, not a kernel start or provider acceptance capture.
pub fn compare_executor_ledger(directory: &Path) -> Result<(), Box<dyn std::error::Error>> {
    let kernel = Keypair::generate(); let executor = Keypair::generate();
    let now = u64::try_from(SystemTime::now().duration_since(UNIX_EPOCH)?.as_millis())?;
    let operation = sha256_hex(b"reviewed-publication-fixture");
    let body: CallerDispatchAuthorizationBodyV1 = serde_json::from_value(serde_json::json!({
        "schema": CALLER_DISPATCH_AUTHORIZATION_SCHEMA,
        "kernel_public_key": kernel.public_key(),
        "executor": {"executor_id":"publication-executor","public_key":executor.public_key(),"key_epoch":1},
        "invocation": {"operation_id":operation,"request_id":"publication-1",
            "request_binding_hash":sha256_hex(b"request"),"capability_id":"publication-grant",
            "capability_digest":sha256_hex(b"grant"),"server_id":"publication","tool_name":"publish",
            "parameters_digest":sha256_hex(b"summary-A")},
        "committed": {"execution_nonce_id":"nonce-1","budget_hold_id":"hold-1",
            "frozen_context_digest":sha256_hex(b"context"),"dispatch_commit":{
                "committed_version":9,"coordinator_lease_id":"coordinator","coordinator_lease_epoch":1,
                "store_fence":{"store_uuid":"authority","lease_id":"owner","owner_epoch":1},
                "provider_attempt":{"operation_id":operation,"attempt_id":"attempt-1",
                    "transport_id":"caller-report:publication","transport_key_epoch":1}}},
        "not_before_unix_ms":now-1_000,"expires_at_unix_ms":now+60_000,
    }))?;
    let authorization = SignedCallerDispatchAuthorizationV1::sign(body, &kernel)?;
    let expected = authorization.authorization.invocation.clone();
    let identity = authorization.authorization.executor.clone();
    let path = directory.join("completed-executor.sqlite");
    let ledger = SqliteCallerExecutionLedger::provision(&path, identity.clone(), 2)?;
    let effects = AtomicUsize::new(0);
    let first = ledger.execute_once(&authorization, &kernel.public_key(), &expected, &executor, || {
        effects.fetch_add(1, Ordering::SeqCst);
        Ok(CallerExecutionReport {output:serde_json::json!({"fixture_summary":"A"}),realized_cost:None})
    })?;
    drop(ledger);
    let ledger = SqliteCallerExecutionLedger::open(&path, identity.clone())?;
    let repeated = ledger.execute_once(&authorization, &kernel.public_key(), &expected, &executor, || {
        effects.fetch_add(1, Ordering::SeqCst);
        Err(KernelError::Internal("a repeated callback must not execute".into()))
    })?;
    assert_eq!(canonical_json_bytes(&first)?, canonical_json_bytes(&repeated)?);
    assert_eq!(effects.load(Ordering::SeqCst), 1);
    println!("REPLAY original signed report after reopen: one callback; identical report bytes");
    let mut wrong_epoch = identity.clone(); wrong_epoch.key_epoch += 1;
    assert!(SqliteCallerExecutionLedger::open(&path, wrong_epoch).is_err());
    assert!(SqliteCallerExecutionLedger::provision(&path, identity.clone(), 2).is_err());
    println!("REFUSE changed executor epoch and replacement ledger: original history retained");
    drop(ledger);

    let unknown_path = directory.join("unknown-executor.sqlite");
    let ledger = SqliteCallerExecutionLedger::provision(&unknown_path, identity.clone(), 2)?;
    let unknown_effects = AtomicUsize::new(0);
    assert!(ledger.execute_once(&authorization, &kernel.public_key(), &expected, &executor, || {
        unknown_effects.fetch_add(1, Ordering::SeqCst);
        Err(KernelError::Internal("lost result after claim".into()))
    }).is_err());
    drop(ledger);
    let ledger = SqliteCallerExecutionLedger::open(&unknown_path, identity)?;
    let repeated = ledger.execute_once(&authorization, &kernel.public_key(), &expected, &executor, || {
        unknown_effects.fetch_add(1, Ordering::SeqCst);
        Ok(CallerExecutionReport {output:serde_json::json!({"unsafe_retry":true}),realized_cost:None})
    });
    assert!(matches!(repeated, Err(CallerExecutionLedgerError::OutcomeUnknown)));
    assert_eq!(unknown_effects.load(Ordering::SeqCst), 1);
    println!("UNKNOWN claimed result after reopen: original custody; no second callback");
    Ok(())
}

REPLAY original signed report after reopen: one callback; identical report bytes
REFUSE changed executor epoch and replacement ledger: original history retained
UNKNOWN claimed result after reopen: original custody; no second callback

Over HTTP, only a client holding the control token can start or report

An embedding that exposes the HTTP protocol pins its executor with ProtectProxy::with_caller_executor before admitting reservations, and requires durable admission and budget authority. A sidecar started by the stock chio api protect or chio start command pins no executor, so its start path refuses this integration.

A router-level middleware puts /v1/caller/start and /v1/caller/report behind the sidecar control gate and rejects any request without the configured bearer token before either handler runs. /v1/evaluate sits outside the gate, so an ordinary agent can reserve an operation while only a trusted control client can commit or reconcile one. A successful start returns dispatch_committed with a statement addressed to one CallerExecutorIdentityV1, which no other executor can claim; a denied start returns HTTP 403 with status deny and its receipt. The executor's process embeds SqliteCallerExecutionLedger, which authenticates and claims the authorization in its own transaction, runs the tool with no lock held, and persists the signed report that travels back over /v1/caller/report to return reconciled, or deny when finalization denies the result.

# Async fragment: existing ChioClient, complete signed capability,
# immutable arguments, stable request_id and operator-owned control_token.
reserved = await client.evaluate_tool_call_mediated(
    capability=capability,
    tool_server="workspace",
    tool_name="read_file",
    parameters=arguments,
    request_id=request_id,
)
if reserved["status"] != "reserved":
    raise RuntimeError(f"Reservation stopped: {reserved['status']}")

started = await client.start_mediated_execution(
    control_token=control_token,
    execution_nonce=reserved["execution_nonce"],
    arguments=arguments,
)
authorization = started["authorization"]

From Python, the lower-level kernel client ChioClient drives the same exchange; the workspace server's read_file is a tool the grant already covers. start_mediated_execution takes the control token as an explicit argument, sends it as the bearer credential, and raises ChioValidationError unless the response carries protocol chio.caller-delivery.v1, status dispatch_committed, and an authorization object. The start call neither claims the authorization nor runs the tool: hand authorization to the host-pinned executor, which checks its independent pins and claims it through its durable ledger, wait for its retained signed report, then submit both:

# After receiving the pinned executor's retained signed report:
finalized = await client.report_mediated_execution(
    control_token=control_token,
    authorization=authorization,
    report=report,
)
# Inspect finalized["status"] and retain its receipt.

A valid report returns reconciled or a finalization deny, and execution_authorized stays false throughout. Send a lost start or report reply to original-operation recovery with its authorization and report; an automatic retry loop or a fresh request would settle nothing.

pub struct SignedCallerDispatchAuthorizationV1 {
    pub authorization: CallerDispatchAuthorizationBodyV1,
    pub signature: Signature,
}

pub struct SignedCallerDeliveryReportV1 {
    pub report: CallerDeliveryReportBodyV1,
    pub signature: Signature,
}

Read more in Control surfaces, which keeps the control credential apart from every other role at the sidecar and the remote MCP edge.

Refusals arrive as fixed HTTP bodies or as typed ledger errors

Over HTTP, the control gate and request parsing refuse first, each with its own body. Past them, both routes match their kernel call as Err(_) => rejected(), so every kernel-call failure returns the same fixed 409.

A failed kernel call includes an authorization or report that fails to verify, converted from CallerDeliveryError to a plain string inside a KernelError before the handler sees it, and every start on a sidecar with no pinned executor. A Rust executor calling SqliteCallerExecutionLedger::execute_once skips the wire encoding and matches the typed error. Its first check authenticates the authorization against the executor's own pins; report checks run when a redelivery replays a persisted report, or when the ledger verifies the report it has signed.

The kernel's report path checks only that the report's recorded start fell inside the execution interval. Check the control gate and the trusted executor first. For a 409, compare the original request, the committed output, and the signed executor report against the retained delivery state: the generic response names no internal cause and authorizes no new operation ID. Use original-operation recovery for a persisted conflict or an uncertain delivery.

Next: recover an interrupted operation