Model-call recovery

An AI SDK worker built on ChioProcessAgent survives a restart without asking the model to decide twice. The adapter saves each complete provider response before any of its tool calls execute, and the kernel admits each of those calls as a native operation of a Chio process.

A receipt records what happened to one tool operation; the response journal records the model decision that chose it. A provider asked again can return a new tool-call ID, which is a second operation. On restart the worker reuses the saved response and its tool-call IDs, and each call resolves its original kernel operation with its outcome and receipt.

Resuming a turn takes both records: the journal holds the plan the model chose, and the kernel holds whether each tool in that plan ran.

1. Download the report worker and build it

In your own project the adapter is npm install @chio-protocol/ai-sdk-process @chio-protocol/process, beside ai and your provider package. To follow the steps here, download ai-sdk.zip, extract it, and work in ai-sdk/ on Linux with Chio, Python 3.11+, and Node 22+.

npm ci
npm run build

npm run build compiles ai-worker.ts and the separate ai-exercise.ts entry point; rebuild after changing either. The JavaScript files run the same application without a TypeScript build. The project holds the worker, one publication tool, and an explicit local test provider, and the framework README lists the exact dependency and build commands.

2. Run generation inside the adapter

The worker separates bootstrap validation, agent construction, and generation. The generation function passes the adapter's bindings to AI SDK inside agent.run.

async function generateReport(agent: ChioProcessAgent, input: WorkerInput) {
  return agent.run(async (bindings) => {
    const options = {
      ...bindings,
      prompt: input.prompt,
      maxRetries: 0,
      stopWhen: stepCountIs(3),
    };
    if (!input.streaming) {
      return (await generateText(options)).text;
    }

    const response = streamText(options);
    for await (const chunk of response.textStream) {
      // Displayed text is provisional until agent.run resolves.
      process.stdout.write(chunk);
    }
    return await response.text;
  });
}

Streaming text may reach the screen at once, but the run is complete only when agent.run resolves. The run plan fixes threadId, turnId, prompt, and modelKey for every attempt, so a replacement worker addresses the same turn.

The onReceipt callback saves each original signed receipt string. Recovery can append the same receipt again; an index can deduplicate verified operation identities while keeping the original text.

import { readFileSync, appendFileSync } from "node:fs";
import { pathToFileURL } from "node:url";
import { generateText, streamText, stepCountIs } from "ai";
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { ProcessClient } from "@chio-protocol/process";
import {
  ChioProcessAgent,
  ProcessSuspendedError,
  type ProcessToolDefinition,
} from "@chio-protocol/ai-sdk-process";

interface WorkerInput {
  threadId: string;
  turnId: string;
  prompt: string;
  endpoint: string;
  model: string;
  modelKey: string;
  receipts: string;
  credentialEnv?: string;
  streaming?: boolean;
}

export interface Bootstrap {
  schema: string;
  attempt: number;
  connection: {
    socket_path: string;
    credential: string;
    tools: ProcessToolDefinition[];
  };
  input: WorkerInput;
}

export function readBootstrap(): Bootstrap {
  const bootstrap = JSON.parse(readFileSync(0, "utf8")) as Bootstrap;
  if (bootstrap.schema !== "chio.process.worker-bootstrap.v1") {
    throw new Error("Unsupported bootstrap");
  }
  const required = [
    "threadId",
    "turnId",
    "prompt",
    "endpoint",
    "model",
    "modelKey",
    "receipts",
  ] as const;
  for (const field of required) {
    if (typeof bootstrap.input[field] !== "string" || !bootstrap.input[field]) {
      throw new Error(`Missing input: ${field}`);
    }
  }
  return bootstrap;
}

function createAgent(bootstrap: Bootstrap, onReceiptSaved: () => void) {
  const { connection, input } = bootstrap;
  const provider = createOpenAICompatible({
    name: "report-provider",
    baseURL: input.endpoint,
    apiKey: input.credentialEnv ? process.env[input.credentialEnv] : undefined,
  });
  return new ChioProcessAgent({
    client: new ProcessClient(connection.socket_path, connection.credential),
    model: provider.chatModel(input.model),
    tools: connection.tools,
    namespace: "report-v1",
    threadId: input.threadId,
    turnId: input.turnId,
    modelKey: input.modelKey,
    maxModelCalls: 8,
    maxConcurrency: 1,
    onReceipt: (event) => {
      appendFileSync(input.receipts, `${event.result.receipt_json}\n`, {
        mode: 0o600,
        flush: true,
      });
      onReceiptSaved();
    },
  });
}

// region ai_run
async function generateReport(agent: ChioProcessAgent, input: WorkerInput) {
  return agent.run(async (bindings) => {
    const options = {
      ...bindings,
      prompt: input.prompt,
      maxRetries: 0,
      stopWhen: stepCountIs(3),
    };
    if (!input.streaming) {
      return (await generateText(options)).text;
    }

    const response = streamText(options);
    for await (const chunk of response.textStream) {
      // Displayed text is provisional until agent.run resolves.
      process.stdout.write(chunk);
    }
    return await response.text;
  });
}

// endregion ai_run

export async function runWorker(
  bootstrap: Bootstrap,
  onReceiptSaved: () => void = () => {},
) {
  try {
    const agent = createAgent(bootstrap, onReceiptSaved);
    const text = await generateReport(agent, bootstrap.input);
    console.log(JSON.stringify({ complete: true, text }));
  } catch (error) {
    if (error instanceof ProcessSuspendedError) {
      process.exitCode = error.exitCode;
    } else {
      const code = error instanceof Error ? error.name : "unknown_error";
      console.error(JSON.stringify({ complete: false, code }));
      process.exitCode = 2;
    }
  }
}

if (
  process.argv[1] &&
  import.meta.url === pathToFileURL(process.argv[1]).href
) {
  await runWorker(readBootstrap());
}

import { readFileSync, appendFileSync } from "node:fs";
import { pathToFileURL } from "node:url";
import { generateText, streamText, stepCountIs } from "ai";
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { ProcessClient } from "@chio-protocol/process";
import {
  ChioProcessAgent,
  ProcessSuspendedError,
} from "@chio-protocol/ai-sdk-process";
export function readBootstrap() {
  const bootstrap = JSON.parse(readFileSync(0, "utf8"));
  if (bootstrap.schema !== "chio.process.worker-bootstrap.v1") {
    throw new Error("Unsupported bootstrap");
  }
  const required = [
    "threadId",
    "turnId",
    "prompt",
    "endpoint",
    "model",
    "modelKey",
    "receipts",
  ];
  for (const field of required) {
    if (typeof bootstrap.input[field] !== "string" || !bootstrap.input[field]) {
      throw new Error(`Missing input: ${field}`);
    }
  }
  return bootstrap;
}
function createAgent(bootstrap, onReceiptSaved) {
  const { connection, input } = bootstrap;
  const provider = createOpenAICompatible({
    name: "report-provider",
    baseURL: input.endpoint,
    apiKey: input.credentialEnv ? process.env[input.credentialEnv] : undefined,
  });
  return new ChioProcessAgent({
    client: new ProcessClient(connection.socket_path, connection.credential),
    model: provider.chatModel(input.model),
    tools: connection.tools,
    namespace: "report-v1",
    threadId: input.threadId,
    turnId: input.turnId,
    modelKey: input.modelKey,
    maxModelCalls: 8,
    maxConcurrency: 1,
    onReceipt: (event) => {
      appendFileSync(input.receipts, `${event.result.receipt_json}\n`, {
        mode: 0o600,
        flush: true,
      });
      onReceiptSaved();
    },
  });
}
async function generateReport(agent, input) {
  return agent.run(async (bindings) => {
    const options = {
      ...bindings,
      prompt: input.prompt,
      maxRetries: 0,
      stopWhen: stepCountIs(3),
    };
    if (!input.streaming) {
      return (await generateText(options)).text;
    }
    const response = streamText(options);
    for await (const chunk of response.textStream) {
      // Displayed text is provisional until agent.run resolves.
      process.stdout.write(chunk);
    }
    return await response.text;
  });
}
export async function runWorker(bootstrap, onReceiptSaved = () => {}) {
  try {
    const agent = createAgent(bootstrap, onReceiptSaved);
    const text = await generateReport(agent, bootstrap.input);
    console.log(JSON.stringify({ complete: true, text }));
  } catch (error) {
    if (error instanceof ProcessSuspendedError) {
      process.exitCode = error.exitCode;
    } else {
      const code = error instanceof Error ? error.name : "unknown_error";
      console.error(JSON.stringify({ complete: false, code }));
      process.exitCode = 2;
    }
  }
}
if (
  process.argv[1] &&
  import.meta.url === pathToFileURL(process.argv[1]).href
) {
  await runWorker(readBootstrap());
}

3. Run one turn against the local test provider

The default plan uses a scripted provider, so recovery needs no API key. Start it in a second terminal; it records every request in model.db.

python3 model_server.py --port 8768

Back in the first terminal, authorize the report tool, run the worker, and verify its receipts. --local-tools starts the configured local command for discovery and signs its launch policy, which authorizes trusted local code without OS confinement.

chio process init --config host.json --state .chio --local-tools
chio process run --state .chio --plan process.json
chio receipt verify --input .chio/receipts.ndjson \
  --trusted-kernel-pubkey .chio/kernel.pub

You should see the reporter complete in one attempt and one row in .chio/reports.db. Running the same plan again keeps the completed attempt.

4. Stop the worker after its first receipt and let it resume

Keep the test provider running. The recovery plan selects ai-exercise.ts, which runs the ordinary worker and exits after its first saved tool receipt. The replacement resumes the saved response and the original tool-call ID.

chio process init --config host-recovery.json --state .chio-recovery --local-tools
chio process run --state .chio-recovery --plan process-recovery.json
chio process status --state .chio-recovery
chio receipt verify --input .chio-recovery/receipts.ndjson \
  --trusted-kernel-pubkey .chio-recovery/kernel.pub

You should see two reporter attempts and one publication in .chio-recovery/reports.db. The saved receipt repeats with its original signed bytes and verifies against the kernel key.

5. Count provider requests against publications

The framework workspace runs complete text, complete streaming, and a truncated stream with independent effect counts. Run exercise.py from the framework download to reproduce this capture and verify the original receipt signatures.

[
  {
    "mode": "generate",
    "publications": 1,
    "provider_calls": 2,
    "complete": true
  },
  {
    "mode": "stream",
    "publications": 1,
    "provider_calls": 2,
    "complete": true
  },
  {
    "mode": "truncated-stream",
    "publications": 0,
    "provider_calls": 1,
    "complete": false
  }
]

You should see two provider responses for each successful run: one tool decision and one final answer. The deliberate restart adds no third provider request and no second publication. The truncated stream leaves one provider request and no tool effect.

Each moment of a turn leaves one durable record

A restart resumes from the last record written. The four moments of a turn, in order:

1. Before the provider request. The model-call ordinal and request hash are reserved in the process checkpoint. A pending call is unknown, and the adapter sends no automatic provider retry.
2. Complete provider response saved. Content, tool-call IDs, and metadata sit in the checkpoint or in immutable blobs. A restart replays the same response without contacting the provider.
3. Tool admitted and completed. The kernel holds the operation outcome and its signed receipt. A restart recovers the same operation under its original key.
4. Application sees the result. The application owns its output and any downstream acknowledgement, so it verifies output and reconciles UI delivery separately.

The auto storage mode keeps responses in immutable native blobs when the host advertises them, with small references in the 1 MiB checkpoint. A provider failure, a truncated stream, or a worker exit before the response checkpoint leaves a pending reservation, and the adapter stops rather than regenerate a different tool plan. Provider inference and billing stay with the provider.

Read more in Checkpoints and state.

Streamed text can arrive before the tool plan is durable

With streamText, text and reasoning flow to the UI as they arrive. The first tool event and the rest of that response wait for the final response to commit, and tool execution waits with them.

A later failure can stop a run after the UI has shown partial text, so the UI owns deduplication and reconciliation by its own stable turn identity. Consume the stream inside agent.run and await completion before reporting success; a stream returned from the callback would outlive its admission scope.

When a failure closes the run, the adapter drains work already admitted. With concurrent tool calls, one failed sibling leaves another sibling's committed effect in place. Model traffic, provider credentials, billing, and direct local callbacks sit outside the native tool guarantee.

ChioProcessAgent owns the response journal; ChioProcessTools uses yours

Both bridges mediate tools and retain native outcomes. They differ in who saves the model's response before the tools run.

Choose ChioProcessAgent unless your application already journals complete provider responses. AI SDK's onStepFinish runs after tool execution, too late to establish that journal.

A saved turn replays its tools without a provider

When your application retains the complete response, ChioProcessTools dispatches its saved calls directly. SavedTurn is the extracted tool plan: thread, turn, tool-call IDs, names, and arguments, all taken from that retained response.

import { ProcessClient, type Json } from "@chio-protocol/process";
import {
  ChioProcessTools,
  type ProcessToolDefinition,
  type ProcessReceiptEvent,
} from "@chio-protocol/ai-sdk-process";

interface Connection {
  socket_path: string;
  credential: string;
  tools: ProcessToolDefinition[];
}

// Application-owned journal record, saved before any of these tools execute.
export interface SavedTurn {
  threadId: string;
  id: string;
  toolCalls: Array<{ id: string; name: string; input: Json }>;
}

export async function replaySavedTurn(
  connection: Connection,
  savedTurn: SavedTurn,
  recordReceipt: (event: ProcessReceiptEvent) => void | Promise<void>,
) {
  const bridge = new ChioProcessTools({
    client: new ProcessClient(connection.socket_path, connection.credential),
    tools: connection.tools,
    namespace: "repository-review",
    threadId: savedTurn.threadId,
    turnId: savedTurn.id,
    onReceipt: recordReceipt,
  });

  return bridge.run(async ({ tools, abortSignal }) => {
    const results = [];
    for (const call of savedTurn.toolCalls) {
      const tool = tools[call.name];
      if (!tool?.execute) throw new Error(`Unknown saved tool: ${call.name}`);
      const result = await tool.execute(call.input, {
        toolCallId: call.id,
        messages: [],
        context: {},
        abortSignal,
      });
      results.push({ toolCallId: call.id, result });
    }
    return results;
  });
}

The caller loads savedTurn from its journal and supplies recordReceipt to save each original signed string. ChioProcessTools binds each native operation to namespace, thread, turn, and tool-call ID, and replaySavedTurn never calls a provider to recreate a call. Model decisions stay in your journal.

Read more in Calls and recovery.

Journal limits and failure behavior

Next: follow one operation key through recovery