Supervise an Application

chio process run launches your workers in dependency order, hands each one its connection, retries a failed attempt within a fixed budget, and picks the same plan back up after an interruption.

The plan names processes the host already created, so it chooses workers and adds no authority. Attempts are OS launches and operations are logical tool calls: attempt two of a worker recovers the receipt of a call that attempt one made. The commands continue the report walkthrough from its report/ directory, after the host is initialized.

1. Declare dependencies and limits

The consumer depends on the producer. Each worker gets two attempts and 30 seconds per attempt, and at most two run at once.

{
  "schema": "chio.process.run.v2",
  "max_parallel": 2,
  "workers": [
    {
      "process": "producer",
      "command": [
        ".venv/bin/python",
        "producer.py"
      ],
      "cwd": ".",
      "input": {
        "directory": ".chio",
        "items": [
          2,
          3,
          5
        ]
      },
      "depends_on": [],
      "max_attempts": 2,
      "timeout_seconds": 30
    },
    {
      "process": "consumer",
      "command": [
        "node",
        "consumer.mjs"
      ],
      "cwd": ".",
      "input": {
        "directory": ".chio",
        "items": [
          2,
          3,
          5
        ]
      },
      "depends_on": [
        "producer"
      ],
      "max_attempts": 2,
      "timeout_seconds": 30
    }
  ]
}

chio process run --state .chio --plan process.json

In a v2 plan, cwd is relative to the plan file and executable paths are relative to that directory. A bare name such as node is resolved from PATH once and retained; arguments stay literal and input stays application JSON. The runner sends each worker its connection and input on stdin, so credentials stay out of arguments, environment variables, and model messages.

Compiled workers follow the same plan, with the same dependencies, attempt ceilings, and completion checks. The Rust starter names its compiled worker, and TypeScript names an absolute Node path and compiled JavaScript; freeze those artifacts before the first run. Run the equivalent supervised applications in each language.

chio process status --state .chio
chio process logs --state .chio --process consumer --attempt 1

You should see both workers completed with one attempt each, and the consumer's first-attempt log.

2. Bound each worker's time and resources

Add a resources block to a worker entry. These ceilings are examples; size them to your runtime first, then to the useful work.

{
  "resources": {
    "max_cpu_seconds": 30,
    "max_open_files": 256,
    "max_file_bytes": 67108864,
    "max_resident_bytes": 268435456
  }
}

CPU, open-file, file-size, and address-space ceilings are inherited OS limits. The runner samples resident memory every 50 ms and checks it against final accounted usage, per worker rather than as an aggregate cgroup limit over descendants. Node reserves large virtual address ranges, so a low address-space ceiling can stop it from starting at all. Each attempt retains at most 64 KiB of stdout and 64 KiB of stderr; the runner drains and discards the rest.

3. Choose what a failure stops

The failure policy decides what happens to other workers when one exhausts its attempts. The report download includes host-branches.json and process-branches.json, where a test producer always fails, the consumer depends on it, and inventory checkpoints on its own.

chio process init --config host-branches.json --state .chio-branches
chio process run --state .chio-branches --plan process-branches.json
chio process status --state .chio-branches

You should see the run exit 1 under continue_independent: inventory finishes, the consumer fails as dependency_failed without spending an attempt, and the run stays incomplete because the producer failed.

Use stop when one failed required worker invalidates the whole plan. Use continue_independent when independent branches still produce useful results; each outcome stays individual. Use supervised when a parent inspects a failed dynamic child and chooses a fallback, and settles that failure explicitly.

4. Let a waiting parent release its slot

An adaptive worker spawns children through host-defined spawn_<template> tools. The host template selects the delegable routes and a maximum share; the identically named run template selects the executable, directory, configuration, and attempt limits. The child receives narrowed authority and a bootstrap input holding the template's configuration and the task from the guarded submission.

{"spawn_templates":[{"id":"reader","tools":[{"server_id":"reports","tool_name":"read"}],"max_budget_share_bps":3000}],"supervised_children":true}

Parent routes: reports/read, chio-process/spawn_reader,
              chio-process/wait_children, chio-process/settle_children
Run plan: failure_policy = "supervised"; templates includes reader.

A waiting parent saves its next join poll in its checkpoint and exits 75. The runner records a suspension and gives the slot to the child; when the child's terminal exit is observed, the parent resumes under the same process identity and attempt budget and reads the retained join. Applications on the Computer facade wait on a whole execution family with run.joinTree(); inside a plan, a parent waits on its direct children.

import sys


def value(response):
    if (
        response["verdict"] != "allow"
        or response["terminal_state"]["state"] != "completed"
        or response["output"]["kind"] != "value"
    ):
        raise RuntimeError("Child operation did not complete")
    return response["output"]["value"]


cp = client.inspect()["checkpoint"]
state = cp["value"] or {}
join = state.get("report.join.v1")
if join is None:
    child = value(
        client.invoke(
            "spawn-report-reader",
            "chio-process",
            "spawn_reader",
            {
                "input": {"source": "report-42"},
                "budget_share_bps": 2000,
            },
        )
    )["process"]
    join = {"child": child, "poll": 0, "done": False}
    client.checkpoint(cp["revision"], {**state, "report.join.v1": join})
if not join["done"]:
    outcome = value(
        client.invoke(
            f"settle-reader-{join['poll']}",
            "chio-process",
            "settle_children",
            {"children": [join["child"]]},
        )
    )
    cp = client.inspect()["checkpoint"]
    current = cp["value"] or {}
    next_join = {**join, "done": outcome["complete"], "outcome": outcome}
    if not outcome["complete"]:
        next_join["poll"] += 1
    client.checkpoint(
        cp["revision"], {**current, "report.join.v1": next_join}
    )
    if not outcome["complete"]:
        sys.exit(75)
# Inspect successful/outcomes and validate artifacts before exiting zero.
# This example reports a handled failure; it does not reissue the child's effects.

Saving the next poll before exit keeps a completed pending poll from replaying forever, and a lost checkpoint reply is read back as committed on restart. The spawn key stays constant. wait_children returns on success only; settle_children observes direct, dynamically spawned children reaching success or failure, and refuses declared workers, siblings, grandchildren, and duplicate targets. Settlement needs both the host's supervised_children and the plan's failure_policy: supervised.

Settlement commits responsibility before it returns its guarded output, and a transport failure leaves that record in place. A completed supervisor covers the failed children it settled and the failures beneath them. A failed supervisor covers none of them on its own. Version 2 run reports keep handled_failures and unhandled_failures with their settlement request identities. A fallback child is a new actor: settlement never authorizes it, or a fresh key, to repeat an uncertain effect of the old child.

5. Put a worker in an immutable container image

Direct workers share the operator's OS account unless an isolation profile is configured. Select the native container profile on a new plan when a worker must run without the operator's filesystem and network. Build the application and its dependencies into the image first; the runner uses only an installed image, by its immutable ID, with a literal in-image argv and cwd /work.

{
  "process": "consumer",
  "container": {
    "image": "sha256:REPLACE_WITH_64_LOWERCASE_HEX_DIGITS"
  },
  "command": [
    "/usr/local/bin/node",
    "/app/consumer.mjs"
  ],
  "cwd": "/work",
  "input": {
    "directory": "/work"
  },
  "max_attempts": 2,
  "timeout_seconds": 60
}

The credential still arrives on private stdin, and the only host mount is the current worker socket inode. Scratch space disappears with the container, so keep durable progress in the process checkpoint or an explicitly mediated resource. The profile's fixed bounds and ownership records are in the run plan reference; process confinement covers choosing and checking the boundary.

6. Resume the same plan after an interruption

Run the same command with the same state and plan. The runner reuses recorded completions, attempts, and dependencies.

chio process run --state .chio --plan process.json

You should see completed workers stay completed with their attempt counts unchanged. A plan binds once: max_attempts, max_suspensions, deadlines, and failure_policy stay fixed, so editing the plan never gives an exhausted run another chance. Recorded failed attempts and suspensions survive a host restart, and an ordinary failure retries after one second while budget remains.

The worker resumes its own application checkpoint and reuses its logical operation keys. A failed exit leaves the tool effect open, and the runner treats it that way: a definite pre-exec refusal releases the unexecuted reservation, while an uncertain launch or a lost wait observation goes to reconciliation. A host-wide cancellation, a listener failure, or a storage failure stops the application under every policy.

7. Verify application completion

Compare all three policies on fresh hosts with the exercise in the report download. It forces a declared producer to exit 1, and in the supervised profile runs one parent and one failing dynamic child at max_parallel: 1.

uv run python exercise-supervision.py --output supervision-exercise

{
  "stop": {
    "complete": false,
    "workers": [
      {
        "process": "consumer",
        "state": "pending",
        "attempts": 0,
        "suspensions": 0,
        "outcome": null
      },
      {
        "process": "inventory",
        "state": "pending",
        "attempts": 0,
        "suspensions": 0,
        "outcome": null
      },
      {
        "process": "producer",
        "state": "failed",
        "attempts": 1,
        "suspensions": 0,
        "outcome": "exit_1"
      }
    ]
  },
  "continue_independent": {
    "complete": false,
    "workers": [
      {
        "process": "consumer",
        "state": "failed",
        "attempts": 0,
        "suspensions": 0,
        "outcome": "dependency_failed"
      },
      {
        "process": "inventory",
        "state": "completed",
        "attempts": 1,
        "suspensions": 0,
        "outcome": "exit_0"
      },
      {
        "process": "producer",
        "state": "failed",
        "attempts": 1,
        "suspensions": 0,
        "outcome": "exit_1"
      }
    ]
  },
  "supervised": {
    "complete": true,
    "workers": [
      {
        "process": "dyn_1",
        "state": "failed",
        "attempts": 1,
        "suspensions": 0,
        "outcome": "exit_1"
      },
      {
        "process": "root",
        "state": "completed",
        "attempts": 2,
        "suspensions": 1,
        "outcome": "exit_0"
      }
    ],
    "handled_failures": [
      {
        "process": "dyn_1",
        "supervisor": "root",
        "settled_child": "dyn_1",
        "settlement_request_id": "process:fd7ca0b8b01a323c88ad40d4d9f5a6158f303f654bd6accd804f9ea2218cac0a"
      }
    ],
    "unhandled_failures": []
  }
}

You should see the dependent consumer never run under stop, inventory complete while the run stays incomplete under continue_independent, and the supervised parent finish after two attempts and one suspension, with dyn_1 listed in handled_failures and its failed attempt in the child's log. A complete run can still list a child as failed. Repeat the completed plan and the attempt counts stay the same.

Exit zero means the OS worker completed according to its program, and it releases declared dependents. The report, patch, or provider effect is yours to check: read the process checkpoint and verify the original signed receipts with chio receipt verify and the host's kernel.pub. A pre-exec refusal, a worker killed after launch, and a lost wait observation each have their own retry rules in Calls and recovery.

To hand a reviewer every worker's outcome as one signed file, stop the host and attest the report run against its plan. The reviewer verifies the artifact with copies of the kernel key and runtime ID you recorded when you initialized the host, without opening .chio.

chio process attest-outcomes --state .chio --plan process.json --out outcomes.json
chio process verify-outcomes --artifact outcomes.json \
  --trusted-kernel-pubkey kernel.pub --runtime-id 5c1e…9a04

{
  "schema": "chio.process.outcome-verification.v1",
  "runtime_id": "5c1e…9a04",
  "workers": [
    { "process": "consumer", "observed_operation_state": "completed" },
    { "process": "producer", "observed_operation_state": "completed" }
  ],
  "artifact_schema": "chio.process.worker-outcomes.v3",
  "checks": ["signer_pin", "runtime_pin", "issued_capabilities", "worker_responses", "runner_completion"],
  "unchecked": ["physical_effects", "scenario_matrix", "original_interrupted_launch", "target_death_without_terminal"]
}

You should see both workers with their observed operation state. The artifact covers interrupted workers too, so an incident keeps its uncertainty; the report itself is still yours to check.

Run plan reference