Node Lifecycle

Reference systemd unit

The reference units make the stop signal explicit and hold the kill escalation until the bounded drain and the receipt-flush margin can finish.

Restart=on-failure
RestartSec=5s
# Deliver SIGTERM first (systemd's default, made explicit) so the graceful drain
# runs, and hold the kill escalation until after the bounded drain plus the
# receipt-flush margin can finish. The drain deadline is 25s; the platform grace
# period must be at least TimeoutStopSec.
KillSignal=SIGTERM
KillMode=mixed
TimeoutStopSec=35s

KillMode=mixed is specific to the MCP edge. The upstream tool server launched after -- shares the unit’s control group, and the default kill mode would signal it the instant the edge began draining. mixed signals the main process first and escalates the rest of the group to SIGKILL only after TimeoutStopSec. The edge is the only serving site that launches a child, so the other reference units omit this line.

Timing constants

The flush hook runs after the drain, outside its timeout, so 25 seconds bounds the drain and not the whole stop. default_request_timeout_stays_within_drain_window checks the constant ordering, and slow_request_times_out_cleanly_inside_the_drain_window runs a slow handler through shutdown and checks its 408 and a clean drain.

Derived drain at the API-protect sidecar

chio api protect writes each receipt synchronously in the handler after the upstream hop returns, so its drain has to outlast the hop. It computes drain_timeout = upstream_request_timeout + PROXY_DRAIN_MARGIN, with the margin a 5-second constant in proxy/state.rs, and runs no request timeout: an outer timeout would drop the handler mid-hop and skip receipt finalization for a call that may already have reached the upstream.

DEFAULT_UPSTREAM_REQUEST_TIMEOUT is 20 seconds, and --upstream-timeout-secs overrides it; the margin is fixed. drain_window_always_outlasts_the_configured_upstream_timeout covers hop ceilings of 1, 20, 30, 60, and 300 seconds, and default_upstream_timeout_preserves_the_default_drain_window holds the default 20-second hop and 25-second drain.

On a platform with a fixed grace, the derived drain is the number that moves. The Cloud Run drain row puts that platform’s grace at a fixed 10 seconds with no service-level field to raise it, so SIGKILL arrives 10 seconds into a 25-second default drain and takes the last 15 seconds. Bring the hop ceiling down until the derived drain fits the grace.

Two-phase drain

A timer wrapped around the whole serve future would kill a healthy server drain_timeout after boot. run_until_drained takes a shutdown receiver instead and bounds only the in-flight drain that follows the signal.

let mut server = Box::pin(server.into_future());
let signalled = wait_for_shutdown(shutdown);

let outcome = tokio::select! {
    // Phase 1: serve until the server exits on its own or the signal fires.
    result = &mut server => match result {
        Ok(()) => DrainOutcome::Clean,
        Err(source) => return Err(ServeError::Io(source)),
    },
    () = signalled => {
        // Phase 2: graceful shutdown has stopped the accept loop; bound only
        // the remaining in-flight drain.
        match tokio::time::timeout(drain_timeout, &mut server).await {
            Ok(Ok(())) => DrainOutcome::Clean,
            Ok(Err(source)) => return Err(ServeError::Io(source)),
            Err(_elapsed) => {
                let drain_ms = u64::try_from(drain_timeout.as_millis()).unwrap_or(u64::MAX);
                warn!(
                    drain_timeout_ms = drain_ms,
                    "drain deadline exceeded; force-closing remaining connections"
                );
                DrainOutcome::Forced
            }
        }
    }
};

The serve future is box-pinned so the forced path can drop it; a stack Pin<&mut _> only borrows. Dropping the owned future releases the accept loop and the listener before the flush hook runs. Axum serves each accepted connection on a detached task that a bounded drain can neither join nor cancel, so a handler may still be finishing.

Signal installation

Both branches of shutdown_signal log and park on pending() when a handler fails to install, so a failed install never fires a spurious shutdown at startup and never becomes a crash loop. If both installs fail, the process runs until the platform kills it, the behavior of a process with no shutdown wiring.

match signal(SignalKind::terminate()) {
    Ok(mut stream) => {
        stream.recv().await;
    }
    Err(source) => {
        error!(%source, "cannot install SIGTERM handler; the Ctrl-C path still governs shutdown");
        std::future::pending::<()>().await;
    }
}

install() spawns one task that awaits shutdown_signal(), publishes true, and exits; nothing re-arms it. Once a handler is registered, Tokio never unregisters the underlying libc handler, so a second SIGTERM or Ctrl-C reaches no listening stream and neither terminates the process nor shortens the drain. On non-Unix targets the SIGTERM branch compiles to pending().

Background tasks

A cooperating task takes a subscribe() receiver and races its own work against it. Subscribing is opt-in: two background tasks cooperate this way, and the API-protect sidecar aborts its reserved-hold reaper instead. The canonical stop check is documented on subscribe:

while !*rx.borrow_and_update() {
    if rx.changed().await.is_err() {
        break;
    }
}

The MCP edge session reaper (remote_mcp/http_service_auth.rs) races its interval sleep against rx.changed(), so it stops promptly. Trust-control passes the receiver into each blocking cluster sync round, so a signal mid-round does not wait out every remaining peer, and the post-drain join gets drain_timeout minus the time already spent since the signal.

cluster_join_never_extends_teardown_past_the_drain_window in service_runtime/init.rs asserts that elapsed time plus the join budget never exceeds one drain window. The timeout abandons the sync task without cancelling its in-flight peer call; outbound sync is catch-up that resumes on the next boot. If the serve future ends without a signal, no elapsed time was recorded and the join takes the full window.

Shutdown controller

Embedding hosts can use ShutdownController::manual with their own signal source; shipped services install the OS-signal handler. Cooperating tasks subscribe to the same watch channel, and closing that channel also wakes the shutdown path. A stop signal is the first SIGTERM or Ctrl-C the process observes; the drain window arms when it fires; the flush hook runs after the drain; a drain ends clean or forced.

on_drained is a Future<Output = Result<(), String>> that runs last, on both outcomes; its error becomes ServeError::Flush. The run_until_drained doc comment states the hook’s contract: it must be safe against a concurrent writer. Sites meet it by making every acknowledged receipt durable synchronously in the handler, or through a commit actor that acknowledges only after the write reaches the WAL, so a straggling handler never costs an acknowledged receipt. flush_error_surfaces_as_serve_error tests the error path.

Serving sites

Six sites share one implementation. Each keeps its own axum::serve expression, appends .with_graceful_shutdown(controller.signalled()), and returns through run_until_drained.

A blanket timeout at trust-control would drop an HA budget authorize after the local exposure write but before its rollback branch; at the MCP edge it would close a healthy idle stream with 408. Trust-control is the only site that joins a cooperating task after the drain.

Drop guard

PostAdmissionDropGuard (chio-kernel/src/kernel/kernel_drop_guard.rs) borrows the request, the capability, the pre-execution budget mutation, and any payment authorization, and owns the buffered child-request receipts that the nested-flow bridge pushes into it during dispatch, so a drop can still flush them. Three instances guard one evaluation, each armed across one await an abandoning caller could drop: the guard pipeline, the runtime-admission readiness wait, and the tool-server dispatch. The first two disarm on return; the third calls mark_dispatch_started before lending its receipt buffer to the bridge.

Before dispatch, handle_pre_dispatch_drop attempts four independent steps: the monetary hold with its payment release, an invocation-only budget increment, the runtime-admission reservations, and a delegated capability’s share of its parent budget. The fourth is gated on budget_lease_acquired, because releasing a holder lease this evaluation never took would free another evaluation’s live share. Only a failing step produces a receipt. After dispatch, the guard flushes child receipts onto the append-only log one at a time, so one failed write does not discard those queued behind it, and records persistence failures as audit faults. Four constants name the cancellation reason:

const POST_ADMISSION_DROP_REASON: &str = "tool evaluation future dropped after admission";
const POST_DISPATCH_CREDENTIAL_COMMIT_FAILURE_REASON: &str =
    "dispatch credential commit failed after tool execution";
const POST_DISPATCH_URL_ELICITATION_REASON: &str =
    "tool server returned URL elicitation after dispatch; outcome is unknown";
const PRE_DISPATCH_CLEANUP_FAULT_REASON: &str =
    "tool evaluation future dropped before dispatch with cleanup fault";

The reason becomes Decision::Cancelled { reason } on a receipt that build_cancelled_response_with_metadata_and_payee_binding signs and records through the normal receipt path. A fault receipt adds a chio_runtime.pre_dispatch_cleanup_faults array naming the failing step, its redacted reason, and the hold or reservation ids it was unwinding, so a stuck hold is locatable from the receipt. On a durable kernel the post-dispatch path also terminalizes the admission record, and terminalize_dispatch_committed_admission refuses when a durable outcome already exists.

Every path logs and continues, because a panic inside Drop during an unwind aborts the process. The audit_fault values are post_admission_drop_receipt_unrecorded, post_admission_drop_child_receipts_unrecorded, post_admission_drop_admission_unterminalized, and pre_dispatch_cleanup_fault_receipt_unrecorded. A receipt store already unreachable when the guard runs produces one of those lines and nothing else.