Node Lifecycle

What a stop signal costs if nothing handles it

A node is one OS process hosting one kernel. Deploys, scale events, and node rotations all end that process the same way: a stop signal, a grace window, then an unconditional kill. With no handler installed, SIGTERM takes its default disposition and the process dies at once, ignoring the whole grace window. On a process that mediates every action an agent takes, that severs in-flight evaluations and loses the receipt for work that already happened.

crates/protocol/chio-http-serve is the leaf crate that turns the signal into a bounded drain. It depends on no other chio-* crate and carries no protocol, capability, or authorization logic. Its remit is two things: when a process stops accepting work, and how much concurrent load one site admits before it denies more. The second half is the hygiene stack (408 request timeout, 503 load shed in front of a concurrency limit, 413 body cap, and a semaphore-backed connection cap in MaxConnListener), and this page covers only the one knob that interacts with the drain window, the request timeout. Backpressure & Limits owns the rest of that stack, including which serve sites turn the request timeout off and what they bound instead. Six axum::serve sites in the repo route their stop path through the crate.

Model

Four terms, used consistently on this page. A stop signal is the first SIGTERM or Ctrl-C the process observes. The drain window is the wall-clock bound that arms when that signal fires. A flush hook is the site-supplied future that runs after the drain. A drain ends clean or forced, and nothing else.

How a node stops

Only the post-signal drain is bounded

The defect this design exists to avoid is a naive tokio::time::timeout(drain_timeout, server) around the whole serve future, which kills a perfectly healthy server drain_timeout after boot. So run_until_drained takes a shutdown receiver rather than wrapping a timer around uptime, and runs two phases:

let mut server = Box::pin(server.into_future());
let signalled = wait_for_shutdown(shutdown);

let outcome = tokio::select! {
    // Phase 1: serve until the server exits on its own or the signal fires.
    result = &mut server => match result {
        Ok(()) => DrainOutcome::Clean,
        Err(source) => return Err(ServeError::Io(source)),
    },
    () = signalled => {
        // Phase 2: graceful shutdown has stopped the accept loop; bound only
        // the remaining in-flight drain.
        match tokio::time::timeout(drain_timeout, &mut server).await {
            Ok(Ok(())) => DrainOutcome::Clean,
            Ok(Err(source)) => return Err(ServeError::Io(source)),
            Err(_elapsed) => {
                let drain_ms = u64::try_from(drain_timeout.as_millis()).unwrap_or(u64::MAX);
                warn!(
                    drain_timeout_ms = drain_ms,
                    "drain deadline exceeded; force-closing remaining connections"
                );
                DrainOutcome::Forced
            }
        }
    }
};

The serve future is box-pinned rather than stack-pinned for one reason: on the forced path it has to be dropped, and a stack Pin<&mut _> only borrows. Dropping the owned future releases the accept loop and the listener before the flush hook runs. What the drop does not do is stop the straggler: axum serves each accepted connection on a detached task that a bounded drain can neither join nor cancel, so the handler may still be finishing.

Clean and forced

The signal is fail-closed against its own installation

A handler that fails to install is the case that turns a shutdown mechanism into a crash loop. Both branches of shutdown_signal log loudly and park on pending() instead of resolving, so a failed install can never fire a spurious shutdown at startup:

match signal(SignalKind::terminate()) {
    Ok(mut stream) => {
        stream.recv().await;
    }
    Err(source) => {
        error!(%source, "cannot install SIGTERM handler; the Ctrl-C path still governs shutdown");
        std::future::pending::<()>().await;
    }
}

If both installs fail the future never resolves, the process keeps running, and the platform escalates to an unconditional kill. That is the behavior of a process with no shutdown wiring at all, which is the point: the degraded mode is the old mode, logged, not a boot loop. On non-unix targets the SIGTERM branch is a compile-time pending(), so Ctrl-C is the only stop signal there.

Background tasks stop on the same channel

A drain that only stops the HTTP loop leaves timers and sync loops running past the deadline. A cooperating task takes a subscribe() receiver and races its own work against it. Subscribing is opt-in, not automatic: two background tasks in the repo cooperate this way, and api-protect's reserved-hold reaper is aborted instead. The canonical stop check is documented on subscribe itself:

while !*rx.borrow_and_update() {
    if rx.changed().await.is_err() {
        break;
    }
}

The MCP edge session reaper (remote_mcp/http_service_auth.rs) races its interval sleep against rx.changed() so it stops promptly rather than after a full interval. Trust-control goes further: its cluster sync loop passes the same receiver into each blocking sync round, so a stop signal landing mid-round does not force it to wait out every remaining peer, and the post-drain join gets only drain_timeout minus the time already spent since the signal. cluster_join_never_extends_teardown_past_the_drain_window in service_runtime/init.rs asserts that elapsed plus join budget never exceeds one drain window, so teardown never stacks two waits inside a grace period sized for one. Only the outbound join is bounded that way: the timeout abandons the sync task rather than cancelling its in-flight peer call, which the code accepts because outbound sync is best-effort catch-up that resumes on the next boot. If the serve future ends without a signal ever arriving, no elapsed time was recorded and the join takes the full drain window.

The drain-window contract

This is the contract the Cloud Run, ECS, and Azure Container Apps recipes have to satisfy. Four numbers, in one order, and every recipe is judged against it:

request_timeout  <  drain_timeout  <=  TimeoutStopSec  <=  platform stop grace
      20s        <       25s       <=       35s        <=   operator-set

The first term is present only where the request timeout is on. Three of the six serve sites run without one, and there the drain deadline is the first bound a long handler meets. The last two terms are documented convention, not code: nothing in the crate reads TimeoutStopSec or the platform grace, so a misconfigured deployment fails silently as a truncated drain.

The first inequality is not a convention, it is asserted. default_request_timeout_stays_within_drain_window fails the suite if the two constants ever invert, and slow_request_times_out_cleanly_inside_the_drain_window proves the behavior it buys: a handler parked for 30 seconds, stopped mid-request, returns 408 and the drain still reports Clean. A site that lengthens its request timeout must lengthen its drain to match.

The last two rows are the operator's half. The reference units make the stop signal explicit and hold the escalation:

Restart=on-failure
RestartSec=5s
# Deliver SIGTERM first (systemd's default, made explicit) so the graceful drain
# runs, and hold the kill escalation until after the bounded drain plus the
# receipt-flush margin can finish. The drain deadline is 25s; the platform grace
# period must be at least TimeoutStopSec.
KillSignal=SIGTERM
KillMode=mixed
TimeoutStopSec=35s

KillMode=mixed is specific to the MCP edge. The upstream tool server launched after -- shares the unit's control group, and the default kill mode would signal it at the instant the edge began draining, tearing the upstream out from under in-flight calls. mixed signals the main process first and escalates the rest of the group to SIGKILL only after TimeoutStopSec. It is the one line of that unit the other reference units do not share, because the edge is the only serve site that launches a child.

One site derives its window instead of taking the default

chio api protect writes each receipt synchronously in the handler after the upstream hop returns, so its drain has to outlast the hop, not a constant. It computes drain_timeout = upstream_request_timeout + PROXY_DRAIN_MARGIN, where the margin is a 5s constant in proxy/state.rs, and runs no request timeout at all, because an outer timeout layer would drop the handler mid-hop and skip receipt finalization for a call that may already have reached the upstream. Two tests pin it: drain_window_always_outlasts_the_configured_upstream_timeout over hop ceilings of 1s, 20s, 30s, 60s, and 300s, and default_upstream_timeout_preserves_the_default_drain_window, which holds the default 20s hop and 25s drain pairing. DEFAULT_UPSTREAM_REQUEST_TIMEOUT is 20s and --upstream-timeout-secs overrides it. The margin is not itself configurable.

The flush-hook contract

on_drained is a Future<Output = Result<(), String>> the site supplies. It runs last, on both outcomes, and its error becomes ServeError::Flush and a non-zero exit. That is loud on purpose: a failed flush means a receipt may not be durable, and an operator has to see that rather than a silent success.

The obligation runs the other way too. On a clean drain the hook has the store to itself. On a forced drain a straggler survived the deadline on a detached task, so the hook must be safe against a concurrent writer. Sites keep that safe by making every acknowledged receipt durable synchronously in the handler, or through a commit actor that acknowledges only after the write reaches the WAL, never in the hook itself. A straggler can then never cost an acknowledged receipt. This is a contract stated in the run_until_drained doc comment, not a property the compiler or the suite enforces: nothing checks a hook's concurrency safety, and no site currently supplies a hook that could violate it.

Note what the drop on the forced path does and does not buy. drain.rs drops the serve future before awaiting the hook, which releases the accept loop and the listener. RFC-0010 reads that as closing the remaining connections before the flush so no handler can enqueue a write afterwards; drain.rs is narrower and states that the drop does not guarantee the handler has stopped. This page follows the crate.

What a dropped evaluation still writes

A forced drain can leave a handler running on a task nothing can join. If that handler was mediating a tool call, the evaluation future inside it is dropped mid-flight, and the question worth answering is not whether the task stopped but whether the governed action it was carrying left a record. That answer is not in chio-http-serve at all. It is a kernel-side guard that runs from Drop, which is what makes it reachable when a future is discarded rather than returned from.

PostAdmissionDropGuard (chio-kernel/src/kernel/kernel_drop_guard.rs) borrows the request, the capability, the pre-execution budget mutation, and any payment authorization, and it owns the buffered child-request receipts the nested-flow bridge pushes into it during dispatch. It owns them rather than the evaluation stack frame precisely so a drop can still flush them. Three instances exist along one evaluation, each armed across exactly one await that an abandoning caller could drop: the guard pipeline, the runtime-admission readiness wait, and the tool-server dispatch. The first two disarm on return. The third calls mark_dispatch_started before lending its receipt buffer to the bridge.

That one bit decides the whole behavior. Before dispatch no side effect is possible, so handle_pre_dispatch_drop reverses every pre-execution mutation in four independently attempted steps: the monetary hold with its payment release, an invocation-only budget increment, the runtime-admission reservations, and a delegated capability's share of its parent budget. Step four is gated on budget_lease_acquired, because releasing a holder lease this evaluation never took would free another evaluation's live share. A clean unwind records no receipt at all, which is the intended exit; only a failing step produces one.

After dispatch starts the guard is fail-closed. The tool server may already have performed the side effect, so releasing the runtime-admission reservations would license a replay. The guard instead flushes the buffered child receipts onto the append-only log one at a time, so a wedged writer that fails one does not discard the receipts queued behind it, retains the reservations, and always records a signed cancellation receipt. Four constants name the reason that receipt can carry:

const POST_ADMISSION_DROP_REASON: &str = "tool evaluation future dropped after admission";
const POST_DISPATCH_CREDENTIAL_COMMIT_FAILURE_REASON: &str =
    "dispatch credential commit failed after tool execution";
const POST_DISPATCH_URL_ELICITATION_REASON: &str =
    "tool server returned URL elicitation after dispatch; outcome is unknown";
const PRE_DISPATCH_CLEANUP_FAULT_REASON: &str =
    "tool evaluation future dropped before dispatch with cleanup fault";

Whichever string applies becomes Decision::Cancelled { reason } on a receipt that build_cancelled_response_with_metadata_and_payee_binding signs and records through the same path a normal receipt takes, with terminal state OperationTerminalState::Cancelled. The fault receipt carries more: a chio_runtime.pre_dispatch_cleanup_faults array whose entries name the failing step, its redacted reason, and the hold or reservation ids that step was unwinding, so a stuck hold is locatable from the receipt rather than by cross-referencing the admission metadata. On a durable kernel the post-dispatch path also terminalizes the admission record, and terminalize_dispatch_committed_admission refuses when a durable outcome already exists, so it cannot overwrite a return that did complete.

Where it is wired

Six sites, one implementation. Each keeps its own axum::serve expression, appends .with_graceful_shutdown(controller.signalled()), and returns through run_until_drained.

Trust-control is the only site that joins a cooperating task after the drain. The MCP edge reaper observes the same signal but is left detached and ends with the process, and api-protect aborts its reaper rather than stopping it cooperatively.

Guarantees and limits

Two further limits, stated outright. shutdown_signal registers SIGTERM and Ctrl-C only, so SIGHUP and SIGQUIT keep their default disposition and kill the process outright with no drain. That follows from the two branches in signal.rs and a grep of the crate; individual binaries were not each audited for a separate handler. And the crate's suite runs in-process against ephemeral listeners with trigger() standing in for the signal; a repo-wide grep finds no SIGTERM-under-load soak. The numbers here are bounds the code enforces, not measured drain times from a fleet.

Next Steps

• Sidecar HTTP Service · the deployed-process view: flags, health endpoints, and supervision
• Cloud Run, ECS Fargate, Azure Container Apps · three vendor recipes judged against the drain-window contract
• Node Observability · what to watch while a node is up, including the drain log line
• Backup & Restore · receipt-store mechanics, and why a synchronous commit is what makes the drain sufficient