Session Lifecycle

A ledger the kernel does not own

The kernel’s pure core lists what it refuses to do, and the list is about state: crates/kernel/chio-kernel-core/src/evaluate.rs fences revocation lookup, budget mutation, and receipt persistence out of the verdict path. Session lifecycle is not on that list because the kernel never sees it. A session id, an idle deadline, and a tombstone are transport-level facts owned by the process that terminates HTTP, and one process answers all of them by reading its own disk. That is the Node rung.

Everything here is the hosted MCP edge: chio mcp serve-http, implemented in crates/protocol/chio-mcp-remote/src/remote_mcp/ and re-exported by chio-hosted-mcp. RemoteSessionLedger is constructed nowhere else in the tree: a stdio edge has one conversation and no ledger to keep, and the sidecar in Sidecar HTTP Service mediates per request. Durability is opt-in and has two preconditions. Without a global --session-db path the ledger is entirely in memory, every mechanism below still runs, and a restart starts from nothing. With that path but no local secret material the two tables diverge: tombstones are written, resumable rows are not. resume_record returns None unless a resume integrity seed can be derived from the authority database, the first configured seed file, the control token, or the auth token, and persist_resumable_record writes nothing when it does. A node given only --session-db therefore blocks reuse of dead session ids across a restart but never resumes a live one.

This is not a second copy of Node Lifecycle. That page is the process stopping: one controller, one stop signal, one bounded drain. This one is a single client conversation ending, which happens many times inside one process lifetime and has its own drain, its own deadline, and its own record on disk.

Two maps, two tables, six states

RemoteSessionLedger is two async-locked maps and a path. Live sessions sit in active as Arc<RemoteSession>, each owning a worker thread, a kernel, and, unless the server was started with --shared-hosted-owner, its own upstream MCP child process. Dead ones sit in terminal as Arc<RemoteSessionDiagnosticRecord>, which is a tombstone: no worker, no kernel, just enough of the session to answer questions about it. lookup checks active first and falls back to terminal.

On disk those two maps are two tables in one SQLite file, created on every open:

pub(super) const SESSION_ACTIVE_TABLE: &str = "remote_active_sessions";
pub(super) const SESSION_TOMBSTONE_TABLE: &str = "remote_session_tombstones";

// ...

pub(super) fn open_session_state_db(path: &FsPath) -> Result<Connection, CliError> {
    let conn = Connection::open(path)?;
    conn.execute_batch(&format!(
        "CREATE TABLE IF NOT EXISTS {active_table} (
            session_id TEXT PRIMARY KEY NOT NULL,
            updated_at INTEGER NOT NULL,
            record_json TEXT NOT NULL
        );
        CREATE TABLE IF NOT EXISTS {table} (
            session_id TEXT PRIMARY KEY NOT NULL,
            terminal_at INTEGER NOT NULL,
            record_json TEXT NOT NULL
        );",
        active_table = SESSION_ACTIVE_TABLE,
        table = SESSION_TOMBSTONE_TABLE,
    ))?;
    Ok(conn)
}

Six states exist, and only one of them is live. The HTTP column is what a client gets for reusing that session id on POST /mcp or GET /mcp, from both the live check (validate_session_lifecycle) and the tombstone check (terminal_session_response). DELETE /mcp does not run the live check at all: it validates the auth context and then terminalizes whatever active session it found, so a draining session answers 204 there rather than 409. Only its tombstone branch returns 410.

The non-ready outcomes are named distinctly on purpose, and the distinction is asserted end to end by mcp_serve_http_admin_drain_shutdown_and_delete_have_distinct_terminal_states. Operator-facing JSON reports the state lowercase through RemoteSessionState::as_str, but the persisted record_json uses the derived serde name, so a row read with sqlite3 says "Ready" where the API says "ready".

Four knobs, read once at startup

const DEFAULT_SESSION_IDLE_EXPIRY_MILLIS: u64 = 15 * 60 * 1000;
const DEFAULT_SESSION_DRAIN_GRACE_MILLIS: u64 = 5 * 1000;
const DEFAULT_SESSION_REAPER_INTERVAL_MILLIS: u64 = 250;
const DEFAULT_SESSION_TOMBSTONE_RETENTION_MILLIS: u64 = 30 * 60 * 1000;
const SESSION_TOUCH_PERSIST_INTERVAL_MILLIS: u64 = 5_000;

read_env_u64 keeps a parsed value only when it is greater than zero, so an unset, unparseable, or zero variable silently takes the default rather than disabling the mechanism. All four are read at startup and never re-read; there is no CLI flag and no policy field for any of them. GET /admin/health reports all four back, which is the reliable way to confirm a tuning actually landed. GET /admin/sessions reports the first three only.

One session, in order

Ready. A session is built by spawn_session before initialize is answered, but it only reaches the ledger if the initialize response is a success. mark_ready records the negotiated protocol version, the initialize params, and the peer capabilities, stamps idle_expires_at = now + idle_expiry_millis, and writes the first resumable row. Insertion into the active map follows. Anything that fails before then leaves no ledger entry and no row.

Touch. Every subsequent POST and GET that resolves to a live session calls touch, which re-stamps last_seen_at and the idle deadline, and does so only in Ready. Persistence is throttled: the row is rewritten only when at least SESSION_TOUCH_PERSIST_INTERVAL_MILLIS has passed since the last sighting, so a chatty client writes SQLite once every five seconds rather than once per call. The consequence is that the persisted deadline trails the live one by up to five seconds, and a restored session can expire slightly earlier than it would have.

Drain. The admin drain route is not a graceful finish. It deletes the resumable row first, then moves the session to Draining and sets drain_deadline_at = now + drain_grace_millis. From that moment POST and GET get 409, and the grace period governs only how long the session lingers before the reaper converts it to Deleted. Deleting the resumable row before the state changes is the point: a crash mid-drain must not leave a row that would resurrect a session an operator has already started retiring. If that delete fails, begin_draining returns the error without touching the state, the admin route answers 500, and the session keeps serving. A drain that reports failure has not started.

The reaper. session_reaper_loop is one spawned task that races its interval against the process shutdown watch with tokio::select!, so a node drain stops it immediately instead of after a full interval. It is not the only caller. cleanup_due_sessions also runs on every session resolution, on GET /admin/health, on GET /admin/sessions, and once at boot after restore. A request never observes a session that was due for expiry when the request arrived, whatever the interval is set to. That coverage is not free. Every cleanup_due_sessions ends in purge_old_terminal_records, which with --session-db set opens the database and issues a DELETE, so every request carrying a session id pays one SQLite open on a connection that is not pooled.

match snapshot.state {
    RemoteSessionState::Ready if snapshot.idle_expires_at <= now => {
        match self.mark_expired(&session).await {
            Ok(()) => {
                self.active.lock().await.remove(&session.session_id);
            }
            Err(error) => {
                warn!(
                    session_id = %session.session_id,
                    error = %error,
                    "failed to expire MCP session without resumable-state risk"
                );
            }
        }
    }
    RemoteSessionState::Ready => {}
    RemoteSessionState::Draining => {
        if snapshot
            .drain_deadline_at
            .is_some_and(|deadline| deadline <= now)
        {
            match self.mark_deleted(&session).await {
                Ok(()) => {
                    self.active.lock().await.remove(&session.session_id);
                }
                Err(error) => {
                    warn!(
                        session_id = %session.session_id,
                        error = %error,
                        "failed to delete drained MCP session without resumable-state risk"
                    );
                }
            }
        }
    }
    RemoteSessionState::Initializing
    | RemoteSessionState::Deleted
    | RemoteSessionState::Expired => {}
    RemoteSessionState::Closed => {
        self.active.lock().await.remove(&session.session_id);
    }
}

Read the error arms. A session is removed from the active map only if its terminal transition succeeded, and the transition fails only when both the tombstone write and the resumable-row delete failed. That is the narrow case worth protecting: dropping a session from memory while its resumable row survives on disk is what would let a dead session come back. A tombstone write that fails on its own, with the row delete succeeding, still returns Ok. The session is then gone from this process with nothing on disk to answer 410, and after a restart that id answers 404.

The tombstone is written first

transition_to_terminal is the single path to Deleted, Expired, and Closed; the other three states are rejected as arguments. It refuses to run twice, returning early when the session is already terminal, because a reaper expiry racing an admin shutdown would otherwise overwrite the first tombstone’s state and push terminal_at forward, extending retention. Then the ordering, which is the whole durability argument:

• Write the tombstone. On success the restore path is already guarded, whatever happens next.
• Delete the resumable row. If this fails but the tombstone landed, warn and continue: the tombstone will beat the stale row at boot.
• If the tombstone did not land and the row delete also failed, return an error. The caller answers 500 and the session stays live, because a session with a surviving resumable row and no tombstone must not be dropped from memory.
• Only then mutate the in-memory state and insert into the terminal map.

The tombstone keeps the session’s auth context, one RemoteSessionCapability per grant (id, issuer_public_key, subject_public_key), its ownership snapshot, and its final lifecycle. That is not diagnostics for its own sake. Reusing a tombstoned session id revalidates the caller against the stored auth context before the terminal status is returned, so a different principal gets 403 rather than learning that the session existed and how it ended. Those ids are what lets POST /admin/sessions/{id}/trust revoke a dead session’s grants after the fact, and that route needs somewhere to write the revocation: without a control plane, a durable-admission runtime, or --revocation-db it answers 409 remote trust admin requires durable revocation state. Its window is the retention window. Once the tombstone is purged the id is unknown and the route answers 404, so a capability whose expiry outlives thirty minutes outlives the only way to revoke it by session id.

Purge is retention-bounded and asymmetric. In memory, retain drops any record older than the retention window. On disk the delete carries a NOT EXISTS subquery against the active table, so a tombstone whose session id still has an active row is kept past its retention rather than purged into a state where the row could be restored. A failed purge warns and the sweep continues.

What survives a restart

Boot order matters. The ledger is constructed first and loads the tombstone table into memory, dropping rows that do not parse and rows whose payload session id disagrees with the primary key. Both are logged and skipped rather than aborting startup. Then load_active_session_records reads the active table with the tombstone id set in hand and drops three classes of row: any id that carries a parseable tombstone, any row whose payload disagrees with its own primary key, and any row that does not parse. Each dropped id is deleted from disk before the process serves.

A tombstone therefore outranks a resumable row for the same session id, which is what makes the crash-mid-terminalization case safe. The one qualification is that a tombstone which cannot be parsed does not block anything, and the test load_active_session_records_keeps_active_row_when_matching_tombstone_is_malformed pins that: a corrupt tombstone is treated as absent, not as a veto.

Surviving rows go through restore_session, which rebuilds a kernel, an upstream server, and an edge worker per record, and refuses for any of these reasons. A refusal is not fatal to the node: it logs, deletes the row, and moves to the next one.

The policy fingerprint is the one contract that behaves differently, and the difference is worth stating plainly: a policy change does not drop the session. It drops the session’s capabilities.

let issued_capabilities = match record.policy_fingerprint.as_deref() {
    Some(stored)
        if stored == policy_fingerprint
            && stored_capabilities_are_current(&record.issued_capabilities)
            && stored_capability_issuers_are_trusted(&kernel, &record.issued_capabilities)
            && record
                .issued_capabilities
                .iter()
                .all(|capability| capability.subject == agent_public_key) =>
    {
        record.issued_capabilities.clone()
    }
    _ => issue_default_capabilities(&kernel, &agent_public_key, &default_capabilities)?,
};

Stored capabilities are carried across the restart only when the policy fingerprint is unchanged, every token is still inside its expiry, every issuer is still trusted by the rebuilt kernel, and every subject is the restored agent key. Fail any one and the session comes back with freshly issued defaults from the current policy, which is why mcp_serve_http_ready_sessions_reissue_capabilities_after_policy_tightening can restart a node with a narrower tool list and watch the same session id get denied on a tool it was previously allowed. Change the auth mode instead and the session is gone: mcp_serve_http_drops_restored_sessions_when_auth_mode_changes swaps a static bearer for JWT and finds the old session id returns 404.

The last step of boot is cleanup_due_sessions, and it is load-bearing. idle_expires_at is absolute wall-clock milliseconds, so every row written before a long outage is already past its deadline and is expired and tombstoned before the first request is served. Restore runs first, though, which means a node returning from a multi-hour outage pays full session construction, worker thread and upstream child process included, for rows it then immediately reaps. The listening socket is bound before restore begins, so that cost shows up as connection latency rather than refused connections.

Guarantees and limits

Next Steps

• Session-Aware Guards · the journal, the cumulative fields, and what a session-aware guard does when it cannot read them
• Remote MCP Edge · the process that owns this ledger: transport, bearer modes, and the gates a request clears before a session is touched
• Node Lifecycle · the process-level drain that stops the reaper and the difference between a session ending and a node stopping
• Node State on Disk · the provisioning, fencing, and schema refusals this database deliberately does not use
• Revocation Store · what actually invalidates a dead session’s capabilities
• Health & Readiness · the other thing one node can answer about itself without contacting a peer