Admin API

Eleven routes, not four

install_admin_routes makes eleven .route(...) calls. Three of the eleven paths carry both a GET and a POST, so fourteen method-and-path pairs reach fourteen handlers. Count them from the router rather than from prose: the operations runbook exercises two, docs/release/OBSERVABILITY.md documents four, and spec/PROTOCOL.md section 8.3 lists seven. Drain, shutdown, and /admin/metrics appear in none of the three.

pub(super) fn install_admin_routes(router: Router<RemoteAppState>) -> Router<RemoteAppState> {
    router
        .route(ADMIN_HEALTH_PATH, get(handle_admin_health))
        .route(
            ADMIN_AUTHORITY_PATH,
            get(handle_admin_authority).post(handle_admin_rotate_authority),
        )
        .route(ADMIN_TOOL_RECEIPTS_PATH, get(handle_admin_tool_receipts))
        .route(ADMIN_CHILD_RECEIPTS_PATH, get(handle_admin_child_receipts))
        .route(ADMIN_BUDGETS_PATH, get(handle_admin_budgets))
        .route(
            ADMIN_REVOCATIONS_PATH,
            get(handle_admin_revocations).post(handle_admin_revoke_capability),
        )
        .route(
            ADMIN_SESSION_TRUST_PATH,
            get(handle_admin_session_trust).post(handle_admin_revoke_session_trust),
        )
        .route(ADMIN_SESSIONS_PATH, get(handle_admin_sessions))
        .route("/admin/metrics", get(handle_admin_metrics))
        .route(ADMIN_SESSION_DRAIN_PATH, post(handle_admin_session_drain))
        .route(
            ADMIN_SESSION_SHUTDOWN_PATH,
            post(handle_admin_session_shutdown),
        )
}

Ten of the paths come from &'static str constants at the top of session_core.rs; /admin/metrics is the one inline literal. The three parameterized paths use the axum 0.8 brace syntax, {session_id}, and the segment is captured by an AxumPath<String> extractor.

One gate, two checks, no scopes

Every one of the fourteen handlers opens with the same two lines. There is no middleware layer and no per-route authorization: the check is copied into each handler body and is identical everywhere.

fn validate_admin_request(headers: &HeaderMap, admin_token: Option<&str>) -> Result<(), Response> {
    validate_origin(headers)?;
    validate_admin_auth(headers, admin_token)
}

fn validate_admin_auth(headers: &HeaderMap, admin_token: Option<&str>) -> Result<(), Response> {
    let Some(expected_token) = admin_token else {
        return Err(plain_http_error(
            StatusCode::FORBIDDEN,
            "remote admin API is disabled",
        ));
    };
    let token = extract_bearer_token(headers, None)?;
    if token.as_bytes().ct_eq(expected_token.as_bytes()).into() {
        Ok(())
    } else {
        Err(unauthorized_bearer_response(
            "missing or invalid admin bearer token",
            None,
        ))
    }
}

validate_origin runs first and is permissive by design: a request with no Origin header passes unconditionally, which is why curl and a Prometheus scraper both work. When the header is present it must parse as a URL and its host must be exactly localhost, 127.0.0.1, or ::1 after bracket trimming; anything else is 403 origin not allowed. That ordering is pinned by validate_admin_request_rejects_disallowed_origin_before_bearer_auth, a unit test in the same file that asserts a disallowed origin returns 403 with no Authorization header present at all. A cross-origin browser page cannot reach these routes; a server-side client with no origin can.

The bearer comparison is subtle::ConstantTimeEq. Read the guarantee exactly as subtle 2.6.1 documents it: the slice implementation short-circuits when the lengths differ and is content-independent only when they match. Token length is observable; token content is not. extract_bearer_token requires the literal prefix Bearer with one space and a non-empty remainder, and it passes None for the protected-resource metadata, so an admin 401 carries a bare WWW-Authenticate: Bearer challenge rather than the resource-metadata challenge the /mcp route emits.

What the gate does not do is the part to plan around. It has no scopes, no roles, and no read-versus-write split: one token that can read /admin/health can also rotate the authority key and revoke every capability in the process. It is not rate limited, because route_layer attaches rate_limit_mcp_request to the /mcp router before install_admin_routes chains the admin paths on, so the 600-request-per-minute per-IP window covers /mcp and nothing else. And it runs inside the handler body, which means every axum extractor in the signature has already run: a query string that fails to deserialize on the four filtered list routes, or a malformed JSON body on POST /admin/revocations, is rejected by the extractor before validate_admin_request is ever called.

Where the token comes from

build_remote_auth_state resolves the admin token once at startup, and the resolution order is the thing to notice.

let admin_token = if let Some(token) = config.admin_token.as_deref() {
    Some(validated_static_bearer_token(token, "--admin-token")?)
} else if let Some(token) = config.auth_token.as_deref() {
    Some(validated_static_bearer_token(token, "--auth-token")?)
} else {
    return Err(CliError::cli_other_error(
        "bearer-authenticated remote MCP edge requires --admin-token for admin APIs"
            .to_string(),
    ));
};

Three consequences follow. serve_http_async calls this unconditionally before it builds the router, so an edge started with neither flag refuses to start rather than starting with the admin plane closed. Because the call always yields Some, the 403 remote admin API is disabled branch in validate_admin_auth is unreachable through chio mcp serve-http. And omitting --admin-token while supplying --auth-token silently collapses the two roles: the bearer that opens an MCP session becomes the bearer that rotates authority. validated_static_bearer_token rejects a token that is empty, padded with whitespace, or contains a control character, naming the offending flag in the error.

Pass the token through the environment. --admin-token is declared #[arg(long, env = "CHIO_ADMIN_TOKEN", hide_env_values = true)] and its own doc comment says to prefer the environment form so the bearer does not leak through ps or /proc/<pid>/cmdline.

Local store or forwarded call

Ten of the fourteen method-and-path pairs consult control_client(&state) before they touch local state. Seven handlers call it in their own body: the authority rotate, both receipt lists, budgets, the revocation list, the capability revoke, and the per-capability loop inside handle_admin_revoke_session_trust. Two more reach it through load_authority_status, GET /admin/authority and the authority block of GET /admin/health. Both methods on /admin/sessions/{session_id}/trust reach it through load_session_revocation_status, and the POST on that path is already among the seven, which is what makes the total ten and not eleven. With no --control-url the function returns Ok(None) and the handler opens a local SQLite store. With one configured it builds a TrustControlClient and the handler forwards, returning the control plane’s response body rather than composing its own.

The two trust rows are the ones an operator misses. Under --control-url the revoked flag on every capability in a session-trust response comes from the control plane, and the POST writes there instead of to local SQLite. The rest of that body, authContext, lifecycle, and ownership, is composed locally either way, so the route is a mixed read rather than a pure proxy. GET /admin/health is mixed for the same reason and is not in the table: it keeps its session and store blocks local and forwards only the authority block.

control_client fails closed on a half-configuration: --control-url without --control-token is 409 remote trust admin requires --control-token when --control-url is configured. Because both helpers open with control_client(state)?, that refusal reaches all ten, including the two an operator reaches for first. On a half-configured edge GET /admin/health returns the 409 instead of a health payload and the per-session trust drill-down returns it instead of a capability list, so the first and third steps of the runbook triage order below fail identically. That is exactly the case a smoke check should catch.

GET /admin/health

The health route is the one the operations runbook uses for post-restore and post-upgrade smoke checks, and it is the only route that reports the edge’s whole configuration in one object. It runs cleanup_due_sessions(), takes a ledger snapshot, loads authority status, then emits nine top-level keys: ok, server, auth, controlPlane, stores, sessions, authority, federation, and oauth.

Read stores as a configuration report, not a reachability report. Every field is an is_some() on a config path, so receiptsConfigured: true means --receipt-db was supplied, not that the file opens. revocationsConfigured and budgetsConfigured are three-way disjunctions that also count a configured control_url or an attached durable-admission runtime, which is why they can read true on a node with no local database at all. The authority block is the one field that does open its backend, and a failure there is a 500 for the whole route rather than an available: false marker.

auth.adminTokenConfigured is always true in a running edge, for the startup reason above, so it distinguishes nothing. auth.mode is the useful field: static_bearer, jwt_bearer, or introspection_bearer, from remote_auth_mode_label. sessions carries activeCount, terminalCount, and all four lifecycle-policy timings, which are set from the CHIO_MCP_SESSION_* environment variables and are the fastest way to confirm a tuning change took effect. The lifecycle model behind those numbers is Session Lifecycle.

Reading and rotating the issuing key

load_authority_status resolves three backends in a fixed order: the control plane if --control-url is set, then --authority-db as backend sqlite, then --authority-seed-file as backend seed_file. Which one answers changes the key set, and the JSON says so.

appliesToFutureSessionsOnly is true on every branch, and it is a literal description of the rotation contract rather than a configurable flag. mcp_serve_http_admin_authority_rotation_only_affects_future_sessions proves it end to end: it opens a session, records the issuer key on its capabilities, rotates, asserts the new key differs, opens a second session whose capabilities all carry the new key, and then re-reads the first session and finds every capability still carrying the old one, at the same count. Rotation issues forward. It revokes nothing.

Under the SQLite backend the key is shared state, not process state. mcp_serve_http_shared_authority_rotation_propagates_across_nodes runs two edges against one --authority-db file, rotates through node A, and reads the new publicKey and the incremented generation from node B’s GET /admin/authority without restarting it. Custody and the multi-node rules around it are Authority & Rotation.

The two receipt lists

The tool route takes four optional filters and a limit; the child route takes five and a limit. All are snake_case and all are #[serde(default)]. Tool receipts filter on capability_id, tool_server, tool_name, and decision; child receipts on session_id, parent_request_id, request_id, operation_kind, and terminal_state. Neither struct carries deny_unknown_fields, so a misspelled key is dropped rather than rejected and the request returns an unfiltered page: toolName=echo_json applies no filter at all.

admin_list_limit is the whole of pagination.

fn admin_list_limit(requested: Option<usize>) -> usize {
    requested
        .unwrap_or(DEFAULT_ADMIN_LIST_LIMIT)
        .clamp(1, MAX_ADMIN_LIST_LIMIT)
}

DEFAULT_ADMIN_LIST_LIMIT is 50 and MAX_ADMIN_LIST_LIMIT is 200, so limit=0 becomes 1 and limit=100000 becomes 200. There is no cursor and no offset. The underlying SQL is ORDER BY seq DESC LIMIT ?, so these routes read the newest N rows and there is no way to walk past them. Bulk export is a different path entirely; see Receipts & Audit.

Both handlers construct ReceiptReadContext::admin_service() and pass it to list_tool_receipts_with_context or its child equivalent. That context is boundary: AdminAll, source: AdminService, include_null_tenant: false, and the store method uses it as a gate rather than a filter: require_admin_list_context accepts AdminAll and rejects a tenant-scoped context, then delegates to a query whose SQL carries no tenant predicate at all. These routes are cross-tenant reads. The boundary type exists so a tenant-scoped caller cannot reach them, not so an admin caller sees less.

mcp_serve_http_admin_receipt_queries_return_tool_and_child_receipts drives both. It calls one tool and one sampling-backed tool through a live session, then asserts backend: "sqlite", exactly one tool receipt matching tool_name=echo_json with decision.verdict == "allow", and exactly one child receipt matching operation_kind=create_message with a string parent_request_id. Note the casing: the envelope keys are camelCase, and the receipt objects inside receipts keep their own snake_case field names.

Budgets and revocations

GET /admin/budgets resolves its store from the durable-admission runtime first and --budget-db second, calls BudgetStore::list_usages, and then narrows what it got. The trait returns a BudgetUsageRecord carrying capability_id, grant_index, invocation_count, updated_at, seq, total_cost_exposed, and total_cost_realized_spend. The handler emits the first four and drops the last three.

"usages": usages.into_iter().map(|usage| json!({
    "capabilityId": usage.capability_id,
    "grantIndex": usage.grant_index,
    "invocationCount": usage.invocation_count,
    "updatedAt": usage.updated_at,
})).collect::<Vec<_>>(),

So a locally-backed GET /admin/budgets answers how many times a grant was used and never how much it cost. The proxied path does not have that gap: the control plane’s BudgetListResponse serializes totalExposureCharged, an optional totalRealizedSpend, and an optional seq alongside the other four. If a monetary read is what you need from a local edge, take it from the receipts or from the store directly rather than from this route. The lifecycle those numbers come out of is Budget Store.

GET /admin/revocations takes capability_id and limit. The revoked field is computed by a second store call and only when a capability_id was supplied; unfiltered it is null locally and absent when proxied. POST /admin/revocations takes a JSON body with one required snake_case key, capability_id, and returns newlyRevoked, which distinguishes a first revocation from a repeat. The store semantics are Revocation Store.

mcp_serve_http_admin_revocation_queries_and_direct_revoke_work walks the sequence against a live edge: read a capability id off the session trust route, confirm revoked: false and count: 0, post the revoke and get newlyRevoked: true, re-read and get revoked: true with one entry, then confirm the entry also appears in an unfiltered list. A companion test, mcp_serve_http_admin_revocation_denies_future_calls_for_session, carries it through to effect: the next tool call on that session comes back as a tool error whose text contains revoked.

The four session routes

GET /admin/sessions is the fleet view and the per-session routes are the drill-down. All four run cleanup_due_sessions() before they answer, either directly or through resolve_session_entry. Reading the admin plane therefore advances lifecycle state: a GET can expire an idle session, delete a drained one past its deadline, and purge old tombstones as a side effect of being called. The background reaper does the same work on an interval; the admin routes just do it sooner.

The three per-session routes resolve the same way. An unknown id is 404 unknown MCP session. A known id resolves to either an Active session or a Terminal tombstone, and every handler accepts both, which is what lets an operator inspect a session that is already gone. The tombstone half survives a restart when --session-db is set.

The two lists report capabilities differently, and a client that reads both needs to know which. /admin/sessions serializes each record through serialize_session_diagnostic_record, whose capability objects are id, issuerPublicKey, subjectPublicKey. The trust route builds its own array through load_session_revocation_status, whose objects are capabilityId, issuerPublicKey, subjectPublicKey, revoked. Different key for the id, and only the trust route pays the per-capability lookup that produces revoked, one is_revoked call against the local store or, under --control-url, one GET /v1/revocations per capability.

authContext is a SessionAuthContext: transport, method, and an optional origin. The method object is internally tagged on kind with snake_case variant names, and its own fields stay snake_case except federatedClaims and enterpriseIdentity, which carry explicit renames. Expect token_fingerprint, not tokenFingerprint. ownership is fully camelCase: hostedIsolation, hostedIdentityProfile, requestStreamOwner, notificationStreamOwner, notificationDelivery, requestStreamActive, notificationStreamAttached, and requestOwnership.

Drain, shutdown, and the terminal case

Drain and shutdown are different operations with different effects. mark_draining calls begin_draining, which deletes the resumable record, sets state to Draining, and stamps a deadline at drain_grace_millis from now. The session stays in the active map and no tombstone is written yet; the reaper deletes it when the deadline passes. mark_closed goes straight to the terminal state Closed, writes the tombstone, and the handler then calls remove_active itself.

mcp_serve_http_admin_drain_shutdown_and_delete_have_distinct_terminal_states pins all three exits on one process. Drain returns lifecycle.state == "draining", the next call on that session is 409, and the session settles to deleted with reconnect.resumable == false, after which calls are 410. A client DELETE /mcp is 204 and lands on deleted directly. Shutdown returns closed and its next call is 410. All three then appear under terminalCount.

handle_admin_revoke_session_trust iterates the session’s capabilities, revokes each one, counts only the ones that were newly revoked into newlyRevokedCount, then re-reads status for the response. The dispatch decision is made inside the loop rather than once above it: every iteration calls control_client(&state) again, so a configured control plane takes one revoke_capability call per capability and the local store is never opened. Because the per-capability revoke result is unwrap_or(false), a failure on either backing and an already-revoked capability contribute the same zero to the count. The re-read is what tells them apart: a capability that failed to revoke comes back revoked: false. Revoking a terminal session’s capabilities still works, which matters because a capability outlives the session that received it.

GET /admin/metrics

The metrics route composes two render sources into one Prometheus text body and sets Content-Type: text/plain; version=0.0.4.

let body = chio_metrics_spec::runtime::compose_metrics_body(&[
    &chio_kernel::render_guard_metrics_prometheus,
    &alert_pack,
]);

render_guard_metrics_prometheus emits the guard families, the OTEL-drop families, SIGNING_QUEUE_BLOCK, SETTLEMENT_UNRESOLVED, AMBIGUOUS_DISPATCH_RETAINED_HOLD, and the receipt watchdog gauges. render_alert_pack_families adds eight more: FAIL_OPEN_SUSPECTED, DISPATCH_FAILURE, CAPABILITY_REVOCATION_LAG, DLQ_DEPTH, SOC_EXPORT_TOTAL, SOC_EXPORT_LAG, ALERT_DISPATCH_TOTAL, and ALERT_DISPATCH_LATENCY.

Every family in chio_metrics_spec::runtime::families is a process-global static. Most are plain const-constructed values; SETTLEMENT_UNRESOLVED and AMBIGUOUS_DISPATCH_RETAINED_HOLD are LazyLock so they can preregister their label sets on first touch. Neither form is scoped narrower than the process. The edge runs one kernel per session, and this route reports one set of counters across all of them: there is no per-session breakdown and no session label. The route takes no state from RemoteAppState beyond the admin token; the handler binds State(state) only to reach state.admin_token. What each family means, and the collector and dashboard wiring around it, is Node Observability.

The scrape is admin-gated like everything else, so a Prometheus job needs the bearer in a bearer_token_file. It sends no Origin, so the origin check passes.

Denial and failure

Admin errors are bare text. plain_http_error is (status, message.to_string()).into_response(): no JSON envelope, no urn:chio:error: code, no registry entry. The status and the exact string are the whole contract, and the strings below are the literals in the source.

The distinction between 403 and 401 tells you which check fired. 403 means the origin was rejected and the token was never examined. 401 means the origin passed and the token did not.

Both 401 bodies against a running edge, first with the header absent and then with a token that fails the comparison. The two strings differ, so the body tells you whether any bearer was presented at all:

$ curl -si http://127.0.0.1:8931/admin/health
HTTP/1.1 401 Unauthorized
content-type: text/plain; charset=utf-8
www-authenticate: Bearer
content-length: 31

missing or invalid bearer token

$ curl -si -H "Authorization: Bearer not-the-admin-token" http://127.0.0.1:8931/admin/health
HTTP/1.1 401 Unauthorized
content-type: text/plain; charset=utf-8
www-authenticate: Bearer
content-length: 37

missing or invalid admin bearer token

Running it

Start the edge with an explicit admin token and three persistent stores, then confirm two routes. The environment-variable form of each token flag exists so the bearer stays out of the process table. The session database is also the durable admission database, and that authority owns revocation and budget state for every participant, so passing --revocation-db or --budget-db alongside it is refused at startup with the durable admission authority owns revocation and budget state.

$ chio mcp serve-http \
  --policy examples/policies/canonical-hushspec.yaml \
  --server-id demo-server \
  --listen 127.0.0.1:8931 \
  --auth-token "$CHIO_EDGE_TOKEN" \
  --admin-token "$CHIO_ADMIN_TOKEN" \
  --receipt-db /var/lib/chio/edge-receipts.sqlite3 \
  --authority-db /var/lib/chio/edge-authority.sqlite3 \
  --session-db /var/lib/chio/edge-sessions.sqlite3 \
  -- \
  python3 tests/conformance/fixtures/mcp_core/mock_mcp_server.py
remote MCP edge listening on http://127.0.0.1:8931/mcp

On a freshly started edge with no client connected, /admin/health reports the configuration the process is running under and /admin/sessions reports two empty maps. Note revocationsConfigured: true with no --revocation-db on the command line: the durable admission authority is the backing.

$ curl -s -H "Authorization: Bearer $CHIO_ADMIN_TOKEN" \
  http://127.0.0.1:8931/admin/health | jq '{ok, server, sessions, stores}'
{
  "ok": true,
  "server": {
    "serverId": "demo-server",
    "serverName": "demo-server",
    "serverVersion": "0.1.0",
    "sharedHostedOwner": false,
    "sharedHostedOwnerStats": null
  },
  "sessions": {
    "activeCount": 0,
    "drainGraceMillis": 5000,
    "idleExpiryMillis": 900000,
    "reaperIntervalMillis": 250,
    "terminalCount": 0,
    "tombstoneRetentionMillis": 1800000
  },
  "stores": {
    "authorityDbConfigured": true,
    "authoritySeedConfigured": false,
    "budgetsConfigured": true,
    "receiptsConfigured": true,
    "revocationsConfigured": true,
    "sessionTombstonesConfigured": true
  }
}

$ curl -s -H "Authorization: Bearer $CHIO_ADMIN_TOKEN" \
  http://127.0.0.1:8931/admin/sessions | jq
{
  "activeCount": 0,
  "activeSessions": [],
  "configured": true,
  "drainGraceMillis": 5000,
  "idleExpiryMillis": 900000,
  "reaperIntervalMillis": 250,
  "terminalCount": 0,
  "terminalSessions": []
}

Those two commands are the post-restore and post-upgrade smoke check, and the runbook’s triage table points lifecycle and auth incidents at /admin/health, /admin/sessions, and /admin/sessions/{session_id}/trust in that order. Follow the order: the first tells you what the process thinks it is configured with, the second tells you how many sessions exist and in what state, the third tells you why one principal is being denied.

Bind the listener to loopback and reach it through the supervisor host or an authenticated reverse proxy. There is no separate admin listener and no separate port: the admin paths share the socket that serves /mcp, so exposing the MCP endpoint publicly exposes the admin paths to the same network. The shipped systemd unit under docs/release/systemd/ sets KillSignal=SIGTERM and TimeoutStopSec=35s so the 25-second drain finishes before SIGKILL. Draining individual sessions ahead of a stop is what POST /admin/sessions/{session_id}/drain is for; there is no route that drains all of them at once, and no chio subcommand for any route on this page.

Guarantees and limits

Next Steps

• Remote MCP Edge · the other half of this process: the /mcp route, the three bearer modes, and the rate limiter these paths sit outside
• Session Lifecycle · the state machine drain and shutdown drive, and the tombstones that outlive both
• Node Observability · what the families behind /admin/metrics mean, and the collector and dashboards around them
• Trust Control Plane · the /v1 service ten of these fourteen pairs forward to, and its own token
• Authority & Rotation · what POST /admin/authority means once more than one node shares the key