Trust Control Plane

The end state: one or three chio trust serve nodes behind TLS, serving shared state to every kernel started with --control-url, with each route admitting only the principals its class allows.

Each kernel still admits its own calls in process; the service supplies the shared state those admissions read. Start with one durable node, then add two peers when a node’s loss must leave writes available.

Kernels call the service; the service signs and keeps the record

One binary, chio trust serve, runs the chio-control-plane crate: the capability authority, the revocation and receipt stores, the budget plane, and peer coordination.

pub fn configure_revocation_store(
    kernel: &mut ChioKernel,
    revocation_db_path: Option<&Path>,
    control_url: Option<&str>,
    control_token: Option<&str>,
) -> Result<(), CliError> {
    match (revocation_db_path, control_url) {
        (Some(_), Some(_)) => {
            return Err(CliError::cli_other_error(
                "use either --revocation-db or --control-url for revocation state, not both"
                    .to_string(),
            ));
        }
        (Some(path), None) => {
            kernel.set_revocation_store(Box::new(chio_store_sqlite::SqliteRevocationStore::open(
                path,
            )?));
        }
        (None, Some(url)) => {
            let token = require_control_token(control_token)?;
            kernel.set_revocation_store(
                trust_control::service_runtime::remote_stores::build_remote_revocation_store(
                    url, token,
                )?,
            );
        }
        (None, None) => {}
    }
    Ok(())
}

The authority holds the signing key; each kernel verifies with public keys it caches for 2 seconds. Revocation is looked up on every admission, and a failed lookup refuses the call. A receipt append that misses its 5,000 ms deadline fails closed, and an unanswered budget hold refuses the call before dispatch.

1. Start one durable control service

Install the Chio CLI, then create a private working directory and the admin service token once. Keep both across restarts.

umask 077
mkdir -p chio-control/state chio-control/secrets
cd chio-control
test -s secrets/service-token.txt || openssl rand -hex 32 > secrets/service-token.txt

Start the service in the foreground. CHIO_TRUST_SERVICE_TOKEN supplies the admin service token, the same value as --service-token, without putting it in the process arguments.

export CHIO_TRUST_SERVICE_TOKEN="$(cat secrets/service-token.txt)"
chio trust serve \
  --listen 127.0.0.1:8940 \
  --authority-db state/authority.sqlite \
  --revocation-db state/revocations.sqlite \
  --receipt-db state/receipts.sqlite \
  --budget-db state/budgets.sqlite

With neither --service-token nor CHIO_TRUST_SERVICE_TOKEN set, argument parsing exits 2 before opening a socket. The loopback listener suits local development; on a network, terminate TLS in front of every node and of the hosted MCP and auth plane, and keep peer traffic on private connectivity.

2. Check that the node answers

In a second terminal, read GET /health. It needs no credential and carries a cluster object beside the store configuration.

curl -s http://127.0.0.1:8940/health | jq '{ok, clustered, leaderUrl, stores, cluster}'

{
  "ok": true,
  "clustered": false,
  "leaderUrl": null,
  "stores": {
    "budgetsConfigured": true,
    "receiptsConfigured": true,
    "revocationsConfigured": true,
    "verifierChallengesConfigured": false
  },
  "cluster": {
    "electionTerm": 0,
    "hasQuorum": false,
    "healthyPeers": 0,
    "lastErrorCount": 0,
    "leaderUrl": null,
    "partitionedPeers": 0,
    "peerCount": 0,
    "quorumSize": 1,
    "reachableNodes": 1,
    "role": "standalone",
    "selfUrl": null,
    "unhealthyPeers": 0,
    "unknownPeers": 0
  }
}

You should see HTTP 200, ok: true, and the role standalone. A node with no peers reports hasQuorum: false, which is expected here. role, hasQuorum, quorumSize, reachableNodes, electionTerm, and leaderUrl together answer whether the node can take a write.

3. Add two peers for failover

Give every peer its own advertised URL, its own state directory, and the other two URLs as peers. Run this on kernel-a.larkfield.example, and the matching command on kernel-b and kernel-c.

# Every setting is a CLI flag; there is no config-file mode.
# CHIO_TRUST_SERVICE_TOKEN carries the required service token.
export CHIO_TRUST_SERVICE_TOKEN="$(cat /run/secrets/chio_service)"
chio trust serve \
    --listen 0.0.0.0:8940 \
    --advertise-url https://kernel-a.larkfield.example:8940 \
    --peer-url https://kernel-b.larkfield.example:8940 \
    --peer-url https://kernel-c.larkfield.example:8940 \
    --cluster-sync-interval-ms 500 \
    --authority-db /var/lib/chio/authority.sqlite \
    --revocation-db /var/lib/chio/revocations.sqlite \
    --receipt-db /var/lib/chio/receipts.sqlite \
    --budget-db /var/lib/chio/budgets.sqlite

These hostnames need working DNS and TLS termination; on a private network, add --allow-local-peer-urls to admit private addresses. Every node must list the same peer set, because each derives its quorum from its own list. Optional file-backed registries turn on federation and identity features when their flags are passed: --scim-lifecycle-file for SCIM provisioning from an external IdP, --enterprise-providers-file, --federation-policies-file, --verifier-policies-file, and the passport and certification registry files.

4. Point every kernel at every node

Pass the whole cluster to --control-url as a comma-separated list, with the admin service token in --control-token, since kernels append receipts and write budgets. Followers forward writes to the leader, so any listed node is a valid entry point. An embedded Rust host attaches the same four remote adapters before it issues capabilities or accepts tool calls.

use chio_control_plane::trust_control::service_runtime::{
    budget::build_remote_budget_store,
    remote_authority::build_remote_capability_authority,
    remote_stores::{build_remote_receipt_store, build_remote_revocation_store},
};

// urls and service_token come from your host configuration.
kernel.set_receipt_store(build_remote_receipt_store(urls, service_token)?)?;
kernel.set_revocation_store(build_remote_revocation_store(urls, service_token)?);
kernel.set_budget_store(build_remote_budget_store(urls, service_token)?);
kernel.set_capability_authority(build_remote_capability_authority(urls, service_token)?);

A kernel uses remote stores or local ones (--receipt-db, --revocation-db, --authority-*), never both. Embedded Node covers host construction. The hosted MCP HTTP edge runs on the joint admission authority, a separate configuration from peer URLs; deploy it with Remote MCP Edge.

5. Confirm that the nodes agree on a leader

Ask a follower for its view.

curl -s https://kernel-b.larkfield.example:8940/health \
  | jq '.cluster | {role, hasQuorum, quorumSize, reachableNodes, leaderUrl, electionTerm}'

{
  "role": "follower",
  "hasQuorum": true,
  "quorumSize": 2,
  "reachableNodes": 3,
  "leaderUrl": "https://kernel-a.larkfield.example:8940",
  "electionTerm": 1
}

You should see every node name the same leaderUrl. A node leads when quorum holds and its URL sorts lowest among itself and its reachable, unpartitioned peers contacted within the lease TTL; at the default 500 ms interval a peer quiet for more than a second stops counting. Below quorum the role is candidate and the leader is null. Two quorate views can still differ when their eligible sets differ, and a load balancer in front of the nodes leaves that contract unchanged. Leader & Failover covers the rule and the term fence.

Each store replicates under its own merge rule

Every node owns four SQLite stores and repairs them from its peers every --cluster-sync-interval-ms, 500 by default, even after a forwarded write succeeds. Reads answer from the responding node’s stores.

The signing seed stays on the node that holds it; Authority & Rotation covers takeover. Look a receipt up across nodes by its signed ID, since each node assigns its own positions and checkpoints. A clustered node stamps budgetAuthority.guaranteeLevel as advisory_posthoc, below the single-node level; a global spending ceiling needs an authority that serializes admitted spend, as Budgets Across Nodes explains. Coordination is leader-local inside one operator’s trust domain.

Each route admits only the principals its class allows

The service resolves every authenticated request to one principal and checks it against the route. The admin service token, set with --service-token or CHIO_TRUST_SERVICE_TOKEN, is the admin principal and the only credential that writes. Each --tenant-read-token tenant_id=token adds a tenant principal confined to one tenant's receipts. Peers authenticate as named nodes, and liveness is public.

A tenant read token on an operator route gets 403, and so does a receipt point-load by ID, which is not tenant-scoped. The service reads the bearer credential only from the Authorization header. Peer routes never accept it as a bearer: the token enters the peer digest and never travels as a header.

curl -si -H "Authorization: Bearer $CHIO_TRUST_SERVICE_TOKEN" \
    http://127.0.0.1:8940/v1/internal/cluster/status

HTTP/1.1 401 Unauthorized
content-type: application/json
www-authenticate: chio.cluster.peer.v1
content-security-policy: default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; connect-src 'self'; img-src 'self' data:
content-length: 58

{"error":"missing or invalid cluster peer authentication"}

You should see the admin token refused on the peer route with 401. An unauthenticated GET /v1/receipts/query gets 401 with www-authenticate: Bearer and missing or invalid control bearer token. A holder of the service token can authenticate as any listed peer, so store it and the authority signing seed in separate secret stores and rotate them on independent schedules. When hosted OAuth is enabled, give it a signing seed of its own.

The container image embeds the trust-control dashboard, the operator web client for receipts, approvals, issuance, and authority-tree and pool views. It calls the same routes under the same principal check, so a tenant credential sees only its receipts and approval routes admit only the admin principal. Receipt Dashboard covers it.

Rotation adds a key, and signatures under retained keys keep verifying

POST /v1/authority rotates the SQLite issuer on the leader. Before rotation the authority is at generation 1 with one trusted key.

# Rotate the authority signing seed on the current leader.
curl -s -X POST -H "Authorization: Bearer $CHIO_TRUST_SERVICE_TOKEN" \
    http://127.0.0.1:8940/v1/authority | jq

{
  "appliesToFutureSessionsOnly": true,
  "backend": "sqlite",
  "configured": true,
  "generation": 2,
  "publicKey": "beb4438776406922a06f9477e07e49f63d95ac5e84a9aede414984c9b1de3639",
  "rotatedAt": 1785603453,
  "trustedPublicKeys": [
    "8d7ddbdce2368deb5143de554fb574fea652fe3e5610acdb2b1be110083fe485",
    "beb4438776406922a06f9477e07e49f63d95ac5e84a9aede414984c9b1de3639"
  ]
}

curl -s -H "Authorization: Bearer $CHIO_TRUST_SERVICE_TOKEN" \
    http://127.0.0.1:8940/v1/authority | jq

{
  "configured": true,
  "backend": "sqlite",
  "publicKey": "8d7ddbdce2368deb5143de554fb574fea652fe3e5610acdb2b1be110083fe485",
  "generation": 1,
  "rotatedAt": 1785603453,
  "appliesToFutureSessionsOnly": true,
  "trustedPublicKeys": [
    "8d7ddbdce2368deb5143de554fb574fea652fe3e5610acdb2b1be110083fe485"
  ]
}

You should see publicKey move to the new key while trustedPublicKeys grows to two entries. A later GET /v1/authority returns the same generation 2 body.

1. Signatures under retained keys still verify; a session’s eligibility still depends on expiry, revocation, and session policy.
2. New capabilities are signed under the new seed.
3. Trusted-key history replicates to every follower in the authority snapshot, and remote authority clients refresh it on a short TTL. Check a client’s observed generation before relying on newly issued credentials.

Failures show as 503s, uncertain writes, or stale kernel views

Covered writes answer 503

The node is below quorum or its lease has passed, so leader-forwarded handlers refuse before their local write. Reads continue from the node’s stores and can lag its peers. Restore peer connectivity, then branch on the status code; Quorum refusals lists the messages, and Leader & Failover adds the authority variants. Replica repair and other handlers keep their own behavior.

A request failed during failover

A client with several control URLs tries another endpoint, and the retry still depends on quorum, the operation’s state, and signing custody. Treat the first attempt as unknown and keep its request and receipt identities. Receipts append idempotently by receipt ID, so a retry is safe. A new leader’s budget counts never decrease, yet it can lack events it has not observed, so read the named admission and witness state in the budget lifecycle before retrying. Check a required revocation on each dependent service, since a lagging peer can still lack it.

A kernel still sees an old key or revocation

A change reaches another node through replication and reaches a kernel through its adapter’s refresh. Wait for both, then confirm the kernel’s observed authority generation or revocation lookup.

Next: follow a write through the cluster