Trust Control Plane

The service is one binary, chio trust serve, over the chio-control-plane crate. That crate owns the capability authority, the revocation and receipt stores, the budget plane, and the cluster consensus this page configures.

Why run a cluster

A single trust-control node is fine for development. In production, three invariants make a cluster non-optional:

• Cross-node receipts: one edge kernel writes a receipt; another edge kernel must be able to query it.
• Cross-node revocation: revoking a capability through one node must be enforced by every other node on the next request.
• Shared budget accounting: invocation budgets must exhaust consistently across nodes, not independently per-process.

The control plane exposes all three through one shared HTTP API, so edge kernels stay thin and stateless with respect to trust.

Topology

Run two or more trust-control nodes in front of a fleet of chio mcp serve-http edges. Every control node owns local durable SQLite state. Writes route to the current leader, follower nodes replicate on a short interval, and edges keep a multi-endpoint client list.

The leader rule

compute_cluster_consensus_locked recomputes the leader on every status read. It is a deterministic pick over itself and the peers one node can currently see, and the pick is gated on a quorum of them:

pub(crate) fn compute_cluster_consensus_locked(
    cluster: &mut ClusterRuntimeState,
) -> ClusterConsensusView {
    let now = unix_timestamp_now();
    let lease_ttl_secs = Duration::from_millis(cluster.lease_ttl_ms).as_secs().max(1);
    let quorum_size = cluster.peers.len().div_ceil(2) + 1;
    let mut candidates = vec![cluster.self_url.clone()];
    for (peer_url, peer_state) in &cluster.peers {
        let contact_is_fresh = peer_state
            .last_contact_at
            .is_some_and(|last_contact_at| now <= last_contact_at.saturating_add(lease_ttl_secs));
        if peer_state.health.is_reachable() && !peer_state.partitioned && contact_is_fresh {
            candidates.push(peer_url.clone());
        }
    }
    candidates.sort();
    let reachable_nodes = candidates.len();
    let has_quorum = reachable_nodes >= quorum_size;
    let leader_url = if has_quorum {
        candidates.first().cloned()
    } else {
        None
    };
    if cluster.last_leader_url != leader_url {
        cluster.election_term = cluster.election_term.saturating_add(1);
        cluster.last_leader_url = leader_url.clone();
        cluster.term_started_at = leader_url.as_ref().map(|_| now);
    }
    cluster.lease_expires_at = if has_quorum {
        Some(now.saturating_add(lease_ttl_secs))
    } else {
        None
    };
    if !has_quorum {
        cluster.term_started_at = None;
    }
    let role = if !has_quorum {
        "candidate"
    } else if leader_url.as_deref() == Some(cluster.self_url.as_str()) {
        "leader"
    } else {
        "follower"
    };
    ClusterConsensusView {
        self_url: cluster.self_url.clone(),
        leader_url,
        role,
        has_quorum,
        quorum_size,
        reachable_nodes,
        election_term: cluster.election_term,
    }
}

Read the candidate filter first. A node always counts itself: candidates starts with self_url, the normalized --advertise-url, and nothing tests local health. A peer joins the set only when all three conditions hold at once, which is stricter than health alone: it is reachable, it is not marked partitioned, and it was last contacted inside the lease TTL. That TTL is three times the sync interval clamped into a fixed band and then floored at one second here, so on the default interval a peer quiet for more than a second stops counting.

quorum_size is a majority of the configured peer list, computed from this node's own --peer-url entries. Two peers means a quorum of two, so a three-node cluster keeps writing through one quiet peer and stops when both go quiet. One peer also means a quorum of two, so in a two-node cluster either node going quiet stops writes on both. The sort is byte-wise over URL strings rather than numeric, so https://ctl-a:10001 sorts ahead of https://ctl-a:9001.

Below quorum there is no leader at all: leader_url is None and the lease stamp and the term start are cleared. That is the third of the three roles a clustered node reports.

election_term advances on any change to the computed leader, including a change to no leader at all, and it is persisted beside the authority state so a restart does not reuse a stale term. The term is a fence on one path only, authority mutation. A node with no peers configured never enters this function and reports the fourth role, standalone, in GET /health.

The full mechanism, the four helpers that check it, and the term fence are in Leader & Failover.

Replication model

Replication is per-store. Every kind of state has an idempotent replication contract so repair syncs converge even after transient peer failures.

Periodic repair sync runs even after successful write forwarding, so missed updates eventually converge. The default interval is --cluster-sync-interval-ms 500.

Budget-state durability

Invocation budgets live in a pluggable BudgetStore. In distributed mode the store is backed by the control plane, so budget increments route to the current leader and replicate to followers as monotonic usage records.

• Local mode: in-memory or local SQLite (SqliteBudgetStore). Budget exhaustion persists across process restart.
• Distributed mode: remote BudgetStore backed by the control plane, so every edge counts against one store instead of its own. A clustered node stamps budgetAuthority.guaranteeLevel as advisory_posthoc, which is weaker than the single-node level, not stronger. What clustering buys on this plane, and what it costs, is Budgets & Quorum.
• Follower merge rule: max observed invocation count per (capability_id, grant_index). Budgets can only ever count up.

Store responsibilities

Capability authority

The capability authority issues ed25519 signatures over capability tokens. The control plane owns the signing seed and the rotated trusted-key history. Edge kernels verify capability signatures against the full trusted set fetched from the control plane on a short TTL.

• Issue capabilities: POST /v1/capabilities/issue.
• Read authority state: GET /v1/authority.
• Rotate authority: POST /v1/authority.

Revocation store

Revocations are idempotent records keyed by capability ID. Once a capability is revoked on any node, every other node observes the revocation on the next repair sync or the next cross-node request. The edge kernel re-checks revocation status at the start of every tool call. A request cannot use a revoked capability after the edge kernel has received the revocation.

• Query: GET /v1/revocations.
• Mutate: POST /v1/revocations.

Receipt store

Tool receipts and child-request receipts are append-only. A receipt emitted on one edge is queryable from another edge as soon as the repair sync has run. Because the store is idempotent on receipt ID, replaying receipts during replication is safe.

• Append tool receipts: POST /v1/receipts/tools.
• Query tool receipts: GET /v1/receipts/tools or GET /v1/receipts/query.
• Append child receipts: POST /v1/receipts/children.
• Query child receipts: GET /v1/receipts/children.

Failover and quorum loss

While a quorum still holds, failover is the pick recomputing. The smallest remaining candidate URL becomes the leader on the next status read and the election term advances. Edge clients keep a multi-endpoint list and retry writes across the set, so a dead first URL is transparent.

Below quorum there is nothing to fail over to. forward_post_to_leader tests quorum before it looks for a leader, so a node in a minority partition refuses every mutating request rather than electing itself:

Reads are unaffected: any node answers them from its own stores. Branch on the status code rather than the sentence; the full set of refusals, including the authority-write variants, is tabled in Leader & Failover. Once a quorum returns, the four stores pick up where they left off:

• Revocations: the new leader already has the last-known revocation set from repair sync. No revocation is lost.
• Receipts: in-flight writes may be retried. Idempotent append on receipt ID makes this safe.
• Budgets: max-observed merge means the new leader picks up the highest usage count seen anywhere. Monotonic guarantee preserved.
• Authority: the trusted-key history is the union across nodes, so existing capabilities keep verifying under the new leader without restart.

Running a node

Single-node (development)

# --service-token is required. Prefer CHIO_TRUST_SERVICE_TOKEN so the
# bearer is not visible via ps / proc.
$ export CHIO_TRUST_SERVICE_TOKEN="$(cat ./secrets/service-token.txt)"
$ chio trust serve \
    --listen 127.0.0.1:8940 \
    --authority-db ./state/authority.sqlite \
    --revocation-db ./state/revocations.sqlite \
    --receipt-db ./state/receipts.sqlite \
    --budget-db ./state/budgets.sqlite
Chio trust control service listening on http://127.0.0.1:8940

Required means required. With neither the flag nor the environment variable set, argument parsing refuses before any socket is opened and the process exits 2:

$ chio trust serve --listen 127.0.0.1:8940 --receipt-db ./state/receipts.sqlite
error: the following required arguments were not provided:
  --service-token <SERVICE_TOKEN>

Usage: chio trust serve --service-token <SERVICE_TOKEN> --listen <LISTEN> --receipt-db <RECEIPT_DB>

For more information, try '--help'.

Three-node cluster

# Every setting is a CLI flag; there is no config-file mode.
# CHIO_TRUST_SERVICE_TOKEN carries the required service token.
$ export CHIO_TRUST_SERVICE_TOKEN="$(cat /run/secrets/chio_service)"
$ chio trust serve \
    --listen 0.0.0.0:8940 \
    --advertise-url https://ctl-a.chio.internal:8940 \
    --peer-url https://ctl-b.chio.internal:8940 \
    --peer-url https://ctl-c.chio.internal:8940 \
    --cluster-sync-interval-ms 500 \
    --authority-db /var/lib/chio/authority.sqlite \
    --revocation-db /var/lib/chio/revocations.sqlite \
    --receipt-db /var/lib/chio/receipts.sqlite \
    --budget-db /var/lib/chio/budgets.sqlite

Run the same command on the other two hosts, each with its own --advertise-url and listing the other two as --peer-url values. Each node then computes the same pick from its own view: the smallest URL among itself and the peers it can currently see, once two of the three are reachable.

Beyond the cluster and store flags, chio trust serve accepts a set of optional file-backed registries that light up federation and identity features. Each is off unless its flag is passed: --scim-lifecycle-file (SCIM provisioning and deprovisioning for an external IdP), --enterprise-providers-file, --federation-policies-file, --verifier-policies-file, and the passport and certification registry files.

Edge kernel wiring

# --control-token / CHIO_CONTROL_TOKEN carries the trust-control
# service token as a value, not a file path.
$ export CHIO_CONTROL_TOKEN="$(cat /run/secrets/chio_service)"
$ chio mcp serve-http \
    --control-url https://ctl-a.chio.internal:8940,https://ctl-b.chio.internal:8940,https://ctl-c.chio.internal:8940

Health and status endpoints

Every node exposes health and status for operators and load balancers:

GET /health needs no credential and is the operator’s read. It carries a cluster object alongside the store configuration, so one unauthenticated call answers both "can this node serve?" and "what does it think the cluster is?". That object is the consensus view rendered as JSON: role, hasQuorum, quorumSize, reachableNodes, electionTerm, and leaderUrl are the six fields that answer whether this node can currently take a write. On a single node with no peers configured, the role is the standalone one and hasQuorum is false without that meaning anything is wrong:

$ curl -s http://127.0.0.1:8940/health | jq '{ok, clustered, leaderUrl, stores, cluster}'
{
  "ok": true,
  "clustered": false,
  "leaderUrl": null,
  "stores": {
    "budgetsConfigured": true,
    "receiptsConfigured": true,
    "revocationsConfigured": true,
    "verifierChallengesConfigured": false
  },
  "cluster": {
    "electionTerm": 0,
    "hasQuorum": false,
    "healthyPeers": 0,
    "lastErrorCount": 0,
    "leaderUrl": null,
    "partitionedPeers": 0,
    "peerCount": 0,
    "quorumSize": 1,
    "reachableNodes": 1,
    "role": "standalone",
    "selfUrl": null,
    "unhealthyPeers": 0,
    "unknownPeers": 0
  }
}

/v1/internal/cluster/status is the peer’s read of the same state, and the service token does not open it. It authenticates under its own scheme, named in the WWW-Authenticate header: a per-node credential carrying node id, election term, and an issued-at stamp that must fall inside the allowed skew window (trust_control/report_validation.rs).

$ curl -si -H "Authorization: Bearer $CHIO_TRUST_SERVICE_TOKEN" \
    http://127.0.0.1:8940/v1/internal/cluster/status
HTTP/1.1 401 Unauthorized
content-type: application/json
www-authenticate: chio.cluster.peer.v1
content-security-policy: default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; connect-src 'self'; img-src 'self' data:
content-length: 58

{"error":"missing or invalid cluster peer authentication"}

The operator-facing routes take the service token and refuse without it. GET /v1/receipts/query with no bearer:

$ curl -si 'http://127.0.0.1:8940/v1/receipts/query?limit=1'
HTTP/1.1 401 Unauthorized
content-type: application/json
www-authenticate: Bearer
content-security-policy: default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; connect-src 'self'; img-src 'self' data:
content-length: 51

{"error":"missing or invalid control bearer token"}

Key rotation

The authority signing seed is rotated over the running node's authority endpoint: POST /v1/authority rotates, GET /v1/authority reads status. Rotation is non-disruptive: existing capabilities keep verifying because the kernel verifies against the full trusted-key history, not only the current key.

# Before: generation 1, one key in the history.
$ curl -s -H "Authorization: Bearer $CHIO_TRUST_SERVICE_TOKEN" \
    http://127.0.0.1:8940/v1/authority | jq
{
  "configured": true,
  "backend": "sqlite",
  "publicKey": "8d7ddbdce2368deb5143de554fb574fea652fe3e5610acdb2b1be110083fe485",
  "generation": 1,
  "rotatedAt": 1785603453,
  "appliesToFutureSessionsOnly": true,
  "trustedPublicKeys": [
    "8d7ddbdce2368deb5143de554fb574fea652fe3e5610acdb2b1be110083fe485"
  ]
}

# Rotate the authority signing seed on the current leader.
$ curl -s -X POST -H "Authorization: Bearer $CHIO_TRUST_SERVICE_TOKEN" \
    http://127.0.0.1:8940/v1/authority | jq
{
  "appliesToFutureSessionsOnly": true,
  "backend": "sqlite",
  "configured": true,
  "generation": 2,
  "publicKey": "beb4438776406922a06f9477e07e49f63d95ac5e84a9aede414984c9b1de3639",
  "rotatedAt": 1785603453,
  "trustedPublicKeys": [
    "8d7ddbdce2368deb5143de554fb574fea652fe3e5610acdb2b1be110083fe485",
    "beb4438776406922a06f9477e07e49f63d95ac5e84a9aede414984c9b1de3639"
  ]
}

Rotation invariant one is visible in that payload: publicKey moved to the new key and trustedPublicKeys grew to two entries rather than replacing the first. A capability signed under generation 1 still verifies against the history. A subsequent GET /v1/authority returns the same generation-2 body.

Rotation invariants:

1. Capabilities issued under the previous seed still verify. Existing live sessions keep working.
2. New capabilities are signed under the new seed.
3. Trusted-key history replicates to every follower as part of the authority snapshot. Remote authority clients refresh trusted-key state on a short TTL, so every edge picks up the new generation without a process restart.

Security stance

The control plane enforces a small and boring set of security invariants:

• Kernel-mediated trust: the control plane stores and issues trust state. It never bypasses kernel checks. Every tool call still runs the full guard pipeline at the edge.
• One service token, optional tenant-read tokens: the required --service-token (env CHIO_TRUST_SERVICE_TOKEN) authenticates every request and keeps cross-tenant admin access. Repeatable --tenant-read-token tenant_id=token entries grant read-only access confined to a single tenant's receipts.
• HTTPS everywhere: terminate TLS in front of both the control plane and the hosted MCP / auth plane.
• Dedicated auth signing seed: when hosted OAuth is enabled, use a signing seed distinct from the capability authority.
• Shared budget in every multi-node deployment: never run edges against independent local budget stores in production. Budgets must exhaust once, everywhere.

Non-goals

The trust control plane is deliberately scoped. It does not attempt:

• Multi-datacenter consensus.
• Byzantine quorum rotation.
• HSM-backed signing.

Those are not part of this service. It provides single-region high availability and hosted OAuth behavior through the same kernel extension interfaces (CapabilityAuthority, RevocationStore, ReceiptStore, BudgetStore) already used in single-node mode.

For background on how the control plane fits into the broader economic model, see Economics and Trust Model.