Operate a Federation

One operator runs three federation-facing processes

The kernel host admits every action under local policy. The trust service (chio trust serve) runs the operator's trust control plane. The signals relay carries partner traffic over HTTPS, authenticated by mutual TLS or signed bodies, and mounts the optional iroh lane with chio pheromone relay serve --iroh-enable plus --iroh-transport-directory and --iroh-transport-key.

The HTTP relay and the iroh lane run together in the relay's dual mode while a fleet migrates. The admission gate fails closed: a directory that expires while the node runs stops admission until a valid bundle is published. Custody of the directory issuer's key and of the revocation authority's signing seed stays with the operator.

1. Re-handshake each partner before its rotation deadline

Every accepted handshake sets the partner's rotation deadline, twelve hours after the handshake by default. Run the handshake again before it passes; a partner past its deadline is refused as stale (PeerStale) until the two kernels handshake again.

A handshake re-proves a key the partner card already pinned. A rotated key reaches your trust store through a new card or the authority key history the card carries. Peer Handshake lists each refusal.

2. Reload the directory on every issuer publication

The relay's reloader re-reads the bundle every 60 seconds, re-verifies it, and swaps it in place without a restart. Alert on any reload outcome other than updated or unchanged.

increase(chio_federation_transport_directory_reload_total{outcome!~"updated|unchanged"}[15m]) > 0

You should see this expression stay at zero. The outcomes it catches are expired_failclosed, binding_revoked, below_min_version, trust_roots_changed, current_body_replaced, trust_roots_empty, and error; an expired bundle drops the gate to deny-all until a valid successor arrives. Transport Directory walks each check.

3. Publish a revocation checkpoint on a fixed cadence

Run the checkpoint command on a schedule. It advances the revocation epoch, and the next assembled trust bundle carries the new checkpoint.

chio federation authority checkpoint \
  --profile profile.json --revocations revocations.json \
  --signing-keys signing-keys.json --out checkpoint.json

You should see checkpoint.json written and signed. Oracle epoch roots are a separate artifact with their own catch-up lane; Revocation Checkpoints covers the request fields and refusals.

4. Refresh treaty scopes after the bundle and checkpoint they name

A treaty scope binds the trust bundle digest, the revocation epoch digest, and the ladder manifest digests. Publish the bundle and the checkpoint first, then the refreshed scope that names them, and refresh before the manifests expire: a stale manifest or scope refuses admission. After publishing the refreshed scope, recompute the ladder intersection from it and both current manifests, and confirm the new expiry:

chio federation treaty intersect --treaty-scope treaty/scope.json \
  --manifest treaty/your-ladder.json --manifest treaty/partner-ladder.json \
  --now-unix-ms "$(date +%s000)" --report treaty/intersection.json
jq -c '{treatyId, expiresAtUnixMs, actions: [.actionClasses[].actionClassId]}' treaty/intersection.json

{"treatyId":"trt-01aa…4c19","expiresAtUnixMs":1791820800000,"actions":["workspace.read","workspace.write","model.invoke","merge"]}

You should see the four action classes the treaty admits and the expiry the refreshed manifests allow. Each kernel runs the same computation on its own, so admission on either side rests on the intersection it computed.

The transport directory is independent of all three. It holds the local kernel id, the partner transport bindings, and the party set of each treaty, so it reloads on its own clock. Treaties covers issuance, expiry, and revocation.

Health shows in the federation metric families

Scrape these families from the workspace metrics registry on the relay and the kernel host.

Each lane accept also opens a tracing span (lane_accept_span) that carries the lane and the handler's outcome (accept, reject, busy, timeout), recorded after the admission gate has passed. Admission decisions themselves count in chio_federation_transport_admission_total.

Each failing signal names its repair

Read the signal, then act on the routine it belongs to.

Admission rejects rise

chio_federation_transport_admission_total rising on reject means the directory no longer lists that endpoint, or something is probing the mesh. Reload the directory bundle if the partner should be admitted.

Verification failures by seam

In chio_federation_transport_verify_failures_total, the identity seam counts directory bundles refused at load and reload; catchup, revocation, fanout, and bilateral each count frames an admitted partner sent on that lane. Signal rejects count in chio_federation_transport_lane_total with outcome reject.

An epoch gap in catch-up

chio_federation_transport_catchup_epoch_gap_total means a catch-up response skipped an epoch and was rejected whole, with nothing merged. Run the catch-up again; the source label names the catch-up walk or the revocation lane.

The router stops

chio_federation_transport_router_alive at 0 means the router died. Restart the relay; the gauge returns to 1 once the router serves again. Accept-path sheds and stalls are on Limits & Backpressure.

Next: bound what each lane accepts