PlatformFleet Operations
Cluster
Policy Rollout
There is no distribution service. Policy arrives as a local file, and the receipt hash is the only fleet-wide version you get.
One row, expanded
The absent mechanism, named
Chio has no policy control plane. Not a partial one, not a feature-gated one, not one behind a flag. A node loads one file from its own filesystem at process start, fingerprints it, and never looks at it again. Every other node does the same thing independently. Nothing compares the results.
One edge does pull its policy over the network
chio api protect has no policy file. Its policy is the OpenAPI spec, and --spec is optional: omit it and discover_spec probes the upstream for /openapi.json, /openapi.yaml, /swagger.json, then /api-docs, taking the first non-empty success. The fetch is bounded by an egress contract pinned to the upstream scheme and authority, four redirects, and 64 MiB. It is still a network fetch of the document that becomes policy_hash, chosen by the upstream, at whatever revision it was serving when this replica booted. Two replicas started an hour apart can enforce different route tables under different hashes with nothing to notice. Pass --spec and the rest of this page applies to that edge too.This needs saying out loud because the surrounding system looks like it would have one. The trust control plane has a leader, peer authentication, delta replication for five streams, and an authority snapshot that propagates key rotation. Assuming policy travels the same way is expensive: two nodes under one authority key can enforce different guard pipelines indefinitely, answer every health probe green, and issue capabilities the other node would have denied.
Five negatives, each checkable
| Claim | How to check it |
|---|---|
| No runtime policy endpoint. | build_router registers the health route, 171 more named by distinct *_PATH constants from trust_control/service_types/paths.rs, one literal /metrics, and a dashboard SPA fallback. None of them reads or writes a guard or kernel policy. Nine constants in that file mention policy and all name different objects: three federation open-admission routes, two passport verifier-policy routes, three credit report routes whose last path segment ends in -policy, and one schema identifier that is not a route at all. |
| No policy delta between peers. | The internal endpoints are cluster status, cluster snapshot, partition control, authority snapshot, admission authority, and deltas for revocations, tool receipts, child receipts, budgets, and lineage. Ten, in the same file. No eleventh. |
| No fetch, no registry. | The only loader wired into resolve_from_path is load_from_filesystem. A scheme-dispatching create_composite_loader exists in chio-policy/src/resolve.rs with no production caller, and even it refuses http:// and https:// with a typed ResolveError::Http: HTTP-based policy loading is not supported in chio-policy. chio replay traffic --against rejects a sha256: prefix, a bare 64-character lowercase hex string, a version: prefix, and a bare name@semver because, in its own words, there is no registry-backed resolver to materialize a verified policy from them. |
| No signed policy bundle. | Signing exists one layer down, per WASM guard module, and only when the operator opts in: if guard-manifest.yaml sets signer_public_key, the loader requires a detached Ed25519 .wasm.sig sidecar over an envelope domain-separated by chio-wasm-guard-v1, and a missing or malformed signature fails closed unless allow_unsigned: true. A manifest with no signer_public_key declares the module unsigned and falls through to that same flag. The policy document itself carries no signature and passes through no verification step. load_policy reads bytes and parses them. |
| No policy version on the wire. | ClusterStatusResponse has eleven fields: self URL, leader URL, role, quorum flag, quorum size, reachable nodes, election term, authority lease, replication heads, peers, budget acknowledgment heads. None of them is a policy hash or a build identifier. The one version a peer does advertise sits inside the replication heads, where revocation_cursor_version selects which revocation delta contract a follower pulls; it says nothing about policy. So there is no policy identity to compare and nothing refuses a mismatch. |
Two things that look like a rollout mechanism and are not
chio-wasm-guards/src/hot_reload.rs implements a real one: a frozen corpus of exactly CANARY_FIXTURE_COUNT = 32 fixtures verified against a MANIFEST.sha256, reload_with_canary replaying every fixture through the replacement module and aborting the swap on one mismatched verdict byte, then an atomic publish that leaves in-flight evaluations on the old epoch. It swaps one guard module inside one process. The Engine type is referenced nowhere outside that crate and its own tests. The CLI does depend on chio-wasm-guards, but only for the manifest, blocklist, and wasmtime backend behind chio guard build, sign, verify, and publish. The reload engine is an embedding API an operator calls in process, and it has no notion of a second node.The CRD that has no controller.
sdks/k8s/crds/chiopolicy-crd.yaml defines ChioPolicy with requiredScopes, a label selector, an enforcement enum of enforce/audit/disabled, a sidecarConfig block, and a status subresource with observedGeneration. No Go or Rust code in the tree references the kind. The shipped controller reconciles Jobs only. The admission webhook reads its entire verification policy from two environment variables, CHIO_WEBHOOK_TRUSTED_KERNEL_KEYS and CHIO_WEBHOOK_REQUIRED_SCOPES, both required with no default, and its mutate path emits no patch at all. Applying a ChioPolicy changes nothing.What actually exists
Three objects, and one of them is only a pair of strings.
- The document. A YAML file at a path the operator passes on the command line. It is either a Chio YAML policy or a HushSpec policy;
load_policydecides by looking for ahushspec:top-level key. - The materialization.
LoadedPolicy: kernel config, default capabilities, a compiled guard pipeline, a post-invocation pipeline, and the optional reputation, runtime-assurance, and threshold-approval policies. It exists in process memory and is moved into the kernel at construction. - The identity.
PolicyIdentity { source_hash, runtime_hash }. This is the whole of Chio’s policy versioning. Everything below is about what those two hashes do and do not cover.
Only one of the two reaches a receipt. build_kernel sets KernelConfig.policy_hash = identity.runtime_hash, and the source hash goes to a log line and to evidence exports. policy_hash is a plain String everywhere in the type system, and nothing on the receipt path constrains its shape. Its derivation is not uniform across edges:
| Edge | What policy_hash holds | Set in |
|---|---|---|
chio run, chio mcp serve, and every other kernel built from a policy file | The runtime hash of the loaded policy. | build_kernel, chio-control-plane/src/lib.rs |
chio api protect | SHA-256 of the OpenAPI spec bytes, whether those bytes came from --spec or from probing the upstream. | chio-api-protect/src/proxy/state.rs |
Anything embedding chio-tower | Whatever string the embedder passed to ChioEvaluator::new. The crate’s own tests pass "test-policy". | chio-tower/src/evaluator.rs |
| SCIM deprovision receipts | A synthesized digest over chio.scim-lifecycle-delete.v1:{provider}:{subject}, unrelated to any loaded policy. | trust_control/config_and_public.rs |
Compare hashes only within one edge class. A fleet mixing sidecars and kernels has two populations of policy hash that were never meant to match.
How policy reaches a node, in order
1. Read, once
load_policy does a read_to_string on the path, hashes those bytes into source_hash, and branches on format. That is the entire ingestion path. There is no watcher, no SIGHUP handler, no reload verb.
2. Resolve, on the local filesystem only
A HushSpec document may carry extends. resolve_from_path follows the chain with a filesystem loader, canonicalizing each hop, detecting cycles by comparing canonical paths, and refusing past DEFAULT_MAX_EXTENDS_DEPTH = 32 or a document larger than DEFAULT_MAX_POLICY_DOCUMENT_BYTES (4 MiB, applied per document, not to the chain). The depth check counts the entry document, which is already on the stack before the first hop, so the ceiling is 32 documents and 31 extends hops. The built-in rulesets (default, strict, permissive, ai-agent, cicd, remote-desktop, panic) are include_str! blobs compiled into the binary and reachable through the library, not through extends on this path. chio mcp serve --preset code-agent writes its embedded YAML to a temp file for exactly that reason: the ordinary path-based loader, and therefore the ordinary hashing, then applies unchanged.
The source hash covers the top document only
source_hash is a digest of the bytes at the path you passed. Parents pulled in through extends are not folded into it. Two nodes with byte-identical top-level policies and different parent files on disk report the same source hash. One auxiliary asset is the exception, on the HushSpec path only: when extensions.detection.threat_intel.pattern_db is configured and not disabled, its resolved path and SHA-256 are folded into both hashes, which the test load_hushspec_policy_identity_tracks_threat_intel_pattern_db_bytes pins by rewriting the pattern database between two loads and asserting both hashes move. The Chio-YAML path folds in no auxiliary assets at all.3. Fingerprint, over a curated field set
The runtime hash is not a digest of the compiled policy. It is a SHA-256 over a canonical JSON object built by hand, and the field list is the whole story:
let fingerprint = serde_json::json!({
"format": PolicyFormat::HushSpec.as_str(),
"kernel": kernel,
"default_capabilities": default_capabilities,
"rules": {
"forbidden_paths": rules.and_then(|entry| entry.forbidden_paths.as_ref()),
"path_allowlist": rules.and_then(|entry| entry.path_allowlist.as_ref()),
"egress": rules.and_then(|entry| entry.egress.as_ref()),
"secret_patterns": rules.and_then(|entry| entry.secret_patterns.as_ref()),
"patch_integrity": rules.and_then(|entry| entry.patch_integrity.as_ref()),
"shell_commands": rules.and_then(|entry| entry.shell_commands.as_ref()),
"tool_access": rules.and_then(|entry| entry.tool_access.as_ref()),
},
"reputation": extensions.and_then(|entry| entry.reputation.as_ref()),
"auxiliary_assets": auxiliary_assets,
});Two of those fields carry less than they look like. kernel on the HushSpec path is KernelPolicyConfig::default(), the same constant for every HushSpec document, because the format has no way to set kernel config; it distinguishes nothing. And reputation is the only member of extensions present at all.
RULE_BLOCK_NAMES in chio-policy/src/models/rules.rs is a fourteen-element array. Seven are named above. Of the seven that are not, human_in_loop can move the hash indirectly, but only conditionally: compile_scope reads it solely on the branch where tool_access is present and enabled, and the compiled scope is what appears as default_capabilities. Without a tool_access block the scope is permissive and human_in_loop is invisible to both hashes. The other six do not move the hash at all, and each compiles into a real guard that changes verdicts: velocity, computer_use, remote_desktop_channels, input_injection, browser_automation, and code_execution. Change code_execution on one node and not another and the two nodes deny different calls under an identical runtime hash.
The gap is wider than the rule blocks. Four more inputs materialize into the running kernel from compile_policy_with_options and load_hushspec_policy, and none of them appears in the fingerprint:
| Unhashed input | What it materializes |
|---|---|
extensions.detection | Up to three guards: prompt-injection, jailbreak, embedding-anomaly. Only the threat-intel pattern database bytes reach the hash, as an auxiliary asset. Thresholds and detector configuration do not. |
extensions.origins | The tightest budgets.tool_calls across all origin profiles collapses into one agent-velocity ceiling over a fixed 60-second window. |
extensions.runtime_assurance | The attestation trust policy the kernel enforces on tiered issuance. |
extensions.chio.human_in_loop.approvers | The threshold-approval quorum: n, the approver key set, and the timeout. The requirement is stamped with the runtime hash, but changing who can approve does not change it. |
The Chio-YAML path is stricter: it serializes the whole guards config into the fingerprint, and has no extensions tree to miss. The gap above is a HushSpec property, not a universal one.
4. Build, adding guards the policy never mentioned
build_kernel registers chio_guards::default_runtime_guard_profile() before it registers the policy-compiled pipeline: an internal-network guard, an agent-velocity guard on default config, and an advisory pipeline. It then appends that profile’s sanitizer hook to the post-invocation pipeline. Those four come from the binary, not from the file, and they are outside both hashes. Two nodes running the same policy on different Chio builds can differ in the base profile with no hash change to show for it. The policy hash identifies a document, never a deployment.
5. Stamp, on every mediated decision
This is the part that works, and it is stronger than a field on a record. ChioReceiptIdInput includes policy_hash, and the receipt id is the content-addressed SHA-256 over that canonical input, so the hash is bound into the receipt’s identity rather than carried alongside it:
/// Compute the authoritative receipt id from canonical receipt body fields.
pub fn chio_receipt_id(body: &ChioReceiptBody) -> Result<String> {
let input = ChioReceiptIdInput::from(body);
let canonical = canonical_json_bytes(&input)?;
Ok(sha256_hex(&canonical))
}
/// Validate, bind the caller nonce, and compute the authoritative receipt id.
pub fn prepare_receipt_body_for_signing(mut body: ChioReceiptBody) -> Result<ChioReceiptBody> {
body.validate_signable_semantics()?;
bind_receipt_signing_nonce(&mut body);
body.id = chio_receipt_id(&body)?;
Ok(body)
}Read that for what it is. Editing policy_hash after signing changes the id, so the edit is detectable by anyone who recomputes it. It does not, on its own, establish that the hash in the body is the one the decision was actually taken under.
The check that does bite is on the durable admission path, on by default because DurableAdmissionMode defaults to SideEffecting. The admission binding folds policy_hash into the operation’s identity, and terminal projection refuses a receipt whose hash differs from the binding it completes, so an operation admitted under one policy cannot be finished under another across a restart. Child request receipts and lineage statements carry the field but pass through neither gate.
6. Stop
KernelConfig is consumed by ChioKernel::new and policy_hash has no setter. Applying a new policy means restarting the process. That is the whole rollout primitive, and the platform you deploy on owns the ordering.
What to do instead
The absence has a workable shape if you stop treating policy as runtime configuration and start treating it as a build input.
- Ship the file with the artifact. Bake the policy into the image or mount it read-only from the same versioned source as the binary. The base
chiostage indeploy/docker/Dockerfileships no policy at all (itsCMDis--help); thechio-mcp-demostage layered on it bakes one in with--policy /opt/chio/examples/policy.yaml. Copy the pattern, not the demo. Because the base guard profile is compiled in, pinning the binary and pinning the policy are one decision, not two. - Compute the expected hashes offline.
chio check --policy ./policy.yaml --tool read_file --jsonprintspolicy_hashandpolicy_source_hashalongside the verdict.--paramsdefaults to{}and--serverto*. Record the pair with the release. One trap in a release script: the command exits 2 when the sample verdict is a deny, so pick a probe your policy allows or handle the exit code, otherwise the step that reads your policy identity fails the build. - Replay before you roll.
chio replay traffic --from capture.ndjson --against path:./candidate.yaml --tenant-pubkey ./tenant.pub --jsonbuilds a fresh ephemeral kernel under the candidate policy, allocates a replay receipt partition type-fenced against production, re-runs every captured invocation through the pre-invocation pipeline, and reports matches, drifts, and errors, withok()true only when drifts and errors are both zero. Receipts are namespacedreplay:<run_id>:<frame_id>. Three limits to plan around. A policy with post-invocation guards is rejected, because tee frames carry a response digest rather than redacted bytes; so is any default capability with a server-specific grant, because frames do not carry the original tool server id. And a frame counts as a match only when the verdict, the attributed guard, and the deny reason string all agree, so a build that reworded a reason registers as drift. This is the closest thing to a canary in the tree, and it runs on your workstation rather than on traffic. - Sequence the restart yourself. Nothing in Chio orders a fleet-wide change. Use the platform’s rollout, and keep the window short: while it is open, the cluster is genuinely running two policies. If the change tightens a guard, roll the strictest nodes last so a permissive node is never the leader; if it loosens one, the ordering does not matter, but the receipts still record which node allowed what.
- Verify from receipts, not from health. The trust-control health body reports
issuancePolicyConfiguredandruntimeAssurancePolicyConfiguredas booleans and never a digest, andTrustServiceConfigdoes not retain the policy identity at all. The receipt store is where the answer lives.
That last check is a direct query, because the API layer has no policy dimension: ReceiptQuery filters on capability, tool server, tool name, outcome, time, cost, agent subject, and tenant, but not policy hash; receipt analytics groups by none of it; and while policy_hash is a NOT NULL column on chio_tool_receipts, the table indexes timestamp, capability, subject, grant, tool, and decision, not it. Read the file on each node, remembering that timestamp is an INTEGER of Unix seconds:
SELECT policy_hash,
COUNT(*) AS receipts,
MIN(timestamp) AS first_seen,
MAX(timestamp) AS last_seen
FROM chio_tool_receipts
WHERE timestamp >= :rollout_started_at
GROUP BY policy_hash
ORDER BY last_seen DESC;One row per node after the rollout window closes is convergence. Two rows that both keep advancing last_seen is drift, and the node whose hash you do not recognize is the one to look at. For an auditable version of the same evidence, POST /v1/evidence/export attaches the policy source file itself plus a metadata block carrying format, sourceHash, runtimeHash, sourcePath, and sourceBytes.
The startup log line is narrower than it sounds
policy_path, policy_format, source_policy_hash, and runtime_policy_hash. The message differs by command: chio run logs loaded policy, chio mcp serve-http logs loaded policy for remote MCP edge, and chio mcp serve over stdio, chio trust serve, and chio api protect emit neither. Both are info events, and the CLI installs a default filter of warn,chio.guard=info when RUST_LOG is unset, so on a default deployment the line never reaches stderr. Set RUST_LOG=info on the two commands that emit it, then alert on an unexpected value and you learn about drift in seconds instead of at the next audit.Guarantees and limits
| Status | Claim | Evidence |
|---|---|---|
| Shipped | Every mediated receipt carries the policy hash the kernel was configured with, and the receipt id is derived from a preimage that includes it. | ChioReceiptIdInput in chio-core-types/src/receipt/body.rs |
| Shipped | An operation admitted under one policy cannot be completed under another: the durable admission binding includes policy_hash and terminal projection refuses a mismatched receipt. On by default, since DurableAdmissionMode defaults to SideEffecting. | admission_operation/identity.rs; the equality chain in admission_coordinator/terminal.rs and admission_operation/projection.rs |
| Shipped | Guard and kernel policy loading is deterministic and local: same bytes, same identity, with no network reference reachable from the resolver. | policy/loader.rs; load_from_filesystem as the only wired loader in chio-policy/src/resolve.rs |
| Shipped | An out-of-band asset referenced by policy is bound into both hashes: the threat-intel pattern database contributes its resolved path and digest. | hushspec_auxiliary_asset_digests; test load_hushspec_policy_identity_tracks_threat_intel_pattern_db_bytes |
| Modeled | Receipt-field coupling, policy hash included, is specified as a conjunction under the chio_creusot_contracts verification cfg. The postcondition restates the conjunction; it says nothing about whether the hash names the intended policy. | receipt_fields_coupled in chio-kernel-core/src/formal_aeneas.rs |
| Limit | On the HushSpec path the runtime hash covers seven of fourteen rule blocks, plus extensions.reputation and the threat-intel pattern-database bytes. Six rule blocks and four extension inputs compile into guards, an attestation trust policy, or an approval quorum while moving neither hash, so two nodes can enforce different pipelines under one hash. | runtime_hash_for_hushspec against models/rules.rs, compiler/rules.rs, compiler/detection.rs, compiler/budgets.rs |
| Limit | chio api protect fetches its effective policy over the network when --spec is omitted, so its policy_hash names whatever the upstream served at boot. Pass --spec to close it. | load_spec_content falling through to discover_spec in chio-api-protect/src/spec_discovery.rs |
| Limit | The default runtime guard profile is registered from the binary and appears in no hash. Policy identity does not identify a deployment. | build_kernel in chio-control-plane/src/lib.rs |
| Limit | Drift is detectable only after a governed action has already been decided, and only by reading the receipt store or an evidence export. There is no read path that compares two nodes, and the boot-time log line is filtered out by default. | No policy filter on ReceiptQuery; no policy dimension in ReceiptAnalyticsQuery; no index on the policy_hash column; the warn,chio.guard=info default in init_redacted_tracing |
| Unsupported | Runtime policy reload. There is no watcher, no reload verb, and no setter for KernelConfig::policy_hash. A policy change is a process restart. | kernel/kernel_struct.rs; load_policy callers |
| Unsupported | Signed policy bundles, policy registries, staged or canaried fleet rollout, and version negotiation over a policy artifact. Guard-module signing and the in-crate canary harness are the adjacent mechanisms; neither crosses a node boundary. | chio-wasm-guards/src/manifest.rs and hot_reload.rs; load_from_filesystem as the only loader wired into resolve_from_path, so no policy identity reaches a peer to negotiate over |
| Not wired | The ChioPolicy CRD. The schema exists; no controller or webhook reads it, and its enforcement, sidecarConfig, and status fields are inert. | No reference to the kind outside the CRD file and prose docs; sdks/k8s/webhooks/config.go |
| Not read | CHIO_POLICY_SOURCE. No crate reads it. The Istio example manifest sets it, the AWS Lambda integration doc sets it to an s3:// object, and an example comment describes fetching a "signed policy bundle" through it. Setting it has no effect. | No occurrence in any .rs file; docs/architecture/reliability/RFC-0005-durable-by-default-wiring.md flags it for removal |
Next Steps
- Cluster Overview · the four bounded claims this page expands, and the three failure domains a per-node probe cannot report
- chio.yaml Configuration · the schema of the file that has to reach every node by hand
- Replication & Convergence · what the pull round does replicate, and the contract violations that demote a peer
- Authority & Rotation · the one object that is propagated between nodes, and how that snapshot differs from a policy
- HushSpec Policy Format · the fourteen rule blocks, seven of which the runtime hash names