PlatformFleet Operations
Cluster
Kubernetes Admission
Use a ChioPolicy CRD and fail-closed admission webhooks to check signed capabilities before Kubernetes admits a pod.
Two Go modules back this page. github.com/backbay-labs/chio-k8s-webhooks (sdks/k8s/webhooks/) is the admission server: it holds the handlers, the capability verifier, and the two admission-configuration manifests. github.com/backbay-labs/chio-k8s-controller (sdks/k8s/controller/) is the Job reconciler. The CRD itself sits beside them at sdks/k8s/crds/.
Policy and admission verification
ChioPolicy is the declarative record an operator authors and kubectl surfaces. The webhook is the enforcement point, and it reads nothing from the CRD: its trust set and required-scope list come from its own environment, and a namespace opts into enforcement with a label. The two use the same canonical scope grammar, so what a policy declares and what the webhook enforces are written the same way.sdks/k8s/webhooks/validating-webhook.yaml:20-31sdks/k8s/webhooks/server.go:53-136at fe56570The ChioPolicy CRD
ChioPolicy is a namespaced custom resource in the chio.world API group, served at v1alpha1. It declares the scopes pods in a namespace must present, an enforcement mode, an optional selector, and default sidecar wiring for matching pods.
apiVersion: chio.world/v1alpha1
kind: ChioPolicy
metadata:
name: payments-guardrail
namespace: payments
spec:
# Every admitted pod must present a capability covering all of these.
requiredScopes:
- "ledger:invoke"
- "resource:receipts/*:read"
# Optional: narrow the policy to matching pods. Omit to cover the namespace.
selector:
matchLabels:
chio.world/governed: "true"
# enforce (default) | audit | disabled
enforcement: enforce
sidecarConfig:
image: ghcr.io/backbay-labs/chio-sidecar:latest
upstream: http://localhost:8080
specPath: /etc/chio/openapi.yaml
receiptStore: /var/lib/chio/receipts.db
autoInject: falsereceiptStore is a path, not a URI
--receipt-store is an Option<PathBuf> documented as an “optional SQLite receipt store path”, and the sidecar opens the file it names. A value with a scheme in front of it would be taken as a relative path. The ECS and Cloud Run manifests both pass a filesystem path here.Install the definition before applying any policy. The CRD ships in the Chio repository at sdks/k8s/crds/chiopolicy-crd.yaml:
kubectl apply -f sdks/k8s/crds/chiopolicy-crd.yaml
kubectl get chiopolicy -n paymentsThe resource is chiopolicies.chio.world, plural chiopolicies, singular chiopolicy, short name chiop. Three additional printer columns put the policy on one line of kubectl get output: Scopes from .spec.requiredScopes, Enforcement from .spec.enforcement, and Age from .metadata.creationTimestamp. The status subresource carries observedGeneration and a standard conditions array.
| Field | Type | Meaning |
|---|---|---|
spec.requiredScopes | string[] (required, min 1) | Canonical Chio scopes every governed pod's capability must cover. |
spec.selector | label selector | matchLabels / matchExpressions (In, NotIn, Exists, DoesNotExist). Omitted means the whole namespace. |
spec.enforcement | enum, default enforce | enforce rejects non-compliant pods; audit admits but logs violations; disabled skips enforcement. |
spec.sidecarConfig.image | string | Container image for the Chio sidecar governing matching pods. |
spec.sidecarConfig.upstream | string | Default upstream URL for the sidecar proxy. |
spec.sidecarConfig.specPath | string | Path to the OpenAPI spec the sidecar enforces. |
spec.sidecarConfig.receiptStore | string | Typed in the CRD as a “receipt storage backend URI”. The sidecar flag it feeds takes a SQLite file path. |
spec.sidecarConfig.autoInject | bool, default false | Inject the sidecar into matching pods without a per-pod annotation. |
Install the webhooks
Enforcement lives in two admission configurations, both shipped under sdks/k8s/webhooks. The validating webhook decides admission; the mutating webhook applies the same decision at CREATE so a mutation pass can never become a bypass.
| Configuration | Webhook | Path | Operations |
|---|---|---|---|
ValidatingWebhookConfiguration | validate.chio.world | /validate | CREATE, UPDATE |
MutatingWebhookConfiguration | mutate.chio.world | /mutate | CREATE |
Both target pods at Namespaced scope, register only v1 in admissionReviewVersions, declare sideEffects: None, and set a 10-second timeout. Both use failurePolicy: Fail: if the webhook is unreachable, admission is denied rather than waved through. The mutating configuration adds reinvocationPolicy: IfNeeded, so the API server may call it again after another mutating webhook has changed the object.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: chio-validating-webhook
labels:
app.kubernetes.io/name: chio-admission-controller
app.kubernetes.io/component: validating-webhook
webhooks:
- name: validate.chio.world
admissionReviewVersions:
- v1
clientConfig:
service:
name: chio-admission-controller
namespace: chio-system
path: /validate
port: 443
# caBundle should be injected by cert-manager or manually set.
# caBundle: <base64-encoded-ca-cert>
rules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["pods"]
scope: Namespaced
failurePolicy: Fail
sideEffects: None
timeoutSeconds: 10
namespaceSelector:
matchLabels:
chio.world/enforce: "true"The webhook serves TLS on port 8443 behind a chio-admission-controller Service in the chio-system namespace, fronted by the Service's port 443. Provision the serving certificate the way you provision any admission-webhook cert. The manifests leave the caBundle for cert-manager to inject, or set it by hand. See Secrets & Signing Keys for cert and key handling.
The mutating webhook emits no patch
sdks/k8s/ reads spec.sidecarConfig, so a deployment that wants the sidecar attached wires it in the pod template or in its own injector rather than expecting this webhook to do it.Opt in with namespace labels
A namespace only reaches the webhooks when it carries the matching label. The validating webhook fires on chio.world/enforce: "true"; the mutating webhook fires on chio.world/inject: "true". Unlabeled namespaces are untouched, so rollout is namespace by namespace.
$ kubectl label namespace payments chio.world/enforce=true
namespace/payments labeledConfigure the trust set
The webhook reads its verification policy from two environment variables, and only these two. There is no default: an unset or empty value fails closed, so a misconfigured deployment denies every pod rather than admitting on a permissive fallback.
| Variable | Value | Empty behavior |
|---|---|---|
CHIO_WEBHOOK_TRUSTED_KERNEL_KEYS | Comma-separated Ed25519 issuer public keys, hex-encoded. | Fail closed. |
CHIO_WEBHOOK_REQUIRED_SCOPES | Comma-separated canonical scopes every capability must cover. | Fail closed. |
Each trusted key is validated at load time as a well-formed 32-byte Ed25519 public key; a malformed entry stops the process from starting. The required-scope strings share the grammar used in spec.requiredScopes, so a namespace's policy and the webhook's enforced list read the same.
env:
- name: CHIO_WEBHOOK_TRUSTED_KERNEL_KEYS
value: "25403c1e...f0,9a7b18d2...c4" # kernel issuer public keys (hex)
- name: CHIO_WEBHOOK_REQUIRED_SCOPES
value: "ledger:invoke,resource:receipts/*:read"
- name: TLS_CERT_FILE
value: /etc/chio/tls/tls.crt
- name: TLS_KEY_FILE
value: /etc/chio/tls/tls.keyWith TLS_CERT_FILE and TLS_KEY_FILE both set the server serves HTTPS with a TLS 1.2 floor; with neither set it logs a warning and serves plaintext for local development only. Override the listen port with PORT; the default is 8443. A GET /healthz endpoint returns ok for liveness and readiness probes.
What the pod carries
A governed pod presents its capability as canonical JSON in the chio.world/capability-token annotation. The token is the standard Chio capability shape: an issuer, a subject, a scope envelope, a validity window, an optional parameter binding, and an Ed25519 signature.
apiVersion: v1
kind: Pod
metadata:
name: ledger-writer
namespace: payments
labels:
chio.world/governed: "true"
annotations:
chio.world/capability-token: |
{"id":"cap-019dbbf8-...","issuer":"25403c1e...f0",
"subject":"25403c1e...f0",
"scope":{"grants":[{"server_id":"*","tool_name":"ledger",
"operations":["invoke"]}],
"resource_grants":[{"uri_pattern":"receipts/*",
"operations":["read"]}]},
"issued_at":1776975000,"expires_at":1776978600,
"parameter_binding":{"operations":["CREATE"],
"namespaces":["payments"],"resources":["pods"]},
"signature":"7be63cdb..."}
spec:
containers:
- name: app
image: myapp:latestThe signature covers a canonical serialization of every field except signature itself: keys sorted, no insignificant whitespace. The webhook reconstructs that byte string from the raw annotation, strips the signature field, and verifies against the issuer key, so re-ordering or reformatting the JSON does not change the verified bytes. For the capability model in full, see Capabilities.
Self-asserted policy is never trusted
chio.world/required-scopes or chio.world/exempt on itself is denied. Required scopes come from the webhook's configuration only, and the webhook honors no self-declared exemption: a workload cannot rewrite the operator's policy or opt itself out.The verification flow
On every admission the webhook decodes the AdmissionReview, extracts the candidate token, and builds an admission context from the request: operation (upper-cased), namespace, and resource. It then runs the full flow, denying on the first failure with a typed reason that Kubernetes surfaces to the caller.
- Deny if no trusted kernel keys are configured.
- Deny if no required scopes are configured.
- Deny if the capability token is missing or malformed.
- Deny if the issuer is not in the trusted-key set.
- Deny if the Ed25519 signature does not verify.
- Deny if the token is not yet valid or has expired.
- Deny if the parameter binding does not cover the admission context.
- Deny if any required scope is not covered by the token's grants.
The validity window is compared against the request time in Unix seconds: a token is rejected while now < issued_at and once now >= expires_at. A token with no parameter_binding is permitted, since scope coverage carries the authorization, but a binding that is present must match: its operations compare case-insensitively, its namespaces and resources exactly. Empty fields in a binding act as wildcards.
A denial is an AdmissionResponse with allowed: false, the reason under status.message, and code: 403. The handler copies the request's uid onto the response and re-encodes the whole review, so the body on the wire also carries the request it answers. The response half:
{
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": {
"uid": "<echoed from request.uid>",
"allowed": false,
"status": {
"message": "webhook denies admission: capability token does not cover required scope \"ledger:invoke\"",
"code": 403
}
}
}Scope grammar
Required scopes and the grants inside a capability are written in the same colon-delimited grammar. The webhook accepts three forms and normalizes operation aliases before matching.
| Form | Expands to | Example |
|---|---|---|
<name>:<op> | tool scope, any server | ledger:invoke |
<kind>:<name>:<op> | resource or prompt scope | resource:receipts/*:read |
tool:<server>:<name>:<op> | fully-qualified tool scope | tool:srv-files:read_file:invoke |
A grant covers a required scope when the server and name patterns match and the operation is present. Patterns are exact, or a trailing * matches by prefix, or a bare * matches anything, so a grant on receipts/* covers a requirement on receipts/2026-07. Operation aliases fold together: call, exec, and execute normalize to invoke; watch to subscribe; result to read_result. The two-part shorthand folds harder than the longer forms: read and write both become invoke there, while in a three- or four-part scope read stays read and write is rejected as an unsupported operation.
Governing batch jobs
Admission gating covers pods at the door. For batch Job workloads that need a grant minted at creation and receipts aggregated across their lifecycle, the Chio Kubernetes controller pairs with the webhook. It watches batch/v1 Jobs labeled chio.world/governed: "true", mints a capability through the sidecar at creation, harvests per-pod receipts, and releases the grant with a signed JobReceipt on completion or failure.
# Controller manifests create the chio-system namespace, RBAC, and Deployment.
$ kubectl apply -f sdks/k8s/controller/config/manager/manager.yaml
$ kubectl apply -f sdks/k8s/controller/config/rbac/role.yamlThe controller ships as ghcr.io/backbay-labs/chio-k8s-controller and reaches the sidecar at CHIO_SIDECAR_URL. Its fail-closed configuration reinforces the webhook: if the sidecar is unreachable at mint time it requeues with backoff and persists no placeholder capability, so the Job's pods stay ungoverned and are rejected at admission rather than starting unsigned.
Where receipts land
JobReceipt and posts it to the sidecar's receipt store. Those receipts are the same cryptographic artifacts the rest of Chio emits. See Receipts for the format and verification path.Next steps
- Capabilities for the token model the annotation carries: scope, delegation, and parameter binding.
- Trust Model for what a trusted kernel key attests and how issuers are rotated.
- Sidecar HTTP Service for the sidecar image referenced by
spec.sidecarConfig. - Secrets & Signing Keys for provisioning the webhook's serving certificate.