Kubernetes Admission

Pods in a labeled namespace start only when they carry a capability token signed by a kernel key you pinned and covering the scopes you require. Here you run your pricing service in the pricing namespace, and only a pod holding a grant for quote from your kernel starts. An empty trust file, an unreachable webhook, or one failed check stops the pod before any of its containers run.

Two components do the work: the admission webhook, and a controller that governs batch Jobs. Each release publishes their images and manifests, with the ChioPolicy CRD, as release assets. Each admitted pod runs the Chio sidecar next to its app.

1. Install the ChioPolicy CRD and declare a policy

ChioPolicy is a namespaced resource in the chio.world API group, served at v1alpha1. It records the scopes a namespace's pods must present, an enforcement mode, an optional selector, and the sidecar wiring for matching pods.

apiVersion: chio.world/v1alpha1
kind: ChioPolicy
metadata:
  name: quote-guardrail
  namespace: pricing
spec:
  # Every admitted pod must present a capability covering all of these.
  requiredScopes:
    - "quote:invoke"
    - "resource:receipts/*:read"
  # Optional: narrow the policy to matching pods. Omit to cover the namespace.
  selector:
    matchLabels:
      chio.world/governed: "true"
  # enforce (default) | audit | disabled
  enforcement: enforce
  sidecarConfig:
    image: ghcr.io/backbay-labs/chio-sidecar:0.2.0
    upstream: http://localhost:8000
    specPath: /etc/chio/openapi.yaml
    receiptStore: /var/lib/chio/receipts.db
    autoInject: false

$ REL=https://github.com/backbay-labs/chio/releases/download/v0.2.0
$ kubectl apply -f $REL/chiopolicy-crd.yaml
$ kubectl apply -f chiopolicy.yaml
$ kubectl get chiopolicy -n pricing

customresourcedefinition.apiextensions.k8s.io/chiopolicies.chio.world created
chiopolicy.chio.world/quote-guardrail created
NAME              SCOPES                                        ENFORCEMENT   AGE
quote-guardrail   ["quote:invoke","resource:receipts/*:read"]   enforce       4s

You should see the policy on one line, in the CRD's printer columns. The webhook enforces the trust set and scope list from its own environment, which step 2 sets; ChioPolicy is the record operators author and kubectl shows. Both use one canonical scope grammar, so the declared and enforced lists read the same. The field reference lists every field.

2. Deploy the webhook with its trust set

The webhook trusts only the kernel keys pinned to the webhook.kernel role in its trust file. Pin your kernel from its card, then ship the file to the cluster as a ConfigMap.

$ chio trust pin webhook.kernel ./larkfield.card --trust ./trust.toml
$ kubectl -n chio-system create configmap chio-trust --from-file=trust.toml

pinned     larkfield  webhook.kernel  ed25519:4f1c…a2e9
file       ./trust.toml
configmap/chio-trust created

Mount the file and name it with CHIO_TRUST, beside the required scopes. Neither has a default: a trust file with no webhook.kernel pin, or an empty scope list, denies every pod.

env:
  - name: CHIO_TRUST
    value: /etc/chio/trust.toml
  - name: CHIO_WEBHOOK_REQUIRED_SCOPES
    value: "quote:invoke,resource:receipts/*:read"
  - name: TLS_CERT_FILE
    value: /etc/chio/tls/tls.crt
  - name: TLS_KEY_FILE
    value: /etc/chio/tls/tls.key
volumeMounts:
  - name: chio-trust
    mountPath: /etc/chio/trust.toml
    subPath: trust.toml
    readOnly: true
# in the pod spec:
volumes:
  - name: chio-trust
    configMap:
      name: chio-trust

Each pinned key must parse as an Ed25519 public key at load time, or the process exits before serving. With TLS_CERT_FILE and TLS_KEY_FILE set, the server serves HTTPS with a TLS 1.2 floor on port 8443 (override with PORT), behind the chio-admission-controller Service on port 443 in chio-system. Provision the serving certificate as for any admission webhook: cert-manager injects the caBundle, or you set it; Secrets & Signing Keys covers the key material.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: chio-validating-webhook
  labels:
    app.kubernetes.io/name: chio-admission-controller
    app.kubernetes.io/component: validating-webhook
webhooks:
  - name: validate.chio.world
    admissionReviewVersions:
      - v1
    clientConfig:
      service:
        name: chio-admission-controller
        namespace: chio-system
        path: /validate
        port: 443
      # caBundle should be injected by cert-manager or manually set.
      # caBundle: <base64-encoded-ca-cert>
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
        scope: Namespaced
    failurePolicy: Fail
    sideEffects: None
    timeoutSeconds: 10
    namespaceSelector:
      matchLabels:
        chio.world/enforce: "true"

$ kubectl apply -f $REL/validating-webhook.yaml
$ kubectl apply -f $REL/mutating-webhook.yaml
$ kubectl -n chio-system port-forward svc/chio-admission-controller 8443:443 &
$ curl -sk https://localhost:8443/healthz

validatingwebhookconfiguration.admissionregistration.k8s.io/chio-validating-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/chio-mutating-webhook created
ok

You should see both configurations created and ok from GET /healthz, the route liveness and readiness probes use. Both configurations set failurePolicy: Fail and a 10-second timeout, so an unreachable webhook denies admission. The mutating webhook runs the same verification at CREATE and returns an allow with no patch, so a mutation pass admits exactly what validation admits.

3. Opt a namespace in with its label

$ kubectl label namespace pricing chio.world/enforce=true

namespace/pricing labeled

You should see the namespace labeled. The validating webhook fires on chio.world/enforce: "true" and the mutating webhook on chio.world/inject: "true". Namespaces without the labels never reach the webhooks, so rollout proceeds one namespace at a time.

4. Admit a pod that carries its capability

A governed pod presents its capability as canonical JSON in the chio.world/capability-token annotation: issuer, subject, scope, validity window, an optional parameter binding, and an Ed25519 signature. Keys and the signature are shortened here; the annotation carries them in full.

apiVersion: v1
kind: Pod
metadata:
  name: pricing-api
  namespace: pricing
  labels:
    chio.world/governed: "true"
  annotations:
    chio.world/capability-token: |
      {"id":"cap-01ab…77d1","issuer":"4f1c…a2e9",
       "subject":"4f1c…a2e9",
       "scope":{"grants":[{"server_id":"pricing","tool_name":"quote",
         "operations":["invoke"]}],
         "resource_grants":[{"uri_pattern":"receipts/*",
         "operations":["read"]}]},
       "issued_at":1776975000,"expires_at":1776978600,
       "parameter_binding":{"operations":["CREATE"],
         "namespaces":["pricing"],"resources":["pods"]},
       "signature":"7be6…3cdb"}
spec:
  containers:
    - name: app
      image: registry.larkfield.example/pricing:2f60a91

pod/pricing-api created

You should see the pod created. The signature covers a canonical serialization of every field except signature, with keys sorted and no insignificant whitespace. The webhook rebuilds those bytes from the raw annotation and verifies them against the issuer key, so reordering or reformatting the JSON leaves the verified bytes unchanged. Declare chio-sidecar in the pod template as a native sidecar: an init container with restartPolicy: Always and a startupProbe on /chio/health.

5. Verify that an uncovered pod is refused

Submit a pod whose token grants receipts/* reads but omits the quote tool: a reporting job that should never price anything.

$ kubectl apply -f pod-uncovered.yaml

Error from server (Forbidden): error when creating "pod-uncovered.yaml": admission webhook "validate.chio.world" denied the request: webhook denies admission: capability token does not cover required scope "quote:invoke"

You should see the refusal with the scope it lacked. The webhook answers with allowed: false, the reason in status.message, and code: 403; the full denial response is in the reference.

The webhook denies at the first failed check

On each admission the webhook decodes the AdmissionReview, extracts the candidate token, and builds an admission context from the request: operation (upper-cased), namespace, and resource. It then runs eight checks in order and stops at the first failure with a typed reason that Kubernetes returns to the caller.

1. Trust set. At least one key is pinned to webhook.kernel.
2. Required scopes. At least one required scope is configured.
3. Token. The capability annotation is present and well formed.
4. Issuer. The issuer is a pinned webhook.kernel key.
5. Signature. The Ed25519 signature verifies over the canonical bytes.
6. Window. The request time falls inside the validity window.
7. Binding. A parameter binding, when present, covers the admission context.
8. Scope. The token's grants cover every required scope.

The window compares Unix seconds: a token is rejected while now < issued_at and once now >= expires_at. A token with no parameter_binding passes the binding check, since scope coverage carries the authorization. A present binding compares operations case-insensitively and namespaces and resources exactly, and an empty field acts as a wildcard.

Required scopes come only from the webhook's configuration. A pod that sets chio.world/required-scopes or chio.world/exempt on itself is denied, so a workload can neither rewrite the operator's policy nor opt out. Capabilities covers the token model.

The sidecar drains inside the pod's grace period

Kubernetes starts a native sidecar before the app containers once its startupProbe passes and stops it after they exit. The readiness probe on /chio/health returns 503 when the receipt store can no longer persist, which removes the pod from its Service endpoints; the liveness probe polls the process-only /chio/live.

On pod deletion the sidecar receives SIGTERM after the app exits, stops accepting connections, and lets admitted requests finish and write their receipts. The drain lasts the hop ceiling plus 5 seconds: 25 seconds at the default --upstream-timeout-secs of 20. terminationGracePeriodSeconds defaults to 30 and counts from deletion, so the app's exit spends the same budget. Raise the grace period with the hop ceiling: at 30 seconds of hop the drain needs 35, and the kubelet's SIGKILL at 30 ends calls before they are receipted.

Read more in Node lifecycle.

Batch jobs get a grant at creation and a record at completion

The controller watches batch/v1 Jobs labeled chio.world/governed: "true". It mints a capability through the sidecar at creation, collects per-pod observations, and on completion or failure releases the grant and submits a JobReceipt.

$ kubectl apply -f $REL/k8s-controller.yaml
$ kubectl apply -f $REL/k8s-controller-rbac.yaml

namespace/chio-system created
serviceaccount/chio-k8s-controller created
deployment.apps/chio-k8s-controller created
clusterrole.rbac.authorization.k8s.io/chio-k8s-controller created
clusterrolebinding.rbac.authorization.k8s.io/chio-k8s-controller created

You should see the namespace, service account, deployment, and RBAC objects created. The manifest names ghcr.io/backbay-labs/chio-k8s-controller:0.2.0; pin its digest for your deployment. The controller reaches the sidecar at CHIO_SIDECAR_URL, and when the sidecar is unreachable at mint time it requeues with backoff and stores no capability until a mint succeeds, so the Job's pods carry no token and admission refuses them.

A JobReceipt carries pod names, phases, and observation times, with no signature field, and receipt-like pod annotations are advisory input. The kernel's signed receipts remain the verifiable record of each call.

An admission failure names the check that stopped the pod

Every pod in the namespace is denied

The trust file has no webhook.kernel pin, or the required-scope list is empty. Pin the kernel with chio trust pin webhook.kernel, recreate the chio-trust ConfigMap, and set CHIO_WEBHOOK_REQUIRED_SCOPES on the webhook Deployment; an empty value fails closed by design.

Pods are denied while the webhook restarts

Both configurations use failurePolicy: Fail with a 10-second timeout, so the API server denies admission until the Service answers. Check GET /healthz and the serving certificate's caBundle.

The webhook process exits at startup

A pin in the trust file is not a well-formed Ed25519 key. chio trust pins --trust ./trust.toml lists each pin; correct the entry, and the process validates every key before it serves.

A pod with its own scope or exemption annotation is denied

Remove chio.world/required-scopes and chio.world/exempt from the pod. The webhook honors only its configured scopes.

The sidecar writes receipts to an unexpected file

The sidecar's --receipt-store flag takes an optional SQLite file path and reads a value with a URI scheme as a relative path. spec.sidecarConfig.receiptStore is typed as a URI and left unvalidated, so a value copied from it into the pod template can carry a scheme. Pass a filesystem path, as the ECS and Cloud Run manifests do.

Policy fields, webhooks, and environment

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": "<echoed from request.uid>",
    "allowed": false,
    "status": {
      "message": "webhook denies admission: capability token does not cover required scope \"quote:invoke\"",
      "code": 403
    }
  }
}

Next: issue the capabilities your pods carry