BuildOperations
Istio ext_authz
Use Chio as an Istio ext_authz provider to allow or deny matched mesh traffic before Envoy forwards a request.
Where the code lives
examples/istio-ext-authz/. The reference manifests target Istio 1.22+ on Kubernetes 1.28+. See ci-validation.md in the example for static-validation recipes (kubeconform, istioctl analyze) you can run without a live cluster.What It Shows
- The chio Envoy ext_authz adapter (
chio-envoy-ext-authz) deployed as a cluster-wideDeploymentin thechio-systemnamespace. - Chio registered under
MeshConfig.extensionProviderswith the gRPC backend atchio-sidecar.chio-system.svc.cluster.local:9091. - Three AuthorizationPolicy objects: a
CUSTOMpolicy that opts matched workloads into chio, aDENYbackstop for unauthenticated requests, and anALLOWrule that lets kubelet probes through. - A demo workload in the
agent-toolsnamespace that opts in viachio.world/secured=true. - A test harness that requires an authenticated POST to return HTTP 200 with an
x-chio-receipt-idheader, and an unauthenticated POST to return 403 from the backstop. The header is a contract on the adapter image you build, not something the ext_authz crate in this repository emits; see what the adapter returns on allow.
Architecture
Envoy in the demo workload pod is configured with the envoyExtAuthzGrpc backend pointing at the Chio sidecar Service. For each matched request, Envoy issues a gRPC CheckRequest against chio. Chio runs the kernel pipeline, signs a receipt, and responds with allow plus header mutations or deny. Envoy then forwards or rejects.
Client
|
v POST /tools/hello (with x-chio-capability-token)
+---------------------+
| Envoy sidecar | <-- AuthorizationPolicy CUSTOM matches
| (demo-tool pod) | and routes ext_authz to chio
+---------------------+
| gRPC Authorization/Check (port 9091)
v
+----------------------------+
| chio-sidecar Deployment | chio-system namespace
| chio-envoy-ext-authz | Service: chio-sidecar:9091/gRPC, 9090/HTTP
| (kernel pipeline + signer) | /health on 9090
+----------------------------+
| CheckResponse: OK, no response headers,
| dynamic metadata chio.verdict = "allow"
v
+---------------------+
| Envoy sidecar | forwards upstream
+---------------------+
|
v
demo-tool app (go-httpbin)Prerequisites
- Istio 1.22+ (the typed-header forwarding used here stabilized in 1.22).
- Kubernetes 1.28+ (required for the
security.istio.io/v1andnetworking.istio.io/v1GA APIs the policies use). kubectlwith cluster access;istioctl1.22+;curlandawkon the workstation that runs the harness.- A dedicated Chio Envoy ext_authz adapter image. The reference manifest uses
ghcr.io/backbay-labs/chio-ext-authz:latestas a placeholder. Replace it with the adapter image you built and published. Do not use the genericghcr.io/backbay-labs/chio-sidecarimage: that is the HTTP sidecar and does not expose the gRPCAuthorization/Checkservice on:9091. - A capability token for the demo workload (or the demo token chio accepts in shadow mode). Export it as
CHIO_DEMO_CAPABILITY_TOKENbefore running the harness.
Files
examples/istio-ext-authz/
00-chio-sidecar-deployment.yaml Namespace, ServiceAccount, Deployment,
ConfigMap, Secret, Service
01-meshconfig-patch.yaml IstioOperator overlay; registers chio-ext-authz
02-authorization-policy.yaml CUSTOM (opt-in), DENY (backstop), ALLOW (probes)
03-demo-workload.yaml Namespace, ServiceAccount, Deployment,
Service, VirtualService (go-httpbin demo)
README.md
ci-validation.md kubeconform + istioctl analyze recipes
test-harness.sh port-forward + curl harnessStep 1: Deploy the Chio Sidecar
kubectl apply -f examples/istio-ext-authz/00-chio-sidecar-deployment.yaml
kubectl -n chio-system rollout status deploy/chio-sidecar --timeout=120s
kubectl -n chio-system get svc chio-sidecarThe Service is a ClusterIP with two named ports: grpc-ext-authz on 9091 with appProtocol: grpc, and http-health on 9090 with appProtocol: http (00-chio-sidecar-deployment.yaml:206-230). Istio reads only the first; the second is what the three probes use.
Smoke-test the adapter from inside the cluster before wiring Istio at it:
kubectl run chio-smoke --rm -it --restart=Never \
--image=curlimages/curl:8.9.1 -- \
curl -sS http://chio-sidecar.chio-system.svc.cluster.local:9090/healthA 200 is what the startup, readiness and liveness probes require on /health (00-chio-sidecar-deployment.yaml:122-146), so a pod that never answers there never becomes ready. That endpoint is part of the contract on the adapter image, not something this repository ships: chio-envoy-ext-authz declares a [lib] and nothing else, with no binary and no HTTP server. The body it returns is yours to choose.
Fail-closed by design
00-chio-sidecar-deployment.yaml:11-14). Kubernetes restarts it, the readiness probe on /health keeps failing, and the provider config sends the refusal back as a 403 through statusOnError: "403" (01-meshconfig-patch.yaml:65). An unavailable Chio adapter results in denial.Step 2: Register Chio in MeshConfig
Pick the install path that matches how your mesh was provisioned.
Fresh install:
istioctl install -y \
-f examples/istio-ext-authz/01-meshconfig-patch.yamlExisting mesh managed via IstioOperator:
kubectl -n istio-system apply \
-f examples/istio-ext-authz/01-meshconfig-patch.yaml
kubectl -n istio-system rollout restart deploy/istiodThe patch registers chio under meshConfig.extensionProviders:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: chio-ext-authz-meshconfig
namespace: istio-system
spec:
# Minimum Istio version. Earlier 1.20/1.21 builds also work for gRPC
# ext_authz but 1.22 stabilised the typed header forwarding knobs below.
meshConfig:
extensionProviders:
- name: chio-ext-authz
envoyExtAuthzGrpc:
# Fully-qualified cluster DNS name of the Service created in
# 00-chio-sidecar-deployment.yaml.
service: chio-sidecar.chio-system.svc.cluster.local
port: 9091
# Chio evaluation budget. Increase if your guard pipeline is slow.
timeout: 0.25s
# Forward identity + capability headers into the CheckRequest.
# `chio-envoy-ext-authz::translate` pulls identity from these names.
includeRequestHeadersInCheck:
- authorization
- x-chio-capability-token
- x-chio-session-id
- x-request-id
# Chio never stores or forwards raw request bodies, but small
# bodies must be available for guards that inspect content.
includeAdditionalHeadersInCheck:
x-chio-source: istio-mesh
# Status propagated back to the client on Chio failure. See
# docs/protocols/ENVOY-EXT-AUTHZ-INTEGRATION.md section 11.3.
statusOnError: "403"
# Inject the Envoy downstream peer certificate so Chio can derive a
# SPIFFE-style CallerIdentity from the mTLS handshake.
includePeerCertificate: trueVerify the provider was picked up:
kubectl -n istio-system logs deploy/istiod | grep -i extensionprovider
istioctl proxy-config bootstrap -n istio-system deploy/istiod \
| grep -A3 chio-ext-authzStep 3: Deploy the Demo Workload and Policy
kubectl apply -f examples/istio-ext-authz/03-demo-workload.yaml
kubectl -n agent-tools rollout status deploy/demo-tool --timeout=120s
kubectl apply -f examples/istio-ext-authz/02-authorization-policy.yaml
kubectl -n agent-tools get authorizationpoliciesThree policies land in agent-tools: chio-tool-authorization (CUSTOM), chio-deny-unauthenticated (DENY), and chio-allow-health-probes (ALLOW). Istio evaluates CUSTOM first, then DENY, then ALLOW. An unauthenticated request matches none of the three CUSTOM rules, so it is never sent to Chio at all; the DENY backstop is what refuses it.
All three select on the same label, chio.world/secured=true, so a workload opts in once and gets all of them. The CUSTOM policy's three rules are ORed: POSTs to /tools/* or /invoke/* with a non-empty x-chio-capability-token, the same POSTs with a Bearer Authorization header, and GETs on /tools/* with one. The whole file, comments included:
# Route mesh traffic through Chio via Istio's AuthorizationPolicy CUSTOM action.
#
# The CUSTOM action delegates the allow/deny decision to the named provider
# registered in MeshConfig (see 01-meshconfig-patch.yaml). This file contains
# three policies:
#
# 1. chio-tool-authorization -- opts workloads labelled
# `chio.world/secured=true` in the
# `agent-tools` namespace into Chio
# evaluation, with a header-driven filter
# so only capability-bearing requests pay
# the ext_authz latency.
# 2. chio-deny-unauthenticated -- fail-closed backstop: anonymous requests
# that slip past the CUSTOM match above
# are denied outright. Ensures there is
# no bypass path when the capability
# header is missing.
# 3. chio-allow-health-probes -- excludes kubelet probe traffic so the
# demo workload's `/healthz` path stays
# reachable without a capability token.
#
# All policies use `security.istio.io/v1` (GA in Istio 1.22+). The CUSTOM
# action uses the provider:name field -- the deprecated
# `.external.httpAuthorizationService` style is explicitly NOT used.
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: chio-tool-authorization
namespace: agent-tools
labels:
app.kubernetes.io/part-of: chio-protocol
spec:
# Apply only to workloads that opted in by labelling themselves.
selector:
matchLabels:
chio.world/secured: "true"
action: CUSTOM
provider:
name: chio-ext-authz
rules:
# Allow-rule #1: route POSTs to /tools/* through Chio when either an
# Chio capability token header or a Bearer Authorization header is
# present. This is the hot path for agent tool invocation.
- to:
- operation:
methods: ["POST"]
paths: ["/tools/*", "/invoke/*"]
when:
- key: request.headers[x-chio-capability-token]
notValues: [""]
# Allow-rule #2: route POSTs through Chio when the request authenticates
# with a Bearer token instead of a capability header.
- to:
- operation:
methods: ["POST"]
paths: ["/tools/*", "/invoke/*"]
when:
- key: request.headers[authorization]
values: ["Bearer *"]
# Allow-rule #3: route GETs on /tools/* (listings, schema fetches)
# through Chio only when an Authorization header is present. This
# illustrates header-based routing -- unauthenticated GETs fall
# through to the deny policy below.
- to:
- operation:
methods: ["GET"]
paths: ["/tools/*"]
when:
- key: request.headers[authorization]
values: ["Bearer *"]
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: chio-deny-unauthenticated
namespace: agent-tools
labels:
app.kubernetes.io/part-of: chio-protocol
spec:
selector:
matchLabels:
chio.world/secured: "true"
action: DENY
rules:
# Anything hitting /tools/* without either header is denied before it
# ever reaches Chio. Saves the ext_authz RTT and keeps receipts focused
# on real authenticated traffic.
- to:
- operation:
paths: ["/tools/*", "/invoke/*"]
when:
- key: request.headers[x-chio-capability-token]
values: [""]
- key: request.headers[authorization]
notValues: ["Bearer *"]
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: chio-allow-health-probes
namespace: agent-tools
labels:
app.kubernetes.io/part-of: chio-protocol
spec:
selector:
matchLabels:
chio.world/secured: "true"
action: ALLOW
rules:
- to:
- operation:
methods: ["GET"]
paths: ["/healthz", "/readyz", "/metrics"]Confirm Envoy on the demo pod picked up the provider reference:
POD="$(kubectl -n agent-tools get pod \
-l app.kubernetes.io/name=demo-tool \
-o jsonpath='{.items[0].metadata.name}')"
istioctl proxy-config listener -n agent-tools "${POD}" \
--port 8080 -o json | grep chio-ext-authzStep 4: Run the Harness
export CHIO_DEMO_CAPABILITY_TOKEN="$(cat ~/.chio/demo.token)"
./examples/istio-ext-authz/test-harness.shThe harness:
- Opens a
kubectl port-forwardtosvc/demo-tool:80. - Sends
POST /tools/hellowithx-chio-capability-tokenand anauthorization: Bearerheader, and requires HTTP 200 with a non-emptyx-chio-receipt-idresponse header. - Sends the same request without credentials and asserts HTTP 403.
On success it prints a four-line summary built from ARTIFACT_DIR, RECEIPT_ID and DENY_STATUS (test-harness.sh:142-148). The receipt id it echoes is whatever the adapter put on the header; a Chio receipt id is 64 lowercase hex characters (crates/core/chio-core-types/src/receipt/body.rs:240-243).
Manual curl Verification
kubectl -n agent-tools port-forward svc/demo-tool 18080:80 &
# Allow path: the CUSTOM rule matches, so chio evaluates
curl -i -X POST \
-H "x-chio-capability-token: ${CHIO_DEMO_CAPABILITY_TOKEN}" \
--data '{"hello":"world"}' \
http://127.0.0.1:18080/tools/hello
# Deny path: no header, so no CUSTOM rule matches and the DENY policy fires
curl -i -X POST --data '{}' http://127.0.0.1:18080/tools/helloWhat the adapter returns on allow
On an allow the ext_authz crate in this repository returns an OkHttpResponse with headers, headers_to_remove and response_headers_to_add all empty (crates/protocol/chio-envoy-ext-authz/src/response.rs:24-37). Its only allow-side output is one Envoy dynamic-metadata field, chio.verdict = "allow" (src/metadata.rs:11-13), which reaches access logs and downstream filters and is not an HTTP header. So a 200 through this adapter carries no x-chio-* header of its own.
The harness asserts a header this crate does not set
test-harness.sh fails the run when the allow response has no x-chio-receipt-id header. If you want the receipt id on the wire, the adapter image you build has to add it on the allow path, or a downstream Envoy filter has to lift chio.verdict and the receipt id out of dynamic metadata. Either way it is your adapter's behaviour, not this crate's.The deny path does write headers. The Verdict::Deny arm builds x-chio-denial-reason and x-chio-denial-guard onto the DeniedHttpResponse, with a body of {"verdict":"deny","reason":...,"guard":...} (response.rs:38-65). The guard name is whatever guard refused; the crate's own tests use ScopeGuard (response.rs:253-280). A fail-closed refusal, meaning translation or kernel evaluation itself errored, is different again: HTTP 500, x-chio-denial-reason: ext_authz request denied fail closed as the only header, and a body whose guard reads fail_closed (response.rs:73-85).
The deny in the curl above is not one of those. No CUSTOM rule matches a request with no capability header, so Chio never runs and the 403 comes from Envoy enforcing the chio-deny-unauthenticated policy. It carries no x-chio-* header for the same reason: nothing in Chio was asked.
Smoke Assertions
The test harness checks the allow and deny cases. From test-harness.sh:
log "issuing allow request (capability token present)..."
ALLOW_STATUS="$(
"${CURL}" -sS \
-o "${ALLOW_BODY}" \
-D "${ALLOW_HEADERS}" \
-w '%{http_code}' \
-H "x-chio-capability-token: ${CAPABILITY_TOKEN}" \
-H "authorization: Bearer ${CAPABILITY_TOKEN}" \
-X POST \
--data '{"hello":"world"}' \
"http://127.0.0.1:${LOCAL_PORT}/tools/hello"
)"
if [[ "${ALLOW_STATUS}" != "200" ]]; then
log "allow response headers:"
cat "${ALLOW_HEADERS}" >&2 || true
fail "expected HTTP 200 on allow, got ${ALLOW_STATUS}"
fi
# Header names are case-insensitive per RFC 7230. Match loosely.
RECEIPT_ID="$(
awk 'BEGIN{IGNORECASE=1} /^x-chio-receipt-id:/ { sub(/\r$/, ""); print $2; exit }' \
"${ALLOW_HEADERS}"
)"
if [[ -z "${RECEIPT_ID}" ]]; then
log "allow response headers:"
cat "${ALLOW_HEADERS}" >&2 || true
fail "allow response missing x-chio-receipt-id header"
fi
log "allow OK: status=200 receipt=${RECEIPT_ID}"
log "issuing deny request (no credentials)..."
DENY_STATUS="$(
"${CURL}" -sS \
-o "${DENY_BODY}" \
-D "${DENY_HEADERS}" \
-w '%{http_code}' \
-X POST \
--data '{"hello":"denied"}' \
"http://127.0.0.1:${LOCAL_PORT}/tools/hello"
)"
if [[ "${DENY_STATUS}" != "403" ]]; then
log "deny response headers:"
cat "${DENY_HEADERS}" >&2 || true
fail "expected HTTP 403 on deny, got ${DENY_STATUS}"
fi
log "deny OK: status=403"Decision rule
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| All requests 403, including authenticated | Chio pod not ready or MeshConfig not reloaded | kubectl -n chio-system get pods; istioctl proxy-config bootstrap ... | grep chio-ext-authz |
Allow missing x-chio-receipt-id | Expected. The allow path sets no response header; see above | Read chio.verdict from Envoy dynamic metadata, or have your adapter image add the header |
| AuthorizationPolicy rejected at apply | API version mismatch (v1beta1 vs v1) | Cluster on Istio 1.22+ and serving security.istio.io/v1 |
| Authenticated POST reaches the app but Chio never ran | Pod not labelled chio.world/secured=true, so the CUSTOM policy selector does not match it | kubectl -n agent-tools get pod -l app.kubernetes.io/name=demo-tool --show-labels |
| 503 from demo pod | Istio sidecar injection disabled on agent-tools | kubectl label ns agent-tools istio-injection=enabled --overwrite |
Teardown
kubectl delete -f examples/istio-ext-authz/02-authorization-policy.yaml
kubectl delete -f examples/istio-ext-authz/03-demo-workload.yaml
kubectl delete -f examples/istio-ext-authz/00-chio-sidecar-deployment.yaml
# Optional: remove the extension provider from MeshConfig by editing the
# istio ConfigMap (or reinstall without 01-meshconfig-patch.yaml).Where to read more