Embedded Node

Two embeddings, two different contracts

Chio ships two ways to enforce inside somebody else’s process, and they are not variants of one design. They share no code, no dependency, and no guarantee.

chio-tower is a tower::Layer. It wraps any tower::Service that speaks http::Request / http::Response, extracts caller identity, buffers and hashes the request body, and delegates the authorization decision to chio_http_core::HttpAuthority, which is backed by chio-kernel. A full kernel runs in the host process. Its Cargo.toml names four chio crates among its dependencies: chio-core-types, chio-http-core, chio-kernel, and chio-metrics-spec.

chio-envoy-ext-authz is a tonic service implementing envoy.service.auth.v3.Authorization/Check. It translates each Envoy CheckRequest into a ToolCallRequest, hands it to an EnvoyKernel implementation, and maps the returned Verdict back onto a CheckResponse. Its [dependencies] table lists tonic, prost, prost-types, tokio, async-trait, thiserror, tracing, sha2, and hex. No chio-* crate appears in it, and the crate’s architecture notes state the rule directly: "Internal: none."

One more scoping fact, because it changes how the Envoy path should be read. No crate in the workspace depends on chio-envoy-ext-authz, and no binary target builds a server around it. The crate is a workspace member and a library. The design document that describes the wider integration, docs/protocols/ENVOY-EXT-AUTHZ-INTEGRATION.md, carries the status line "Tier 0 -- proposed April 2026" and describes a header-injection scheme, an HTTP-mode adapter, and a shadow mode that the shipped crate does not implement. Read the crate for what exists; read the protocol document for intent.

What ChioLayer needs from the host

The layer is a configuration wrapper. ChioLayer::layer builds a ChioService<S> from a cloned ChioEvaluator and the configured body cap; every decision lives in the service and the evaluator. Four constructors exist, and the difference between the first two is the difference between a deployment that works and one that denies every side-effecting call.

The trait bounds on Service<http::Request<ReqBody>> are the real compatibility contract, and one of them is unusual. The request body must implement From<Bytes>, because the middleware reconstructs a fresh body from the bytes it buffered and hands that to the inner service.

impl<S, ReqBody, ResBody> Service<http::Request<ReqBody>> for ChioService<S>
where
    S: Service<http::Request<ReqBody>, Response = http::Response<ResBody>> + Clone + Send + 'static,
    S::Future: Send,
    S::Error: Into<Box<dyn std::error::Error + Send + Sync>>,
    ReqBody: Body + From<Bytes> + Send + 'static,
    ReqBody::Data: Send,
    ReqBody::Error: Into<Box<dyn std::error::Error + Send + Sync>>,
    ResBody: Default + From<Bytes> + Send + 'static,

axum::body::Body and http_body_util::Full<Bytes> satisfy it, and both are exercised by the crate’s integration tests. Real tonic::body::Body replay is not. The crate root says so in as many words, and tests/tonic_integration.rs opens by naming itself an approximation: it drives Full<Bytes> bodies with gRPC-shaped headers and a /my.service.MyService/GetItem path, which proves the middleware handles gRPC-looking HTTP, not that it can wrap a live tonic service.

Two hooks are plain function pointers, not closures. IdentityExtractor is fn(&http::HeaderMap) -> CallerIdentity and RouteResolver is fn(&str, &str) -> String. Neither can capture state, so anything an extractor needs has to arrive in the headers or be compiled in. The default resolver returns the raw path unchanged, which means an unconfigured deployment records /pets/secret-id as its route_pattern rather than a template.

The default extractor reads three header families in a fixed order and hashes the secret before it becomes an identity: Authorization: Bearer to bearer:<first 16 hex>, then x-api-key to apikey:<first 16 hex>, then the first Cookie pair to cookie:<first 16 hex>, then anonymous. The bearer and API-key branches run their value through valid_secret_value, which rejects anything blank or anything that differs from its own trimmed form, so Bearer falls through and resolves anonymous. The cookie branch is looser: it trims the first pair’s value itself and falls through only when nothing is left, so a padded cookie value still becomes an identity, hashed over the trimmed bytes. Every one of these sets verified: false; the extractor validates nothing, it only records what was presented.

The body cap is enforced frame by frame

The middleware has to read the request body to compute the content hash the receipt binds to, which makes body size a memory-safety question before it is a policy question. DEFAULT_MAX_BODY_BYTES is 8 MiB, and with_max_body_bytes on either the layer or the service overrides it.

Two guards run, and the second is the one that matters. buffer_request_body first checks size_hint().upper() and rejects an advertised size above the cap without pulling a byte. A body that advertises nothing, or lies, reaches the loop.

loop {
    let frame = std::future::poll_fn(|cx| body.as_mut().poll_frame(cx)).await;
    let frame = match frame {
        Some(Ok(frame)) => frame,
        Some(Err(error)) => return Err(BufferBodyError::Inner(error.into())),
        None => break,
    };
    let mut data = match frame.into_data() {
        Ok(data) => data,
        // Trailers and other non-data frames carry no body bytes and are
        // dropped: the kernel's content hash and length only cover the
        // request payload, not HTTP/2 trailers.
        Err(_non_data) => continue,
    };
    let chunk_len = data.remaining();
    if buffer.len().saturating_add(chunk_len) > limit {
        return Err(BufferBodyError::TooLarge);
    }
    while data.has_remaining() {
        let slice = data.chunk();
        let slice_len = slice.len();
        buffer.extend_from_slice(slice);
        data.advance(slice_len);
    }
}

Peak buffer use is bounded by the cap plus one in-flight frame. A collect() would not be: a streaming POST whose size hint hides its true length would be fully materialized before anyone measured it. buffer_request_body_aborts_streaming_body_during_collection pins exactly that, with a hand-written Body that returns four 8-byte frames under a default (unbounded) size hint against a 16-byte cap, and asserts the third frame aborts the read.

Three consequences of the loop are worth stating plainly. Non-data frames are skipped, so HTTP/2 trailers are excluded from both body_hash and body_length: the hash covers the payload and nothing else. An empty body produces body_hash: None rather than the SHA-256 of the empty string. And the buffered bytes are replayed byte-identical through ReqBody::from(collected), which buffer_request_body_hashes_and_replays_raw_bytes asserts against the original payload.

An oversized body gets a signed 413, or nothing at all

A transport-level rejection still produces evidence. The middleware resolves the caller identity before pulling any body bytes, precisely so a request that aborts during buffering can still carry a caller identity hash into a signed deny receipt. build_payload_too_large_response constructs the 413, and the deny it signs carries the guard name chio_tower_request_body_limit_guard and the reason request body exceeds {max_body_bytes}-byte limit for chio_tower.

On the full path the response carries three copies of the same fact: the receipt id in x-chio-receipt-id, the serialized HttpReceipt as a application/json body, and the receipt itself in the response extensions. service_413_response_has_signed_receipt_body asserts all three agree and that both the extension receipt and the deserialized body receipt verify under the embedded kernel key. A signed 413 is what lets an auditor tell a Chio decision from a stray network-layer rejection.

Three branches produce something else, and an operator reading a 413 should know which one they are looking at.

The second row is the one that reads wrong at first and is right on reflection. Refusing to sign a 413 because no store can record it is the same rule the normal request path applies: a denial’s audit record must be as durable as an allow’s, so a misconfigured deployment gets a server error rather than a signed rejection whose evidence was dropped. service_413_fails_closed_without_durable_receipts asserts the 502 and asserts the absence of both the extension and the header.

The fourth row is a real gap rather than a design choice. An unsupported method reaches the body guard before the route resolver runs, so an oversized BREW /pets gets a 413 with no evidence attached at all; service_413_unsupported_method_rejects_before_route_resolver pins that ordering with a resolver that panics if reached. parse_method accepts GET, POST, PUT, PATCH, DELETE, HEAD, and OPTIONS, case-insensitively, and rejects everything else.

A signing failure lands in that same fourth row, and it behaves differently from the append failure one row above. The signing arm ends in .ok(), so a failed sign_transport_deny_receipt yields None and a bare 413 rather than a 502; the function’s own doc comment states the rule, that the deny path is never blocked by a signing error. A store that rejects the append fails the request closed. A keypair that cannot sign does not.

Duplicate capability query parameters force a deny

A capability may be presented in the x-chio-capability header or in the chio_capability query parameter, header first. The query form has a failure mode the header form does not: a query string is a list of pairs, and RequestMetadata collapses it into a HashMap where a repeated key keeps the last value. Trusting that collapse would let a caller present two capabilities and have the middleware silently pick one.

/// Invalid capability marker used to force fail-closed authority evaluation
/// when the transport presented an ambiguous query capability.
pub(crate) const DUPLICATE_QUERY_CAPABILITY_PRESENTATION: &str =
    "duplicate chio_capability query parameters";

pub(crate) fn presented_capability_override(&self) -> Option<&'static str> {
    self.has_duplicate_query_capability
        .then_some(DUPLICATE_QUERY_CAPABILITY_PRESENTATION)
}

The counting happens after percent-decoding, inside the url::form_urlencoded::parse loop, so chio%5Fcapability and chio_capability in one query string count as two occurrences of the same key. That case has its own test.

The override is not a local 403. It replaces the presented capability with a sentinel string that cannot parse as a capability token, so validate_presented_capability records an invalid_reason, and projected_verdict returns that reason as a deny under the CapabilityGuard label before it ever consults the policy mode:

fn projected_verdict(
    policy: HttpAuthorityPolicy,
    presented_capability: &PresentedCapabilityState,
) -> Verdict {
    if let Some(reason) = &presented_capability.invalid_reason {
        return Verdict::deny(reason, "CapabilityGuard");
    }

    match policy {
        HttpAuthorityPolicy::SessionAllow => Verdict::Allow,
        HttpAuthorityPolicy::DenyByDefault => match &presented_capability.capability_id {
            Some(_) => Verdict::Allow,
            None => Verdict::deny(
                "side-effect route requires a capability token",
                "CapabilityGuard",
            ),
        },
    }
}

The ordering is what makes this bite. Safe methods (GET, HEAD, OPTIONS) run under HttpAuthorityPolicy::SessionAllow and would otherwise allow; everything else runs under DenyByDefault. Because the invalid-reason branch precedes the match, a GET carrying duplicate capability parameters is denied 403 even though a GET carrying none would have been allowed. service_denies_duplicate_query_capability_even_when_one_value_is_valid drives exactly that: a GET whose two chio_capability values are one garbage string and one properly signed token, asserting 403, a receipt id header, and receipt.is_denied(). Presenting a valid capability alongside an invalid one is worse than presenting nothing.

A configured store that cannot record the receipt fails the request

Durable-by-default is enforced at four call sites, all of them the same two lines: sign, then persist_http_receipt, and propagate the error rather than swallowing it. The embedded kernel persists its own receipts to the configured store; these four are the outer HTTP receipts that ChioService signs at the edge and that would otherwise live only in the response extensions.

The decision receipt is skipped entirely when no durable sink is attached. That is deliberate: with nothing to persist to, there is no durable audit trail to guarantee, so the ephemeral path avoids the extra signing rather than fail a request closed on a store that does not exist. service_fails_closed_when_durable_http_receipt_append_fails drives the configured case with a store that accepts the kernel’s own first append and rejects every one after, standing in for a volume that fills up mid-request. Its inner service panics if called, and the test asserts that any response returned is a server error.

The fourth site deserves its own reading. Finalizing on the response status happens after the inner service has already produced a response, so an append failure there returns an error for a request whose effect completed. The audit record is missing and the caller sees a failure; the two disagree. Nothing in the middleware compensates or retries. That is the residual gap in durable-by-default for the embedded path, and sizing the receipt volume is the only mitigation the code offers.

The one path that lets a request through unenforced

Fail-open is opt-in, off by default, and narrower than its name suggests. It applies to exactly one branch: a prepare error. A deny verdict is never bypassed by it.

let prepared = match prepared {
    Ok(r) => r,
    Err(e) => {
        if evaluator.is_fail_open() {
            crate::metrics::record_fail_open_suspected("tower");
            tracing::warn!(
                error = %e,
                "Chio evaluation failed; fail-open enabled, forwarding request WITHOUT enforcement"
            );
            return inner.call(req).await.map_err(Into::into);
        }
        tracing::error!("Chio evaluation failed: {e}");
        let mut response = http::Response::new(ResBody::default());
        *response.status_mut() = http::StatusCode::BAD_GATEWAY;
        return Ok(response);
    }
};

What reaches that branch is worth enumerating, because it is a wider set than "the kernel crashed". A caller identity that cannot be hashed, a content-hash failure, a kernel error, and a request that requires human approval all arrive as ChioTowerError converted from HttpAuthorityError. So does the durability refusal: a store-less, non-ephemeral evaluator produces a kernel error here, which means turning fail-open on converts a misconfigured deployment from "502 on every call" to "every call forwarded with no enforcement at all". An unsupported HTTP method reaches the same branch by a shorter route: parse_method fails inside the evaluator before HttpAuthority is consulted at all, returning ChioTowerError::Evaluation directly.

Receipt signing is not in the set. prepare never signs: sign_decision_receipt, finalize_receipt, and sign_transport_deny_receipt all run after it returns. On the normal request path the service propagates each with ?, so a signing failure there fails the request closed whatever fail_open is set to.

Every bypass increments chio_fail_open_suspected_total{surface="tower"}, and ChioService::new seeds the series at zero through a std::sync::Once so an absence-based alert fires on a scrape gap rather than on a deployment that has never failed open. The counter family lives in chio-metrics-spec as CHIO_FAIL_OPEN_SUSPECTED_TOTAL so a serving process can render it without depending on chio-tower. The tower layer is currently the only producer.

The second stack: dispatch with no HTTP shape

chio-tower also ships a stack that has nothing to do with HTTP. KernelService takes a KernelRequest (a chio_kernel::ToolCallRequest plus a TenantId) and dispatches straight to ChioKernel::evaluate_tool_call. Capability validation and receipt signing happen inside the kernel, not in this stack. build_layered(kernel, per_tenant_limit, request_timeout) composes, outermost first, KernelTraceLayer, a timeout-error normalizer, TimeoutLayer, TenantConcurrencyLimitLayer, and then KernelService.

The tenant limiter never waits. Each TenantId gets its own LoadShed<ConcurrencyLimit<S>> bucket, and a saturated bucket returns KernelServiceError::Overloaded immediately rather than queueing. The bucket table is bounded by DEFAULT_MAX_TENANT_CONCURRENCY_BUCKETS, 1024. A bucket becomes reapable only once it has sat idle past tenant_idle_reap_secs, 3600 by default, with no in-flight call, so a table of 1024 tenants all touched inside the last hour returns the distinct TenantTableFull variant even with nothing in flight. The in-flight condition is not a tie-breaker: a bucket with a live call is never reaped, because recreating its semaphore would let that tenant exceed its own limit. map_kernel_error routes a kernel RSS shed to the same Overloaded variant tower-side shedding produces, so both kinds of backpressure reach a caller as the same retryable signal.

The ext_authz check, and what it does not carry

ChioExtAuthzService::check is three steps with no branches worth hiding: translate, evaluate, respond. Translation failure and kernel failure both take the same exit.

let tool_call = match check_request_to_tool_call(&check) {
    Ok(call) => call,
    Err(err) => {
        warn!(error = %err, "ext_authz translation failed");
        return Ok(Response::new(fail_closed_response()));
    }
};

match self.kernel.evaluate(tool_call).await {
    Ok(verdict) => Ok(Response::new(verdict_to_response(&verdict))),
    Err(err) => {
        warn!(error = %err, "ext_authz kernel evaluation failed");
        Ok(Response::new(fail_closed_response()))
    }
}

check_request_to_tool_call derives a tool identity of the form http.<method>.<path segments>, lowercasing the method and joining non-empty path segments with dots. Any byte in a segment that is not ASCII alphanumeric or one of -, _, ~ is percent-escaped, which is what keeps /admin.list (http.get.admin%2Elist) distinct from /admin/list (http.get.admin.list). A root path yields http.post with no segment. Every call reports server_id as envoy, the value of ENVOY_SERVER_ID, so a policy can restrict itself to Envoy-originated traffic.

Secrets are removed before anything leaves translation. collect_policy_headers keeps a seven-entry allowlist (content-type, content-length, host, user-agent, x-request-id, x-chio-session-id, x-chio-source) and drops everything else, with authorization and x-chio-capability-token on an explicit strip list ahead of the allowlist check. A bearer token survives only as a SHA-256 hex digest and a bearer: subject prefix. Response headers a caller might spoof inbound (x-chio-receipt-id, x-chio-verdict, x-chio-denial-reason, x-chio-denial-guard) are not on the allowlist, and translate_strips_chio_internal_response_headers asserts each one is absent from the forwarded map.

Caller identity resolves in three steps and stops at the first hit: x-chio-capability-token, then Authorization: Bearer, then the mTLS peer principal Envoy reported. Only the mTLS branch sets verified: true, and it does so because Envoy asserted it, not because the adapter checked anything. The capability branch is the one to read carefully: it copies the header value verbatim into both subject and AuthMethod::Capability, with verified: false. The adapter never parses, verifies, or checks the revocation of a capability token. Validating it is the implementing kernel’s job.

Note also that the header name differs across the two embeddings. The Tower path reads x-chio-capability (and the chio_capability query parameter); the ext_authz path reads x-chio-capability-token. A client written against one will present nothing to the other.

Verdict to CheckResponse

Verdict::Allow becomes Code::Ok plus an OkHttpResponse with every mutation list empty: no headers added, none removed, no query parameters set. The allow path injects nothing into the upstream request. Verdict::Deny becomes Code::PermissionDenied plus a DeniedHttpResponse carrying x-chio-denial-reason and x-chio-denial-guard and a hand-built JSON body.

Two details in the deny mapping are load-bearing. Header values pass through sanitize_header_value, which replaces every control character with a space and substitutes unspecified for an empty result, so a guard name or denial reason cannot inject control bytes into a response header. And envoy_status_code maps an arbitrary status onto Envoy’s enum, falling back to Forbidden when it cannot represent the input faithfully; 418 and 451 are explicitly mapped to 403 for that reason. The mapped value, not the requested one, is what the chio.http_status metadata field reports, so the status the client sees and the status the access log records never disagree. deny_verdict_metadata_reports_admitted_envoy_status pins that with a requested status of 999 and asserts both read 403.

Every response carries dynamic_metadata under the chio. namespace for Envoy access logs: chio.verdict always, chio.denial_reason, chio.denial_guard, and chio.http_status on a deny, and chio.fail_closed = true on the fail-closed path.

The fail-closed response is a fixed 500 under Code::Internal with the reason ext_authz request denied fail closed and the guard fail_closed. It is the same object for a malformed CheckRequest and for a kernel that returned an error, and it never carries the underlying fault: the real error reaches only the tracing::warn. kernel_error_redacts_internal_message_from_denied_response drives that with the error text db password=secret-token unavailable and asserts it appears in neither the body, the header, nor any metadata field. And missing_attributes_fails_closed_without_kernel_call asserts a CheckRequest with no attributes short-circuits with the kernel call count still at zero.

Guarantees and limits

Next Steps

• HTTP Framework Middleware · the wiring this page assumes: adding the layer to an axum router, custom identity, custom route patterns
• In-Process Library · the kernel the Tower layer delegates to, its embedded signing key, and what a host process owns at startup
• Envoy ext_authz · filter configuration, mesh registration, and the deployment topologies for the adapter
• Fail-Closed Semantics · the rule the durability gates implement, stated once for every enforcement path
• Sidecar HTTP Service · the alternative when the host process is not Rust, and what changes when the decision crosses a socket