Embedded Node
Run the kernel inside an HTTP service so each request is admitted, bound to its bytes, and recorded before the handler runs.
On this page
An HTTP service can carry its own kernel. The Tower layer checks each request’s authority in process, binds the body the handler will receive, and records a signed decision before the handler runs.
The service keeps its router and handlers. ChioLayer wraps them, and an in-process kernel answers through HttpAuthority. Envoy deployments use the ext_authz adapter instead, which translates each Envoy check for a kernel you supply. HTTP Framework Middleware and Envoy ext_authz cover the wiring.
The decision is on disk before the handler runs
With a durable receipt sink attached, an allowed request reaches the handler only after its signed decision record is appended. The final record, which binds the response status, comes after the effect.
The decision record lands before the handler runs
With a durable sink attached, the layer appends the signed HTTP decision receipt and calls the handler only after the append succeeds. A second receipt binds the response status once the handler returns. Node State on Disk.
The line between the two records is what an operator reads. Everything before it can still refuse with nothing run: an oversized body, a deny verdict, a store that rejects the decision append. After it, the handler’s effect has happened. If the final append fails, the caller receives an error for a request that completed, so treat that outcome as unknown and reconcile it before you retry a side effect.
The kernel persists its own receipts to the configured store. The layer adds the outer HTTP receipts it signs at the edge, at four sites: the transport deny, the final receipt on a deny verdict, the decision receipt on an allow, and the final receipt on the response status.
Read more in Unknown Outcomes.
Missing durable state fails closed
The constructor decides where receipts go, and a layer with nowhere to record a decision refuses to run side effects. Choose the constructor by the state you attach.
| Constructor | Receipt sink | A side-effecting call |
|---|---|---|
ChioLayer::builder | receipt_store, revocation_store | Runs, with durable records |
ChioLayer::from_evaluator | Whatever the evaluator carries | Follows the evaluator |
ChioLayer::new_ephemeral | In memory, opt-in taken | Runs; receipts vanish on restart |
ChioLayer::new | None | Refused, 502 |
build is fallible because attaching a durable receipt store hydrates checkpoint counters. from_evaluator is the route for a custom IdentityExtractor or RouteResolver. The ephemeral layer keeps its receipt log and revocation store in memory and returns receipts in the response extensions.
fail_open is off by default. Enabling it forwards any preparation error to the handler without enforcement: a missing durable store, an unsupported method, a pending approval. A deny verdict is never bypassed by it, and each bypass increments chio_fail_open_suspected_total{surface="tower"}.
The receipt binds the bytes the handler receives
The layer reads the body before evaluation, hashes it, and hands the handler a body rebuilt from the same bytes. The kernel decides on exactly what the handler will see.
- Buffer. Data frames are collected up to the cap, 8 MiB by default. The cap is enforced while collecting, so a false or absent size hint does not lift it, and peak memory stays at the cap plus one in-flight frame.
- Hash. The payload alone is hashed. Trailers and other non-data frames are excluded from
body_hashandbody_length, and an empty body producesbody_hash: None. - Replay. The handler receives
ReqBody::from(collected), byte-identical to what was hashed.
A body over the cap stops before evaluation, and the handler never runs. The layer signs a deny receipt and answers 413 with the receipt id in x-chio-receipt-id, the serialized HttpReceipt as the JSON body, and the receipt in the response extensions. A signed 413 is how an auditor tells a Chio decision from a network-layer rejection.
Because the layer buffers the whole body to bind it, raising max_body_bytes raises the per-request memory ceiling, multiplied by concurrency. The node’s own listener ceilings are a separate set, in Backpressure & Limits.
An ambiguous capability is refused, even beside a valid one
A caller presents a capability in the x-chio-capability header or the chio_capability query parameter, header first. Two query values force a deny, whatever either value holds.
fn projected_verdict(
policy: HttpAuthorityPolicy,
presented_capability: &PresentedCapabilityState,
) -> Verdict {
if let Some(reason) = &presented_capability.invalid_reason {
return Verdict::deny(reason, "CapabilityGuard");
}
match policy {
HttpAuthorityPolicy::SessionAllow => Verdict::Allow,
HttpAuthorityPolicy::DenyByDefault => match &presented_capability.capability_id {
Some(_) => Verdict::Allow,
None => Verdict::deny(
"side-effect route requires a capability token",
"CapabilityGuard",
),
},
}
}A duplicate replaces the presented capability with an unparseable sentinel, so the invalid-reason branch denies under CapabilityGuard before the policy match. Safe methods (GET, HEAD, OPTIONS) run under SessionAllow and everything else under DenyByDefault, yet a GET with two capability values still receives 403 and a signed deny receipt, even when one value is a valid token.
Counting runs after percent-decoding, so chio%5Fcapability and chio_capability are the same key. Only that key is counted. Other repeated query keys collapse last-write-wins into the map that feeds the content hash, so a policy that reads a repeatable parameter should treat it as unbound.
Envoy hands each check to a kernel you supply
The ext_authz adapter turns an Envoy CheckRequest into a tool-call-shaped request and maps the verdict back to a CheckResponse. Your EnvoyKernel implementation validates the capability, applies policy, and signs the receipt; the gRPC server that hosts it is yours too.
let tool_call = match check_request_to_tool_call(&check) {
Ok(call) => call,
Err(err) => {
warn!(error = %err, "ext_authz translation failed");
return Ok(Response::new(fail_closed_response()));
}
};
match self.kernel.evaluate(tool_call).await {
Ok(verdict) => Ok(Response::new(verdict_to_response(&verdict))),
Err(err) => {
warn!(error = %err, "ext_authz kernel evaluation failed");
Ok(Response::new(fail_closed_response()))
}
}Translation failure and kernel failure take the same exit, a fixed 500 that never carries the underlying fault. Secrets leave translation as hashes or not at all. The hash covers the body bytes Envoy supplied, and declared length comes from a separate field, so configure Envoy’s request-body handling to send the complete body when the decision depends on it.
The two embeddings read different headers. Tower reads x-chio-capability and the chio_capability query parameter; ext_authz reads x-chio-capability-token. A client written against one presents nothing to the other.
Embedding reference
Try 413, fail_open, KernelService, or CheckResponse.
Service bounds
ChioLayer::layer builds a ChioService<S> from a cloned ChioEvaluator and the configured body cap; every decision lives in the service and the evaluator. The trait bounds are the compatibility contract. The request body implements From<Bytes> because the layer rebuilds it from the buffered bytes, and the response body implements Default + From<Bytes>.
impl<S, ReqBody, ResBody> Service<http::Request<ReqBody>> for ChioService<S>
where
S: Service<http::Request<ReqBody>, Response = http::Response<ResBody>> + Clone + Send + 'static,
S::Future: Send,
S::Error: Into<Box<dyn std::error::Error + Send + Sync>>,
ReqBody: Body + From<Bytes> + Send + 'static,
ReqBody::Data: Send,
ReqBody::Error: Into<Box<dyn std::error::Error + Send + Sync>>,
ResBody: Default + From<Bytes> + Send + 'static,axum::body::Body and http_body_util::Full<Bytes> satisfy these bounds, and the crate’s integration tests exercise both. tests/tonic_integration.rs drives Full<Bytes> bodies with gRPC-shaped headers and a /my.service.MyService/GetItem path.
Identity extraction and route mapping
IdentityExtractor is fn(&http::HeaderMap) -> CallerIdentity and RouteResolver is fn(&str, &str) -> String. Both are plain function pointers that capture no state, so anything an extractor needs arrives in the headers or is compiled in. The default resolver returns the raw path, so an unconfigured deployment records /pets/secret-id as its route_pattern; configure a resolver to record templates.
Authorization: Bearerbecomesbearer:<first 16 hex>.x-api-keybecomesapikey:<first 16 hex>.- The first
Cookiepair becomescookie:<first 16 hex>; otherwise the caller is anonymous.
The default extractor reads those header families in that order and hashes each secret before it becomes an identity. The bearer and API-key branches pass their value through valid_secret_value, which rejects a blank value or one that differs from its trimmed form, so Bearer resolves anonymous. The cookie branch trims the first pair’s value itself and falls through only when nothing is left, so a padded cookie value becomes an identity hashed over the trimmed bytes. Every identity carries verified: false: the extractor records what was presented.
Body buffering
DEFAULT_MAX_BODY_BYTES is 8 MiB, and with_max_body_bytes on the layer or the service overrides it. buffer_request_body first checks size_hint().upper() and rejects an advertised size above the cap without reading a byte; a body that advertises nothing, or advertises less than it sends, is stopped by the loop.
loop {
let frame = std::future::poll_fn(|cx| body.as_mut().poll_frame(cx)).await;
let frame = match frame {
Some(Ok(frame)) => frame,
Some(Err(error)) => return Err(BufferBodyError::Inner(error.into())),
None => break,
};
let mut data = match frame.into_data() {
Ok(data) => data,
// Trailers and other non-data frames carry no body bytes and are
// dropped: the kernel's content hash and length only cover the
// request payload, not HTTP/2 trailers.
Err(_non_data) => continue,
};
let chunk_len = data.remaining();
if buffer.len().saturating_add(chunk_len) > limit {
return Err(BufferBodyError::TooLarge);
}
while data.has_remaining() {
let slice = data.chunk();
let slice_len = slice.len();
buffer.extend_from_slice(slice);
data.advance(slice_len);
}
}A collect() would materialize a streaming POST before measuring it. buffer_request_body_aborts_streaming_body_during_collection drives a body of four 8-byte frames under an unbounded size hint against a 16-byte cap and asserts the third frame aborts the read. buffer_request_body_hashes_and_replays_raw_bytes asserts the replayed bytes match the original payload. The layer has no streaming evaluation path: an upload larger than the cap is refused.
Oversized request responses
The layer resolves the caller identity before reading the body, so a request that aborts while buffering still carries a caller identity hash into its signed deny. build_payload_too_large_response builds the 413; its deny names the guard chio_tower_request_body_limit_guard and the reason request body exceeds {max_body_bytes}-byte limit for chio_tower.
| Condition | Status | Receipt |
|---|---|---|
Receipts audited, signing and append succeed | 413 | Header, JSON body, extension |
No durable sink and no ephemeral opt-in ( receipts_ false) | 502 | None: no header, body, or extension |
The configured store rejects the append | 502 | None; the signed 413 is discarded |
Method outside the seven parse_method accepts, identity hash fails, or sign_ fails | 413 | None: bare 413, no content-type, no header |
The 502 rows apply the normal request path’s rule: a denial’s record is as durable as an allow’s, so a deployment that cannot record the 413 returns a server error. parse_method accepts GET, POST, PUT, PATCH, DELETE, HEAD, and OPTIONS, case-insensitively. An unsupported method reaches the body guard before the route resolver runs, so an oversized BREW /pets gets the bare 413. The signing arm ends in .ok(), so a failed signature yields a bare 413 rather than a 502: the deny path is never blocked by a signing error, while a rejected append fails the request closed.
Persistence sites
Each site signs, then calls persist_http_receipt and propagates its error.
| Site | When it runs | Effect already happened |
|---|---|---|
Transport deny | Oversized body, before evaluation | No; an append failure turns the 413 into a 502 |
Final receipt on a deny | After evaluation, before any response | No; the inner service is never called |
Decision receipt | Allow verdict, when has_ | No; this append gates the inner service |
Final receipt on the response status | After the inner service returns | Yes; the request then fails |
The decision receipt is skipped when no durable sink is attached: with nothing to persist to, the ephemeral path saves the extra signature. persist_http_receipt first converts the HTTP receipt into a core receipt under the sink keypair. A receipt that cannot convert, such as one signed under a foreign kernel key, is skipped with a tracing::warn and the call returns Ok(()), because a durable log receives only records the store can re-verify; kernel and decision records written before it remain. Watch that warning in the logs.
The fail-open branch
let prepared = match prepared {
Ok(r) => r,
Err(e) => {
if evaluator.is_fail_open() {
crate::metrics::record_fail_open_suspected("tower");
tracing::warn!(
error = %e,
"Chio evaluation failed; fail-open enabled, forwarding request WITHOUT enforcement"
);
return inner.call(req).await.map_err(Into::into);
}
tracing::error!("Chio evaluation failed: {e}");
let mut response = http::Response::new(ResBody::default());
*response.status_mut() = http::StatusCode::BAD_GATEWAY;
return Ok(response);
}
};A caller identity that cannot be hashed, a content-hash failure, a kernel error, and a request that requires human approval all arrive here as ChioTowerError converted from HttpAuthorityError. So does the durability refusal of a store-less, non-ephemeral evaluator, and an unsupported method, which fails parse_method and returns ChioTowerError::Evaluation. Preparation calls kernel authorization and can produce kernel records. Errors in the later decision-receipt and final-receipt operations propagate, and oversized bodies take their own path before preparation.
ChioService::new seeds chio_fail_open_suspected_total at zero through a std::sync::Once, so an absence-based alert fires on a scrape gap. The family is registered in chio-metrics-spec as CHIO_FAIL_OPEN_SUSPECTED_TOTAL, so a serving process renders it without depending on chio-tower. The Tower layer produces it.
KernelService without an HTTP envelope
An application that builds ToolCallRequest values uses KernelService directly. It takes a KernelRequest, a chio_kernel::ToolCallRequest plus a TenantId, and dispatches to ChioKernel::evaluate_tool_call; capability validation and receipt signing happen inside the kernel. build_layered(kernel, per_tenant_limit, request_timeout) composes, outermost first, KernelTraceLayer, a timeout-error normalizer, TimeoutLayer, TenantConcurrencyLimitLayer, and KernelService.
The tenant limiter never queues. Each TenantId gets its own LoadShed<ConcurrencyLimit<S>> bucket, and a saturated bucket returns KernelServiceError::Overloaded at once. The bucket table holds DEFAULT_MAX_TENANT_CONCURRENCY_BUCKETS, 1024. A bucket is reapable once it has sat idle past tenant_idle_reap_secs, 3600 by default, with no call in flight, so 1024 tenants touched within the hour return TenantTableFull even with nothing in flight. A bucket with a live call is kept, because recreating its semaphore would let that tenant exceed its limit. map_kernel_error routes a kernel RSS shed to the same Overloaded variant, so both kinds of backpressure reach the caller as one retryable signal.
Envoy check translation
check_request_to_tool_call derives a tool identity of the form http.<method>.<path segments>, lowercasing the method and joining non-empty segments with dots. A byte outside ASCII alphanumerics and -, _, ~ is percent-escaped, which keeps /admin.list (http.get.admin%2Elist) distinct from /admin/list (http.get.admin.list). A root path yields http.post. Every call reports server_id as envoy, the value of ENVOY_SERVER_ID, so a policy can restrict itself to Envoy traffic.
collect_policy_headers keeps a seven-entry allowlist: content-type, content-length, host, user-agent, x-request-id, x-chio-session-id, and x-chio-source. authorization and x-chio-capability-token sit on a strip list ahead of the allowlist, and a bearer token survives only as a SHA-256 hex digest with a bearer: subject prefix. Response headers a caller might spoof inbound (x-chio-receipt-id, x-chio-verdict, x-chio-denial-reason, x-chio-denial-guard) are dropped, which translate_strips_chio_internal_response_headers asserts.
Caller identity resolves at the first hit: x-chio-capability-token, then Authorization: Bearer, then the mTLS peer principal Envoy reports. Only the mTLS branch sets verified: true, on Envoy’s assertion. The capability branch copies the header value verbatim into subject and AuthMethod::Capability with verified: false; parsing, verifying, and revocation-checking the token is the EnvoyKernel implementation’s job. The crate depends on no other Chio crate, defines the EnvoyKernel trait, and signs nothing itself.
Verdict to CheckResponse
Verdict::Allow becomes Code::Ok with an OkHttpResponse whose mutation lists are empty, so an allow forwards the request unchanged. Verdict::Deny becomes Code::PermissionDenied with a DeniedHttpResponse carrying x-chio-denial-reason, x-chio-denial-guard, and a JSON body. sanitize_header_value replaces control characters with spaces and substitutes unspecified for an empty result.
envoy_status_code maps a status onto Envoy’s enum and falls back to Forbidden when it cannot represent the input; 418 and 451 map to 403. The chio.http_status metadata field reports the mapped value, so the client’s status and the access log agree; deny_verdict_metadata_reports_admitted_envoy_status requests 999 and asserts both read 403. Every response carries dynamic_metadata under chio.: chio.verdict always; chio.denial_reason, chio.denial_guard, and chio.http_status on a deny; chio.fail_closed = true on the fail-closed path.
The fail-closed response is a fixed 500 under Code::Internal with the reason ext_authz request denied fail closed and the guard fail_closed, for a malformed CheckRequest and a kernel error alike; the fault reaches only the tracing::warn. kernel_error_redacts_internal_message_from_denied_response uses the error text db password=secret-token unavailable and asserts it appears in no body, header, or metadata field. missing_attributes_fails_closed_without_kernel_call asserts a request with no attributes returns with the kernel call count at zero.
Envoy body coverage
The adapter hashes a nonempty raw_body, or else a nonempty body. Declared length comes from Envoy’s size field. No supplied bytes means no hash, and a partially supplied body hashes the partial bytes, so a present hash shows what Envoy sent. Configure Envoy’s request-body handling to match the evidence your policy requires.
Tower layer tests
service_413_response_has_signed_receipt_body: header, JSON body, and extension agree, and both receipts verify under the embedded kernel key.service_413_fails_closed_without_durable_receipts: 502, with no extension, nox-chio-receipt-id, and an empty body.service_413_unsupported_method_rejects_before_route_resolver: the body guard runs first, with a resolver that panics if reached.service_413_without_receipt_body_omits_json_content_type: a bare 413 carries nocontent-type.service_fails_closed_when_durable_http_receipt_append_fails: a store that accepts the kernel’s first append and rejects the rest, with an inner service that panics if called; any response is a server error.skips_an_http_receipt_that_cannot_convert_to_a_verifiable_record: the append count stays at zero and no error propagates.service_denies_duplicate_query_capability_even_when_one_value_is_valid: a GET with one garbage value and one signed token gets 403, a receipt id header, andreceipt.is_denied().parse_duplicate_non_capability_query_does_not_set_override:tag=a&tag=byieldsband no override.
The duplicate sentinel and its override live in request_metadata.rs:
/// Invalid capability marker used to force fail-closed authority evaluation
/// when the transport presented an ambiguous query capability.
pub(crate) const DUPLICATE_QUERY_CAPABILITY_PRESENTATION: &str =
"duplicate chio_capability query parameters";
pub(crate) fn presented_capability_override(&self) -> Option<&'static str> {
self.has_duplicate_query_capability
.then_some(DUPLICATE_QUERY_CAPABILITY_PRESENTATION)
}RequestMetadata collapses the query into a HashMap where a repeated key keeps its last value; the count runs inside the url::form_urlencoded::parse loop, after percent-decoding. The sentinel makes validate_presented_capability record an invalid_reason, which projected_verdict returns as a deny.