Backpressure & Limits

Deny, never queue

A node is one OS process hosting one kernel, and every bound on this page is something that process can decide by reading its own state: a semaphore it holds, a counter it keeps, its own resident set size. No peer is consulted and no shared budget is drawn down, which is what keeps this at the Node rung rather than Budgets Across Nodes.

The bounds live in three places that never call each other. crates/protocol/chio-http-serve is a leaf crate: it depends on no other chio-* crate, carries no protocol, capability, or authorization logic, and its denials are bare status codes. Two product crates add their own limiters on top, keyed by something the hygiene crate cannot see (a peer IP, a federation subject). The kernel’s own ceiling is different in kind again: it produces a typed KernelError::Overloaded and a signed deny receipt. Knowing which layer refused a request is the difference between an unreceipted 503 and an audit record, so the three are worth keeping apart in your head.

All three are shipped, on different terms. The HTTP bounds are on by default at six serve sites. The two site-local limiters are on at their sites, one of them only for a federation policy that asks for it. The memory ceiling is implemented, tested at the shed, and off in every binary in the repo, for reasons the last two sections state plainly.

One config, five knobs

ServeHygieneConfig is a plain struct read by three call sites, not a framework. apply_server_hygiene reads three of its fields; max_connections goes to MaxConnListener::new and drain_timeout goes to run_until_drained, both by hand at the site.

The body cap defaults to None on purpose, and hygiene_defaults_are_conservative asserts it: a site with a large upload limit on one route wants that limit preserved rather than replaced by a service-wide number. Everything else defaults on. None does not mean unbounded: axum applies its own 2 MB default to Bytes-based extractors when no DefaultBodyLimit layer is in scope, so a site that configures nothing still gets that, and a handler that reads the raw body itself gets neither.

Layer order is the whole design of the concurrency limit. In a ServiceBuilder the first layer listed is outermost, giving HandleError(LoadShed(ConcurrencyLimit(router))). Load shedding sits outside the limit, so a request arriving with no permit free fails fast instead of parking on the semaphore. Both inner services are fallible, so the pair is wrapped in a HandleErrorLayer that turns the shed error into a response and keeps the result an infallible Router:

async fn shed_to_status(error: BoxError) -> StatusCode {
    if error.is::<tower::load_shed::error::Overloaded>() {
        StatusCode::SERVICE_UNAVAILABLE
    } else {
        StatusCode::INTERNAL_SERVER_ERROR
    }
}

The 20s request ceiling is not a round number chosen for taste. It stays strictly below the 25s drain window so a request admitted just before a stop signal reaches its own 408 and completes cleanly, rather than being severed when the forced-drain timer force-closes the connection. Two tests pin it: one asserts the constants stay ordered, the other drives a 30-second handler through a 100ms timeout inside a 600ms drain and asserts both the 408 and a clean drain outcome. A site that lengthens the request timeout must lengthen the drain to match.

The first bound is before the socket

axum::serve exposes no connection cap, so the cap lives as a listener adapter. The ordering inside accept is the point:

async fn accept(&mut self) -> (Self::Io, Self::Addr) {
    // Acquire a permit BEFORE pulling the next connection off the OS queue,
    // so a saturated server stops draining the accept backlog rather than
    // accepting sockets it cannot serve. `acquire_owned` only errors when
    // the semaphore is closed, which this adapter never does; treat that
    // impossible case as "serve without a permit" so it can never wedge the
    // accept loop.
    let permit = Arc::clone(&self.permits).acquire_owned().await.ok();
    let (io, addr) = self.inner.accept().await;
    (
        PermittedIo {
            io,
            _permit: permit,
        },
        addr,
    )
}

A permit is held for the connection’s whole lifetime and released when the connection IO drops, not when a request finishes. The semaphore is never closed, so a drain cannot revoke the permit of a connection still finishing. acquire_owned only errors on a closed semaphore, which this adapter never does, and the impossible case is treated as "serve without a permit" so it can never wedge the accept loop. usize::MAX is a valid uncapped sentinel: the count is clamped to Semaphore::MAX_PERMITS rather than panicking at startup.

Wrapping the listener changes its accepted IO type, which moves the site out of axum’s built-in Connected<_> for SocketAddr impl. A site that still needs the peer address serves into_make_service_with_connect_info::<CappedPeerAddr>(). That detail is load-bearing twice over: it is what lets the MCP edge key a rate limiter by IP and the proxy run its loopback checks while still holding the connection cap.

Six serve sites, three that turn the timeout off

Every site wires the connection cap and the drain. They diverge on the blanket request timeout and the body cap, and the divergences are documented at each site rather than inherited.

The three disables are not laziness, and each names a different hazard. The proxy writes its receipt synchronously in the handler after the upstream hop returns; an outer timeout layer would drop the handler mid-hop and skip receipt finalization for a call that may already have reached the upstream, so the hop is bounded by the configured upstream timeout instead and the drain is held a margin above it. Trust-control’s budget authorize parks in a rollback-aware quorum wait that is bounded on its own; a blanket timeout firing after the local exposure write but before that wait returns would drop the handler before its rollback branch and leave a charged, leader-visible write the client saw fail. The MCP edge answers GET and POST with Server-Sent Event streams that stay open while a session waits for notifications, so a blanket timeout would close a healthy idle stream with 408 and push resumable clients into reconnect churn; that page quotes the source comment saying so.

Two site-local limiters, and what a body cap actually caps

One site runs a per-IP limiter. The MCP edge keys McpRateLimiter by ip:<peer ip>, taken from CappedPeerAddr, and allows 600 requests per fixed 60-second window. The port is dropped, so every connection from one host shares one bucket, and a NAT egress shares one bucket for everything behind it. The map itself is bounded, which matters more than the rate: at MCP_RATE_LIMIT_MAX_KEYS = 4_096 a request from an unseen key first triggers a retain that drops every window that is not the current one, and only if the table is still full does the request fail. Denials return a plain-text 429 with a retry-after header computed from the window edge. A poisoned mutex is treated as a denial with the same retry hint, so the limiter fails closed.

Trust-control runs the second one, and it is keyed by identity rather than address. FederationAdmissionRateLimiter buckets federation admission attempts by policy_id:subject_key and only engages for a federation policy whose record declares an anti_sybil.rate_limit; the window and request count come from that record, not from a constant. Its table is capped at MemoryBudgetConfig::admission_key_cap (4096), threaded from the operator-configured budget rather than read fresh, and at cap it denies an unseen subject rather than evicting an active bucket. That ordering is the point: evicting an active bucket would reset its window, so a flood of throwaway subjects could clear an attacker’s own limit. It prunes only expired buckets and then saturates.

Body caps need one clarification, because DefaultBodyLimit is not a byte-counting middleware. It inserts a limit into the request extensions that FromRequest extractors built on Bytes consume. A handler that takes the raw body and drains it itself is not covered. That is exactly why the MCP edge does not rely on the layer at all: it rejects an oversized Content-Length up front and then reads with an explicit 8 MiB ceiling, returning a JSON-RPC-shaped 413 either way. The proxy also drains its own bodies, but not to the same standard: it has no Content-Length pre-check, and an over-limit read falls out of to_bytes as a plain 400 with failed to read request body, indistinguishable from a truncated upload.

Where the layer is used, a route can raise it. Trust-control sets 1 MiB service-wide and then overrides it on four routes carrying three ceilings: 384 MiB for internal admission authority, 128 MiB on each of the two receipt-append routes (a stream receipt carries one 64-hex-char chunk digest per retained chunk, and DEFAULT_MAX_STREAM_CHUNKS is 1,048,576, so a full receipt reaches 64 MiB of digest text alone and the cap is sized to leave envelope room), and 64 MiB for evidence import. Precedence runs the right way for that: DefaultBodyLimitService inserts its limit into the request extensions unconditionally, so the innermost route-local layer writes last and wins over the service-wide one. Each override is still a bound, and the buffered decode cannot grow without one.

The relay adds structural caps on top of its byte cap. A batch over max_batch_frames (128 in the production profile) is refused, a catch-up request over max_catchup_frames (256) is refused, and a catch-up response is bounded by max_catchup_bytes (1 MiB). Those are per-peer values from the signed peer directory, and directory validation refuses any peer entry that declares a value above the profile ceiling or a zero.

The RSS soft ceiling, and who signs the denial

The HTTP bounds cannot see memory. The kernel’s ceiling is declared in MemoryBudgetConfig::rss_soft_limit_bytes and enforced by a dedicated OS thread, not a tokio task. When a limit is configured, construction spawns RssSamplerHandle, which reads /proc/self/statm every rss_sample_interval_secs (default 30) and stores rss > soft_limit_bytes into a shared AtomicBool. It waits in 200ms slices so a stop is prompt, and joins on drop. Resident pages are multiplied by the real page size from sysconf(_SC_PAGESIZE), cached once: a host with 64 KiB pages would otherwise undercount RSS by the page-size ratio and shed far past the configured ceiling.

Three admission paths load that flag, each immediately after the emergency-stop check, so the hot path pays one relaxed atomic load. The flag both raises and clears, so a process that falls back under the ceiling resumes admitting without intervention. It is one flag for the whole process: there is no per-tenant or per-session fairness in the shed, so under pressure every admission is refused, not the expensive ones.

Reading the hygiene module and the kernel module together answers the question that matters for audit: nothing in chio-http-serve mints a receipt, because that crate holds no signing key and no capability logic. The kernel mints it. On the two tool-call paths, record_overload_shed_deny_receipt builds a local, non-federated v1 deny under guard name kernel.overload, with the reason kernel shed load to stay within its memory budget (resource: Allocation), and persists it through the same fail-closed path the emergency stop uses (durable store first, in-process mirror if the store is serving-closed). Only then does the shed return Overloaded { resource: Allocation } so a backpressure-aware caller still sees a retryable error. A failure to persist that receipt is logged and does not mask the shed. Why an overload denial carries a receipt at all is the fail-closed page’s claim, not this one’s: Fail-Closed Semantics.

cgroup, overcommit, and the OOM ordering

The soft ceiling is the in-process analog of a cgroup hard limit, and it only helps if the OS is configured to let allocations fail early and locally instead of succeeding and being reaped later. Four settings, in the order they take effect.

Under systemd these are MemoryHigh=, MemoryMax=, OOMScoreAdjust=, and LimitAS=, with the untrusted sidecar unit set to OOMScoreAdjust=500. The reference units shipped under docs/release/systemd/ do not set them: they set KillSignal=SIGTERM and TimeoutStopSec=35s for the drain, then carry a comment naming the directives the operator has to add (MemoryMax, MemoryHigh, OOMScoreAdjust, LimitNOFILE, LimitAS) so both units share one OOM posture. Treat docs/architecture/reliability/deployment-memory.md as guidance an operator applies, not as a default a deployment inherits.

The collections behind the ceiling

A soft ceiling that only sheds is a stopgap; the reason a node stays under it is that its long-lived collections have capacity policies. crates/core/chio-bounded exists for exactly one invariant: no long-lived collection in a serving process without a capacity policy and a live size metric. It exports three things and depends on no kernel or federation crate. Ring<T> is a fixed-capacity append-only buffer that evicts oldest-first and hands the evicted item back. BoundedMap<K, V> is a capacity-bounded cache with an optional idle sweep that runs inline, every 256 inserts, rather than as a background task. SizeGauge is a cloneable atomic handle readable by anything but writable only from inside the crate. On both collections capacity == 0 is an explicit disabled mode: nothing is stored, and the item is handed straight back.

Two BoundedMap properties matter before you rely on it. Eviction order is oldest-insert, not true LRU: a re-insert moves a key to newest, but get only refreshes the idle timestamp and does not reorder, so a key that is read constantly and never rewritten can still be evicted ahead of a colder one. And the idle sweep is driven by insert count alone, so a map that stops taking inserts stops sweeping and holds its entries past their TTL until the next write.

In the kernel that shows up as a receipt mirror capped at 4096, a child receipt mirror at the same cap, and two federation artifact caches at 8192 with a one-hour idle TTL. Eviction has a real consequence and the code says so: an evicted receipt is fine when the durable store can point-load it by id, and unresolvable when the store is append-only or remote without ReceiptStore::load_chio_receipt. In that second case a dependent call-chain claim fails closed, which is a deny and never a false allow. A test builds precisely that deployment and asserts the deny.

Those four are the labelled structures, not the whole budget. MemoryBudgetConfig::defaults() carries five more caps that no gauge reports, and two of them deny rather than evict.

Guarantees and limits

Next Steps

• Node Lifecycle · the other half of this crate: one stop signal, one bounded drain, and the flush hook contract
• Remote MCP Edge · the listener the per-IP limiter is attached to, and the gates that run after it
• Health & Readiness · what a saturated or shedding node reports to a probe, and the writer-liveness gate that denies before dispatch
• Node State on Disk · the receipt commit queue, a bounded channel that refuses rather than growing
• Fail-Closed Semantics · why a shed is a denial with a receipt rather than a dropped request
• Performance & Tuning · sizing the deployment so these bounds are never the thing you hit