Performance & Tuning

Guard evaluation is CPU work in chio-guards and chio-wasm-guards. Persisting the decision is disk work in chio-store-sqlite. Above a few thousand requests a second the second one dominates, so sizing a deployment is a storage question before it is a throughput one.

Latency by guard

The external-guard row spans three orders of magnitude between a hit and a miss, so the miss rate, not the hit cost, sets the tail. A longer TTL lowers the miss rate and a larger capacity lowers LRU eviction, and both trade against how stale a verdict may be when it admits a call. TtlCache takes a per-entry TTL on insert and evicts on expiry or at capacity, so the two knobs are set per deployment against that freshness budget rather than to a recommended number.

A WASM guard's number is a policy choice rather than a property of the module. Custom WASM guards shows chio guard bench reading the fuel a module actually burns on a fixed request, which is the floor the per-module ceiling has to clear.

Throughput by pipeline shape

These are per-process numbers. Horizontal scaling is the answer for traffic above 5K req/s, but it shifts the bottleneck from CPU to the receipt store. See Bottlenecks below.

What a receipt costs to keep

A row in chio_tool_receipts stores the canonical receipt JSON verbatim in raw_json alongside the columns the store projects out of it for querying, and every one of those projections that carries an index pays for it again in a b-tree. Receipt volume, not request volume, is what sizes the disk.

How this was measured

Two receipt stores, each written by 200 preflight chio check invocations against the starter policy chio init writes, every one of them admitted. The first sends a small parameter object; the second sends the same call plus one 2 KiB string, so the difference between the two rows is payload and nothing else. The script below reads the schema and the b-tree page totals out of the finished databases through SQLite's dbstat virtual table, so the bytes are the bytes on disk rather than an estimate of them.

import sqlite3

c = sqlite3.connect("file:small.db?mode=ro", uri=True)
names = [r[0] for r in c.execute(
    "SELECT name FROM sqlite_master WHERE type='index' "
    "AND tbl_name='chio_tool_receipts' AND name NOT LIKE 'sqlite_%' ORDER BY name")]
auto = [r[0] for r in c.execute(
    "SELECT name FROM sqlite_master WHERE type='index' "
    "AND tbl_name='chio_tool_receipts' AND name LIKE 'sqlite_%' ORDER BY name")]
cols = [r[1] for r in c.execute("PRAGMA table_info(chio_tool_receipts)")]
c.close()
print("chio_tool_receipts: %d columns, %d named indexes, %d implicit"
      % (len(cols), len(names), len(auto)))
for n in names:
    print("  " + n)
print()
print("per receipt, averaged over each store:")
print("%-22s %8s %9s %8s %8s %7s" % ("parameter object", "receipts", "raw_json", "table", "indexes", "total"))
for label, path in (('{"name": "..."}', "small.db"), ("plus a 2 KiB string", "large.db")):
    c = sqlite3.connect("file:%s?mode=ro" % path, uri=True)
    n = c.execute("SELECT COUNT(*) FROM chio_tool_receipts").fetchone()[0]
    avg = c.execute("SELECT AVG(LENGTH(raw_json)) FROM chio_tool_receipts").fetchone()[0]
    tbl = c.execute("SELECT SUM(pgsize) FROM dbstat WHERE name = 'chio_tool_receipts'").fetchone()[0]
    idx = c.execute("SELECT SUM(pgsize) FROM dbstat WHERE name LIKE '%chio_tool_receipts%'"
                    " AND name <> 'chio_tool_receipts'").fetchone()[0]
    print("%-22s %8d %7.0f B %6.0f B %6.0f B %5.0f B"
          % (label, n, avg, tbl / n, idx / n, (tbl + idx) / n))
    c.close()

The result

$ python3 receipt-footprint.py

chio_tool_receipts: 16 columns, 9 named indexes, 1 implicit
  idx_chio_tool_receipts_capability
  idx_chio_tool_receipts_cost
  idx_chio_tool_receipts_cost_global
  idx_chio_tool_receipts_decision
  idx_chio_tool_receipts_grant
  idx_chio_tool_receipts_subject
  idx_chio_tool_receipts_tenant
  idx_chio_tool_receipts_timestamp
  idx_chio_tool_receipts_tool

per receipt, averaged over each store:
parameter object       receipts  raw_json    table  indexes   total
{"name": "..."}             200    2630 B   4116 B    532 B  4649 B
plus a 2 KiB string         200    4640 B   5140 B    532 B  5673 B

Every one of those 400 receipts is an allow. A canonical receipt for a two-field tool call runs about 2.6 KB and costs about 4.6 KB of store, roughly 1.8 times its own JSON. Two thousand bytes of extra parameters add about 2 KB of JSON and about 1 KB of store, which pulls the multiplier down to about 1.2. The overhead is not proportional, so the ratio improves as receipts grow.

The split explains it. The index column does not move between the two stores, because the nine named indexes and the implicit b-tree behind the UNIQUE on receipt_id all key on projected columns and none of them touches raw_json. What moves is the table, and it moves in whole SQLite pages, so a receipt smaller than a page still pays for the page. Six of those nine indexes are created at bootstrap and three more arrive through schema migration, which is why counting them from a live store beats counting them from any one source file.

For sizing, take the small-payload figure as the floor and scale by the parameter objects the deployment actually sends. A kernel sustaining 5K req/s writes 432 million receipts a day, which at that floor is about two terabytes a day before payload. RetentionConfig in chio-kernel is the control that bounds it, and receipt volume rather than request volume is what the disk is sized against.

The rest of the resident set

Bottlenecks

Four bottlenecks dominate, in this order:

1. Receipt store I/O. Every allowed or denied call writes one receipt. SQLite INSERT latency is the floor for kernel evaluation throughput on a single node. The store opens in WAL mode and refuses to run in any other journal mode, so that part is not a tuning decision; what is left is checkpoint batch size and per-tenant store sharding.
2. Session journal locks. The session-aware guards, DataFlowGuard and BehavioralSequenceGuard, each hold an Arc<SessionJournal> and take that journal's Mutex per evaluation. High concurrency on the same session serializes. Mitigate by sharding sessions across journals or by batching low-stakes calls outside the session. The velocity guards are a separate case: VelocityGuard and AgentVelocityGuard hold no journal, only a guard-wide Mutex over their own token-bucket map (VelocityGuard keys buckets by (capability_id, grant_index)), so journal sharding does nothing for them. Relieving that contention means sharding the bucket map itself.
3. WASM fuel. A WASM guard that exhausts its fuel returns Verdict::Deny, and the reason it carries classifies as reason_class = "fuel" on chio_guard_deny_total. Tail latency before that deny is the full fuel ceiling, so lowering the per-module limit lowers the tail as well as the blast radius.
4. External guard circuit breaker. A degraded provider can stall a synchronous pipeline; the breaker prevents that but at the cost of dropping calls during the open window. Mitigate by tuning RetryConfig::max_retries and CircuitBreakerConfig::reset_timeout for your provider's actual SLA.

Tuning knobs

These knobs size a deployment so it stays inside its limits. The limits themselves, and what each one returns when a request crosses it, are Backpressure and limits.

• Checkpoint cadence. checkpoint_interval in the receipts section (default 100 receipts per Merkle batch; it feeds the kernel's checkpoint_batch_size). Raise this to amortize signing cost; lower it for shorter recovery windows. 0 disables checkpointing.
• Receipt retention. RetentionConfig::retention_days (default 90) and max_size_bytes (default 10 GB), with check_interval_secs (default one hour) setting how often rotation is evaluated. Rotation moves aged rows into the archive database named by archive_path, and checkpoint validation opens that archive on its own read-only connection, so a receipt stays verifiable against its Merkle root after it leaves the live store.
• Session journal sharding. Shard by agent ID or session ID. Sharding by agent splits hot sessions across journals; pick the dimension that matches your contention pattern.
• WASM fuel limits. Per-module ceiling, expressed in Wasmtime fuel units. Lower ceilings cut tail latency; raise them only when a module hits the ceiling on legitimate input.
• AsyncGuardAdapter cache TTL. cache_ttl on AsyncGuardAdapterConfig, a Duration (default Duration::from_secs(60)). Bigger TTLs raise hit rate at the cost of evidence freshness.
• AsyncGuardAdapter rate limit. rate_per_second and rate_burst (defaults 20 / 20). Sized to typical provider QPS budgets; raise after confirming your contract.

A worked example at 5K req/s

A six-replica horizontally-scaled fleet running the default pipeline, a policy-wired content-safety provider, and a custom WASM classifier. ~833 req/s per replica. Each key below is a supported ChioConfig field in chio-config. Every section is deny_unknown_fields, so a typo or an invented section fails to parse.

kernel:
  signing_key: "${CHIO_SIGNING_KEY}"

adapters:
  # At least one adapter is mandatory. An absent or empty adapters block
  # parses but fails validation with "at least one adapter is required".
  - id: petstore
    protocol: openapi
    upstream: "http://petstore.example/api"

receipts:
  store: "sqlite:///var/lib/chio/receipts.db"
  # Receipts between Merkle checkpoints. Larger batches amortize signing
  # cost; at 833 req/s/replica, 500 checkpoints about every 0.6s.
  checkpoint_interval: 500
  # Live retention window. Aged receipts move to a read-only archive on
  # rotation and stay verifiable against their checkpoint roots.
  retention_days: 30

logging:
  level: info
  format: json

telemetry:
  enabled: true
  endpoint: "http://otel-collector:4317"
  service_name: chio-edge

guards:
  # Force these guards to run on every request, regardless of route.
  required:
    - internal-network
    - agent-velocity

wasm_guards:
  # wasm_guards is a LIST of entries, each with its own fuel_limit.
  - name: content-classifier
    path: /etc/chio/guards/content-classifier/content_classifier.wasm
    fuel_limit: 5000000
    priority: 100

Two things this configuration does not do:

• It does not enable any CircuitOpenVerdict::Allow or RateLimitedVerdict::Allow fail-open paths. Those are reserved for advisory guards; the deny defaults remain in place.
• It does not co-locate the receipt store with the agent. At 5K req/s, the SQLite receipt file is on the kernel's local disk; cross-replica receipt aggregation happens out-of-band via archive rotation or a streaming receipt sink.

See also

• Failure and recovery · what each fail mode costs in latency and verdict shape
• Node observability · the histograms and counters you build dashboards from
• The sidecar HTTP service · the localhost hop the sidecar shape adds to every call
• External guards · adapter knobs that drive the network-bound tail
• Custom WASM guards · reading the fuel a module burns before choosing its ceiling