Node Observability

Three signals, three roles

Operators tend to want all three, for very different reasons.

Signed receipts are the audit-of-record. Traces and metrics are operational telemetry. For dispute resolution and compliance evidence, query receipts. For "is the node healthy right now?", read metrics. For "why did request X fail at 03:14", read the trace and join it to the receipt for X.

What /metrics carries

The sidecar mounts /metrics on its own ingress port, the port --listen binds, ahead of the catch-all proxy route so axum prefers it. It sits behind the same sidecar-control middleware that gates the approval routes, so a scraper presents the control token rather than reaching an open endpoint. The response carries text/plain; version=0.0.4.

/// Compose the Prometheus scrape body from the kernel guard families, the
/// http-core mediation-edge families, and the alert-pack families.
async fn handle_metrics() -> impl axum::response::IntoResponse {
    let alert_pack = || {
        let mut out = String::new();
        chio_metrics_spec::runtime::render_alert_pack_families(&mut out);
        out
    };
    let body = chio_metrics_spec::runtime::compose_metrics_body(&[
        &chio_kernel::render_guard_metrics_prometheus,
        &chio_http_core::metrics::render_http_core_metrics_prometheus,
        &alert_pack,
    ]);
    (
        [(
            axum::http::header::CONTENT_TYPE,
            "text/plain; version=0.0.4",
        )],
        body,
    )
}

The body is three renderers concatenated, and knowing which one owns a family is what tells you where to look when a series goes missing.

The receipt watchdog gauges are chio_receipt_uncheckpointed_seq_range and chio_receipt_seconds_since_last_checkpoint. The second one measures checkpoint progress rather than write freshness on purpose: in a store where writes keep committing while checkpointing stalls, a commit-based gauge would read near zero and the staleness alert would never fire.

The three guard descriptor tables

chio_wasm_guards::metrics declares three static descriptor tables, and which one a registry serves is a constructor argument rather than a constant. GuardMetricRegistry::with_families(families, max_guards) is the only real constructor; new() and with_max_guards(n) are wrappers that pass GUARD_METRIC_FAMILIES. Reading only that first table gives you seven of the thirteen declared families and none of the tenant-labelled ones.

The seven guard families

GUARD_METRIC_FAMILIES, every one of them labelled with guard_id.

The three pool families

GUARD_POOL_METRIC_FAMILIES. All three carry guard_id and tenant_id. A blank tenant folds to unknown and an over-cap tenant folds to __overflow__, so the label domain stays finite whatever a caller passes.

The three runtime families

RUNTIME_METRIC_FAMILIES, the signing-queue and OTel drop counters.

The grafana/chio-perf.json dashboard is built on this third table: it plots the signing-queue blocking counter and the two OTel drop counters.

Histogram buckets

Both histogram families carry a fixed bucket set, shared by every guard, so cross-guard quantile comparisons are meaningful without per-guard configuration.

pub const EVAL_DURATION_BUCKETS_SECONDS: &[f64] = &[
    0.0001, 0.0005, 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0,
];

pub const HOST_CALL_DURATION_BUCKETS_SECONDS: &[f64] = &[
    0.00001, 0.00005, 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1,
];

Label vocabulary

Every label value on the guard families comes from a closed set, so each series has a finite label domain.

classify_deny_reason_class is what keeps chio_guard_deny_total{reason_class} finite: it substring-matches a guard's free-form deny reason and never returns a raw string, folding anything unrecognized or absent into other. Seeing reason_class="other" on a dashboard means the classifier assigned the fallback category, not that a guard emitted it. Which denials carry the label at all is narrower than the metric table suggests; Fail-Closed Semantics owns that matrix.

Cardinality limits

The registry caps unique guard IDs at MAX_GUARD_METRIC_CARDINALITY = 1024. Beyond that, registration fails closed with GuardMetricRegistrationError code E_GUARD_METRIC_CARDINALITY_EXCEEDED and the guard is dropped from the metric family. The error carries the guard ID, the attempted count, and the limit. The registry derives the stable 12-character guard ID from the guard digest:

pub fn guard_id_label_from_digest(digest: &str) -> String {
    digest
        .strip_prefix("sha256:")
        .unwrap_or(digest)
        .chars()
        .take(12)
        .collect()
}

Two guards sharing the same first 12 hex digits would collide. That is a non-issue at 1024 active guards; a fleet approaching the cap should raise the limit explicitly with GuardMetricRegistry::with_max_guards(n) rather than truncate the ID to fewer characters. The pool registry takes the same limit as a tenant ceiling through GuardPoolMetrics::with_max_tenants(n).

Prometheus scrape config

Scrape the ingress port. The reference ECS task definition advertises it with "prometheus.io/path": "/metrics" and "prometheus.io/port": "9090", and the Cloud Run and Azure manifests both say the same about the endpoint on their own container port. The bearer token is the sidecar-control token the deployment already provisions.

# prometheus.yml
scrape_configs:
  - job_name: 'chio-sidecar'
    scrape_interval: 15s
    metrics_path: /metrics
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/chio-sidecar-control-token
    static_configs:
      - targets:
          - chio-sidecar:9090

OTel collector wiring

A standard OpenTelemetry Collector is the right aggregation point for everything except receipts. Receipts go to the receipt store first; the SIEM exporter or the OTel receipt exporter reads from there. The collector endpoint is resolved from CHIO_OTEL_RECEIPT_EXPORTER_ENDPOINT first and OTEL_EXPORTER_OTLP_ENDPOINT second, which is the order chio doctor reports it in.

The collector config below receives OTLP, strips the high-cardinality attributes before anything Prometheus-shaped sees them, and fans out to Tempo and Loki.

# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

  # Strip high-cardinality attributes from anything bound for Prometheus.
  # These attributes are safe on spans and logs; they are not safe as
  # Prometheus label values.
  attributes/strip-cardinality:
    actions:
      - key: gen_ai.tool.call.id
        action: delete
      - key: chio.receipt.id
        action: delete
      - key: chio.replay.run_id
        action: delete
      - key: chio.tenant.id
        action: delete

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki]

High-cardinality attribute denylist

The denied set is a fixed four-element array. Two of its entries are re-exported from chio_kernel::otel and two are declared beside the array as ATTR_CHIO_REPLAY_RUN_ID and ATTR_CHIO_TENANT_ID.

pub const PROMETHEUS_DENIED_ATTRIBUTES: [&str; 4] = [
    ATTR_GEN_AI_TOOL_CALL_ID,
    ATTR_CHIO_RECEIPT_ID,
    ATTR_CHIO_REPLAY_RUN_ID,
    ATTR_CHIO_TENANT_ID,
];

The stripping helpers are exported for callers that push spans to a Prometheus-shaped sink of their own. Call strip_denied_span_attributes before the push.

pub use denylist::{
    denied_attribute_keys, is_denied_attribute, strip_denied_attributes,
    strip_denied_batch_attributes, strip_denied_span_attributes, PROMETHEUS_DENIED_ATTRIBUTES,
};

The receipt exporter crate

chio-otel-receipt-exporter is the bridge from OTLP trace ingress to the Chio receipt store. No workspace binary depends on it: a caller embeds it. Two pieces:

• OtlpGrpcIngress · accepts OTLP/gRPC trace exports in a narrow Rust representation. A bounded variant, BoundedOtlpGrpcIngress, adds a queue with the drop counters the scrape reports.
• ReceiptStoreSink · builds a ChioReceipt per OTLP span, signs it with ReceiptStoreSinkConfig.signing_keypair, and appends it to the configured Arc<dyn ReceiptStore>.

The sink's public surface, in crates/observability/chio-otel-receipt-exporter/src/sink.rs, is three methods: new(store, config), export_traces(&OtlpGrpcTraceExport) returning a ReceiptStoreSinkSummary, and receipt_for_span(&OtlpSpan) returning one ChioReceipt. Both fallible methods return OTelReceiptExportError.

use chio_otel_receipt_exporter::{ReceiptStoreSink, ReceiptStoreSinkConfig};

let config = ReceiptStoreSinkConfig {
    signing_keypair,                       // Ed25519 keypair
    policy_hash: "otel-receipt-exporter".into(),
    default_capability_id: "otel-ingress".into(),
    default_tool_server: "otel-collector".into(),
    default_tool_name: "gen_ai.tool.call".into(),
    tenant_id: Some("acme-prod".into()),
};

let sink = ReceiptStoreSink::new(receipt_store, config);
let summary = sink.export_traces(&otlp_export)?;
println!("appended {} receipts", summary.appended_receipts);

Those five string defaults are the ones ReceiptStoreSinkConfig::new fills in, with tenant_id defaulting to None.

The sink is single-purpose: one ChioReceipt per span, with ToolCallAction carrying the sanitized attributes, appended through the same ReceiptStore trait the kernel uses. Span identity carries through: trace_id and span_id are validated before signing. The sink calls strip_denied_attributes on every span before signing the receipt body, so the canonicalized form is stable across re-export: ingest the same span twice and you get the same receipt body bytes.

Trace propagation

Chio respects W3C Trace Context. When the upstream caller passes a traceparent header, the kernel joins that trace and emits its spans as children. The receipt then carries provenance.otel.trace_id and provenance.otel.span_id as fields, which is what makes the Tempo and Jaeger dashboards able to pivot from a receipt to a trace. The span itself is named gen_ai.tool.call with operation tool.call, both locked as constants in chio_kernel::otel.

deploy/dashboards/README.md lists the fields every shipped dashboard expects on spans and log lines. A panel that renders empty is usually a missing one of these:

• chio.receipt.id
• chio.tenant.id
• chio.policy.ref
• chio.verdict
• chio.tee.mode
• chio.deny.reason
• chio.guard.outcome
• provenance.otel.trace_id
• provenance.otel.span_id
• redaction_pass_id
• redaction_elapsed_micros

Tracing and structured logging

Every Chio crate uses the tracing crate. Guards, kernel evaluation, and adapters emit structured events through five standard macros. Guard spans are pinned to the chio.guard target, because EnvFilter directives match on the target rather than the span name.

• tracing::trace! · per-step state in evaluation. Off in production.
• tracing::debug! · per-call detail useful for local development. Off in production unless investigating.
• tracing::info! · lifecycle: kernel started, policy reloaded, checkpoint written, revocation propagated.
• tracing::warn! · recoverable degradation and fail-closed fallbacks: cache-miss escalation, a guard rejecting invalid config and constructing a deny-all fallback, external-guard call failures, advisory verdicts.
• tracing::error! · fail-closed events that produce Verdict::Deny via the fallback path: caught panics, poisoned mutexes, signing failures, store unreachable.

There is no single fixed field set across guards. The guards that emit tracing share one convention, a guard field carrying the guard's kebab-case Guard::name(), and otherwise attach whatever fields describe that guard's decision. Many deterministic guards, forbidden-path among them, emit no tracing at all; their evidence lives in the receipt, not the log. Two representative call sites:

tracing::warn!(
    guard = "code-execution",
    module = %name,
    "denying code execution: dangerous module detected"
);

tracing::warn!(
    guard = "content-review",
    service = %service,
    endpoint = %endpoint,
    detected_categories = ?categories,
    "content-review denied outbound message"
);

The guard field is the one stable join key between a log line and its receipt. Its value is the same kebab-case string the guard reports through Guard::name() (code-execution, content-review, forbidden-path) and the same string the pipeline records as GuardEvidence.guard_name on the receipt, so the two correlate on that value. Every other field (module, service, endpoint, code_len) is guard-specific. Do not assume a decision, path, or reason field on an arbitrary guard's events.

Four queries that carry the load

Before the shipped dashboards and before the rule pack, four ad-hoc queries answer most day-to-day questions. Paste them into Grafana Explore or the Prometheus console. The rule pack below is what turns the same signals into paging alerts.

• Deny rate per guard. rate(chio_guard_deny_total[5m]) grouped by guard_id, reason_class. A jump above a guard's baseline is the early-warning channel for both attack signals and policy regressions.
• Fuel exhaustion tail. histogram_quantile(0.99, ...) on chio_guard_eval_duration_seconds with verdict="deny", joined against chio_guard_deny_total{reason_class="fuel"}. Indicates WASM modules approaching their fuel ceiling.
• External-guard call failures. The breaker state machine does not log its own transitions, so the observable signal is the adapter's failure path: every failed provider call logs tracing::warn!(guard, error, "external guard failed") and pushes the circuit breaker toward open. Take the rate of those per guard; a sustained rate means a provider is degraded and the breaker is shedding load. See External Guard Adapters for the breaker contract behind that signal.
• Reload outcomes. chio_guard_reload_total grouped by outcome. A rising canary_failed or rolled_back count means hot-reload deployments are not landing cleanly.

The shipped Grafana dashboards

Five Grafana dashboard JSON files are documented in deploy/dashboards/README.md. All five expect the telemetry fields listed above and use Grafana datasource variables (DS_LOKI, DS_TEMPO, DS_JAEGER) so import works across stacks. The one-command import script lives at the top of that README. It globs every *.json under deploy/dashboards/, so it also installs two dashboards outside the documented five: grafana/chio-perf.json (signing queue blocking and OTel drop counters) and chio-pheromone-relay-observability.json (relay retry queue, dead letters, stale leases, bounded rejections).

loki/chio-tee.json

Title: Chio TEE Loki receipt view. Targets Loki. Four panels: TEE records by verdict, TEE mode and verdict density heatmap, raw log stream, and a receipt-to-trace lookup table (top 50 by receipt id, with policy ref, verdict, deny reason). Filterable by receipt id, tenant id, trace id, verdict, and TEE mode.

loki/verdict-drift.json

Title: Chio replay verdict drift. Targets Loki. Filters chio-replay logs to event="verdict_drift" and renders a heatmap of from_verdict × to_verdict, a stacked time series of reason_delta, a top-100 drift records table, and a raw drift log stream.

jaeger/receipt-span-lookup.json

Title: Chio Jaeger receipt span lookup. Targets Jaeger. Pivots from receipt id, trace id, span id, tenant id, and tool name to traces. Four panels: trace search by tags, table of receipt and span lookups, traces-by-verdict time series, and trace-duration time series.

tempo/span-timeline.json

Title: Chio Tempo span timeline. Targets Tempo. The primary panel runs the TraceQL query { span.chio.receipt.id = "$receipt_id" } to fetch every trace that touched a given receipt. Additional panels: a span rows table, a GenAI tool-call rate time series, and a span-duration-by-verdict line chart.

tempo/redaction-latency.json

Title: Chio Tempo redaction latency. Targets Tempo. Tracks redaction-pass latency by redaction_pass_id and chio.guard.outcome. Four panels: p95 latency, latency distribution histogram, redaction pass spans table, and span rate.

Alert rules

Chio ships a rule pack in deploy/prometheus/ (the “T1.5 SRE rule pack” per deploy/prometheus/README.md), not a collection of ad hoc alerts. Three .yml files sit in that directory. Two carry the core rules: chio-recording-rules.yml precomputes the error-ratio and p95 series the SLOs burn against, and chio-alert-rules.yml layers multi-window burn-rate and missing-data alerts on top.

Recording rules

Burn-rate alerts fire on ratios, so the ratios are recorded first, all in one group at a 30-second interval.

groups:
  - name: chio-sre-recording
    interval: 30s
    rules:
      - record: chio:decision_latency:histogram_quantile_p95_5m
        expr: |
          histogram_quantile(
            0.95,
            sum by (surface, outcome, le) (rate(chio_kernel_decision_latency_seconds_bucket[5m]))
          )

      - record: chio:receipt_write_error_ratio_5m
        expr: |
          sum(rate(chio_receipt_write_total{outcome="error"}[5m]))
          /
          clamp_min(sum(rate(chio_receipt_write_total[5m])), 1)

clamp_min(..., 1) in the denominator keeps a quiet window from dividing by zero and reporting a false 100% error ratio. Every recorded series in the group:

The one-hour and six-hour pairs exist because each burn-rate alert requires both windows to be over budget at once. The five-minute receipt-write ratio and the two _5m gauge and latency rollups feed dashboards and SOC export rather than a paging rule.

Burn-rate SLO alerts

Each SLO uses a two-window burn-rate: the short window catches a fast burn, the long window suppresses flapping. Receipt writes hold a 0.1% error budget.

- alert: ChioReceiptWriteErrorBudgetBurn1h
  expr: |
    chio:receipt_write_error_ratio_1h > (14.4 * 0.001)
    and
    chio:receipt_write_error_ratio_6h > (6 * 0.001)
  for: 2m
  labels:
    severity: p1
    notification_route: pagerduty
    opsgenie: "true"
    slo: receipt-write
  annotations:
    summary: Chio receipt writes are burning error budget
    runbook: docs/operator-runbook/slo.md

The same shape covers ChioSidecarErrorBudgetBurn1h (0.5% budget, p2) and ChioAlertDispatchErrorBudgetBurn1h / ChioSocExportErrorBudgetBurn1h (1% budget each). The 14.4 and 6 multipliers are the standard fast and slow burn factors applied to whichever budget the SLO holds.

Missing-data and zero-tolerance alerts

A silent exporter looks identical to a healthy one on a ratio, so a burn-rate alert is paired with an absent_over_time watchdog. Some conditions carry no budget at all and page on the first occurrence:

- alert: ChioFailOpenSuspected
  expr: increase(chio_fail_open_suspected_total[5m]) > 0
  for: 0m
  labels:
    severity: p0
    notification_route: pagerduty
    opsgenie: "true"
    slo: fail-closed
  annotations:
    summary: Chio observed a suspected fail-open path
    runbook: docs/operator-runbook/slo.md

The whole pack, with its labels

Fourteen rules, one group. The label contract is the handoff to the chio-siem and chio-wall PagerDuty and OpsGenie dispatch path, and it is not uniform: severity, notification_route, and slo are on every rule, and three rules carry no opsgenie label at all. A dispatcher keying on opsgenie alone would silently drop those three.

The watchdog pairing is close to one-for-one and not quite: seven *MetricsMissing rules cover the four burn-rate SLOs and three of the four zero-tolerance rules. ChioReceiptCheckpointStale has no counterpart, so a receipt-log watchdog gauge that stops being reported goes unnoticed by this pack. Every rule carries a summary and a runbook annotation pointing at docs/operator-runbook/slo.md.

SIEM integration

For long-term retention and security workflow, receipts go to a SIEM rather than a metrics or trace backend. The exporter is independent of the kernel and uses a sequence cursor so it never loses data on retry. Read SIEM Export for the design, configuration, and OCSF field mapping.

A worked compose stack

A minimal stack that runs an OTel collector, Prometheus, Loki, Tempo, and Grafana beside a sidecar. The sidecar service needs a real subcommand: the image CMD is ["--help"], so a service definition that passes no arguments prints usage and exits before anything can scrape it. The arguments below are the ones the reference ECS task definition uses.

# docker-compose.yaml
services:
  chio-sidecar:
    image: chio-sidecar:latest
    command:
      - api
      - protect
      - --upstream
      - http://app:8080
      - --spec
      - /etc/chio/spec/openapi.yaml
      - --listen
      - 0.0.0.0:9090
      - --receipt-store
      - /var/lib/chio/receipts.db
      - --authority-seed-file
      - /etc/chio/seed/authority.seed
    environment:
      CHIO_LOG_LEVEL: info
      CHIO_SIDECAR_CONTROL_TOKEN: ${CHIO_SIDECAR_CONTROL_TOKEN}
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317
    ports:
      - "9090:9090"      # proxy ingress, /chio/health, /chio/live, /metrics

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector.yaml"]
    volumes:
      - ./otel-collector.yaml:/etc/otel-collector.yaml:ro
    ports:
      - "4317:4317"      # OTLP gRPC
      - "4318:4318"      # OTLP HTTP

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro

  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"

  tempo:
    image: grafana/tempo:latest
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml:ro

  grafana:
    image: grafana/grafana:latest
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    ports:
      - "3000:3000"
    volumes:
      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/ds.yaml:ro

Once Grafana is up, the dashboard import is the one loop deploy/dashboards/README.md carries, run from the Chio repository root:

$ find deploy/dashboards -name '*.json' -print0 | \
    while IFS= read -r -d '' dashboard; do
      jq -n --argjson dashboard "$(cat "$dashboard")" \
        '{dashboard: $dashboard, overwrite: true}' \
      | curl -fsS -H 'Content-Type: application/json' \
          -X POST http://admin:admin@127.0.0.1:3000/api/dashboards/db -d @- \
          >/dev/null
    done

Next steps

• Sidecar HTTP Service for the flags and health endpoints the scrape target shares a port with.
• Receipts & Audit for the audit-of-record signal that metrics and traces complement.
• SIEM Export for receipt forwarding to Splunk HEC and Elasticsearch with dead-letter handling.
• Confidential Node for the chio-tee Loki dashboard and TEE-specific signals.
• Receipt Dashboard for the receipt-first operator UI that is separate from Grafana.
• Performance & Tuning for which signals point at which bottlenecks.
• Custom Guards for how a guard registers its own metrics.