Node Observability

What /metrics carries

The sidecar composes the body from three renderers. Knowing which one owns a family tells you where to look when a series goes missing.

/// Compose the Prometheus scrape body from the kernel guard families, the
/// http-core mediation-edge families, and the alert-pack families.
async fn handle_metrics() -> impl axum::response::IntoResponse {
    let alert_pack = || {
        let mut out = String::new();
        chio_metrics_spec::runtime::render_alert_pack_families(&mut out);
        out
    };
    let body = chio_metrics_spec::runtime::compose_metrics_body(&[
        &chio_kernel::render_guard_metrics_prometheus,
        &chio_http_core::metrics::render_http_core_metrics_prometheus,
        &alert_pack,
    ]);
    (
        [(
            axum::http::header::CONTENT_TYPE,
            "text/plain; version=0.0.4",
        )],
        body,
    )
}

The receipt watchdog gauges are chio_receipt_uncheckpointed_seq_range and chio_receipt_seconds_since_last_checkpoint. The second measures checkpoint progress, deliberately: in a store where writes keep committing while checkpointing stalls, a commit-based gauge would read near zero and the staleness alert would never fire.

The three guard descriptor tables

chio_wasm_guards::metrics declares three static descriptor tables, and the table a registry serves is a constructor argument. GuardMetricRegistry::with_families(families, max_guards) is the one constructor; new() and with_max_guards(n) wrap it with GUARD_METRIC_FAMILIES. That first table holds seven of the thirteen declared families and none of the tenant-labelled ones.

The seven guard families

GUARD_METRIC_FAMILIES, each labelled with guard_id. A guard can register metrics of its own; see Custom Guards.

The three pool families

GUARD_POOL_METRIC_FAMILIES, each carrying guard_id and tenant_id. A blank tenant folds to unknown and an over-cap tenant to __overflow__, so the label domain stays finite whatever a caller passes.

The three runtime families

RUNTIME_METRIC_FAMILIES: the signing-queue counter and the two OTel drop counters. The grafana/chio-perf.json dashboard plots all three.

Histogram buckets

Both histogram families use one fixed bucket set for every guard, so cross-guard quantiles compare without per-guard configuration.

pub const EVAL_DURATION_BUCKETS_SECONDS: &[f64] = &[
    0.0001, 0.0005, 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0,
];

pub const HOST_CALL_DURATION_BUCKETS_SECONDS: &[f64] = &[
    0.00001, 0.00005, 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1,
];

Label vocabulary

classify_deny_reason_class keeps chio_guard_deny_total{reason_class} finite. It substring-matches a guard's free-form deny reason and returns only a listed value, folding anything unrecognized or absent into other. Which denials carry the label at all is set out in Fail-Closed Semantics.

Cardinality limits

The registry caps unique guard ids at MAX_GUARD_METRIC_CARDINALITY = 1024. Beyond that, registration fails closed with GuardMetricRegistrationError code E_GUARD_METRIC_CARDINALITY_EXCEEDED, carrying the guard id, the attempted count, and the limit, and the guard is left out of the metric family. The stable 12-character guard id derives from the guard digest.

pub fn guard_id_label_from_digest(digest: &str) -> String {
    digest
        .strip_prefix("sha256:")
        .unwrap_or(digest)
        .chars()
        .take(12)
        .collect()
}

Two guards sharing their first 12 hex digits would collide, which stays negligible at 1,024 active guards. A fleet near the cap raises the limit with GuardMetricRegistry::with_max_guards(n) and keeps the full id length. The pool registry takes the same limit as a tenant ceiling through GuardPoolMetrics::with_max_tenants(n).

Stripping helpers

Two entries of the denylist are re-exported from chio_kernel::otel; ATTR_CHIO_REPLAY_RUN_ID and ATTR_CHIO_TENANT_ID are declared beside the array. A caller that pushes spans to a Prometheus-shaped sink of its own calls strip_denied_span_attributes before the push.

pub use denylist::{
    denied_attribute_keys, is_denied_attribute, strip_denied_attributes,
    strip_denied_batch_attributes, strip_denied_span_attributes, PROMETHEUS_DENIED_ATTRIBUTES,
};

The receipt exporter crate

chio-otel-receipt-exporter turns OTLP trace ingress into signed receipts. The kernel host's serve mode runs its OTLP gRPC ingress listener when one is configured, and a Rust caller can embed the same two pieces:

• OtlpGrpcIngress. Accepts OTLP/gRPC trace exports in a narrow Rust representation. BoundedOtlpGrpcIngress adds a queue with the drop counters the scrape reports.
• ReceiptStoreSink. Builds one ChioReceipt per OTLP span, signs it with ReceiptStoreSinkConfig.signing_keypair, and appends it to the configured Arc<dyn ReceiptStore>.

The sink's public API in crates/observability/chio-otel-receipt-exporter/src/sink.rs is three methods: new(store, config), export_traces(&OtlpGrpcTraceExport) returning a ReceiptStoreSinkSummary, and receipt_for_span(&OtlpSpan) returning one ChioReceipt. Both fallible methods return OTelReceiptExportError.

use chio_otel_receipt_exporter::{ReceiptStoreSink, ReceiptStoreSinkConfig};

let config = ReceiptStoreSinkConfig {
    signing_keypair,                       // Ed25519 keypair
    policy_hash: "otel-receipt-exporter".into(),
    default_capability_id: "otel-ingress".into(),
    default_tool_server: "otel-collector".into(),
    default_tool_name: "gen_ai.tool.call".into(),
    tenant_id: Some("prod".into()),
};

let sink = ReceiptStoreSink::new(receipt_store, config);
let summary = sink.export_traces(&otlp_export)?;
println!("appended {} receipts", summary.appended_receipts);

The five string defaults are the ones ReceiptStoreSinkConfig::new fills in, with tenant_id defaulting to None. Each receipt carries the sanitized attributes in a ToolCallAction and goes through the same ReceiptStore trait the kernel uses. The sink validates trace_id and span_id and calls strip_denied_attributes on every span before signing, so the canonicalized form is stable across re-export. A retried partial batch re-appends only its unappended receipts, with the same ids.

Span and log fields the dashboards expect

deploy/dashboards/README.md lists the fields every shipped dashboard expects on spans and log lines: chio.receipt.id, chio.tenant.id, chio.policy.ref, chio.verdict, chio.tee.mode, chio.deny.reason, chio.guard.outcome, provenance.otel.trace_id, provenance.otel.span_id, redaction_pass_id, and redaction_elapsed_micros.

Log levels and fields

• tracing::trace!. Per-step evaluation state. Off in production.
• tracing::debug!. Per-call detail for local development. Off in production except while investigating.
• tracing::info!. Lifecycle: kernel started, policy reloaded, checkpoint written, revocation propagated.
• tracing::warn!. Recoverable degradation and fail-closed fallbacks: cache-miss escalation, a guard rejecting invalid config and constructing a deny-all fallback, external-guard call failures, advisory verdicts.
• tracing::error!. Fail-closed events that produce Verdict::Deny through the fallback path: caught panics, poisoned mutexes, signing failures, an unreachable store.

Guards share no fixed field set beyond guard. Each attaches the fields that describe its decision, and many deterministic guards, forbidden-path among them, emit no tracing at all; their evidence lives in the receipt. Two representative call sites:

tracing::warn!(
    guard = "code-execution",
    module = %name,
    "denying code execution: dangerous module detected"
);

tracing::warn!(
    guard = "content-review",
    service = %service,
    endpoint = %endpoint,
    detected_categories = ?categories,
    "content-review denied outbound message"
);

The guard value identifies the component (code-execution, content-review, forbidden-path), never an individual call. Every other field (module, service, endpoint, code_len) is guard-specific, so expect no decision, path, or reason field on an arbitrary guard's events.

The shipped Grafana dashboards

deploy/dashboards/README.md documents five dashboards. All five expect the fields above and use the datasource variables DS_LOKI, DS_TEMPO, and DS_JAEGER, so import works across stacks. The import loop globs every *.json under deploy/dashboards/, so it also installs grafana/chio-perf.json (signing-queue blocking and OTel drop counters) and chio-pheromone-relay-observability.json (relay retry queue, dead letters, stale leases, bounded rejections).

loki/chio-tee.json

"Chio TEE Loki receipt view", on Loki. Four panels: TEE records by verdict, a TEE mode and verdict density heatmap, the raw log stream, and a receipt-to-trace lookup table (top 50 by receipt id, with policy ref, verdict, and deny reason). Filters by receipt id, tenant id, trace id, verdict, and TEE mode. See Confidential Node for the TEE signals.

loki/verdict-drift.json

"Chio replay verdict drift", on Loki. Filters chio-replay logs to event="verdict_drift" and renders a heatmap of from_verdict × to_verdict, a stacked time series of reason_delta, a top-100 drift records table, and a raw drift log stream.

jaeger/receipt-span-lookup.json

"Chio Jaeger receipt span lookup", on Jaeger. Pivots from receipt id, trace id, span id, tenant id, and tool name to traces. Four panels: trace search by tags, a table of receipt and span lookups, traces by verdict over time, and trace duration over time.

tempo/span-timeline.json

"Chio Tempo span timeline", on Tempo. The primary panel runs the TraceQL query from step 3 to fetch every trace that touched a receipt. Further panels: a span rows table, a GenAI tool-call rate time series, and span duration by verdict.

tempo/redaction-latency.json

"Chio Tempo redaction latency", on Tempo. Tracks redaction-pass latency by redaction_pass_id and chio.guard.outcome. Four panels: p95 latency, a latency distribution histogram, a redaction pass spans table, and span rate.

Recording rules

Burn-rate alerts fire on ratios, so the ratios are recorded first, in one group at a 30-second interval. clamp_min(..., 1) in the denominator keeps a quiet window from dividing by zero and reporting a false 100% error ratio.

groups:
  - name: chio-sre-recording
    interval: 30s
    rules:
      - record: chio:decision_latency:histogram_quantile_p95_5m
        expr: |
          histogram_quantile(
            0.95,
            sum by (surface, outcome, le) (rate(chio_kernel_decision_latency_seconds_bucket[5m]))
          )

      - record: chio:receipt_write_error_ratio_5m
        expr: |
          sum(rate(chio_receipt_write_total{outcome="error"}[5m]))
          /
          clamp_min(sum(rate(chio_receipt_write_total[5m])), 1)

The one-hour and six-hour pairs exist because each burn-rate alert requires both windows over budget at once. The five-minute receipt-write ratio and the two _5m gauge and latency rollups feed dashboards and SOC export rather than a paging rule.

Burn-rate SLO alerts

Each SLO uses two windows: the short one catches a fast burn, the long one suppresses flapping. The receipt-write rule in step 5 sets the shape. ChioSidecarErrorBudgetBurn1h uses it with a 0.5% budget at p2, and ChioAlertDispatchErrorBudgetBurn1h and ChioSocExportErrorBudgetBurn1h with 1% each. The 14.4 and 6 multipliers are the standard fast and slow burn factors, applied to whichever budget the SLO holds.

Missing-data and zero-tolerance alerts

A silent exporter looks identical to a healthy one on a ratio, so each burn-rate alert has an absent_over_time watchdog. Some conditions carry no budget at all and page on the first occurrence.

- alert: ChioFailOpenSuspected
  expr: increase(chio_fail_open_suspected_total[5m]) > 0
  for: 0m
  labels:
    severity: p0
    notification_route: pagerduty
    opsgenie: "true"
    slo: fail-closed
  annotations:
    summary: Chio observed a suspected fail-open path
    runbook: docs/operator-runbook/slo.md

The whole pack, with its labels

Fourteen rules in one group. The labels are the handoff to the chio-siem and chio-wall PagerDuty and OpsGenie dispatch path: severity, notification_route, and slo are on every rule, and three rules carry no opsgenie label, so a dispatcher that keys on opsgenie alone would drop those three.

Seven *MetricsMissing rules cover the four burn-rate SLOs and three of the four zero-tolerance rules; ChioReceiptCheckpointStale pages on its own threshold. Every rule carries a summary and a runbook annotation pointing at docs/operator-runbook/slo.md.

A third file, deploy/prometheus/chio-pheromone-relay-observability-rules.yml, defines the signal relay's operator alerts with the same severity, notification_route, and opsgenie labels: ChioPheromoneRelayDeadLetters, ChioPheromoneRelayRetryPressure, ChioPheromoneRelayStaleLeases, and ChioPheromoneRelayReplayStorm.

Source files

The facts above come from crates/products/chio-api-protect/src/proxy/router.rs, crates/guards/chio-wasm-guards/src/metrics.rs, crates/observability/chio-metrics-spec/src/runtime.rs, crates/observability/chio-otel-receipt-exporter/src/, deploy/dashboards/README.md, and deploy/prometheus/.