OpenTelemetry

What It Shows

• OTel collector wired to Tempo (TraceQL) and Jaeger (lookup by trace ID and tag) with Grafana on top.
• The locked GenAI tool-call attribute set chio targets: gen_ai.system, gen_ai.operation.name, gen_ai.request.model, gen_ai.tool.call.id, gen_ai.tool.name, plus chio attributes (chio.receipt.id, chio.tenant.id, chio.policy.ref, chio.verdict, chio.tee.mode) and OTel provenance (provenance.otel.trace_id, provenance.otel.span_id).
• Bidirectional lookup between the receipt ID and span ID through the exporter.
• Metric-safety policy on the collector: high-cardinality attributes are stripped from metric pipelines while the receipt-id field on traces is preserved.

Prerequisites

• Docker with docker compose v2.
• Rust toolchain (only for the contract test).
• Optional: jq and curl for the dashboard import script.

Files

examples/otel-genai/
  ARCHITECTURE.md
  Cargo.toml
  README.md
  docker-compose.yml
  otel-collector-config.yaml
  tests/bidirectional_lookup.rs
  tests/support/mod.rs

Seven files, and the last one carries most of the example: tests/support/mod.rs holds the in-memory receipt store, the demo span, the sink config, and the lookup index builder, so bidirectional_lookup.rs is nothing but assertions.

Run the Collector Demo

cd examples/otel-genai
docker compose up

The compose file boots four services, all pinned:

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.115.1
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol/config.yaml:ro
    ports:
      - "4317:4317"
      - "4318:4318"
      - "8889:8889"
    depends_on:
      - tempo
      - jaeger
    restart: "no"

  tempo:
    image: grafana/tempo:2.6.1
    command:
      - "-target=all"
      - "-server.http-listen-port=3200"
      - "-distributor.receivers.otlp.protocols.grpc.endpoint=0.0.0.0:4317"
      - "-distributor.receivers.otlp.protocols.http.endpoint=0.0.0.0:4318"
      - "-storage.trace.backend=local"
      - "-storage.trace.local.path=/tmp/tempo/traces"
      - "-compactor.compaction.block-retention=1h"
    ports:
      - "3200:3200"
    restart: "no"

  jaeger:
    image: jaegertracing/all-in-one:1.62
    environment:
      COLLECTOR_OTLP_ENABLED: "true"
    ports:
      - "16686:16686"
    restart: "no"

  grafana:
    image: grafana/grafana:11.5.0
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
      GF_SECURITY_ADMIN_PASSWORD: admin
    ports:
      - "3000:3000"
    depends_on:
      - tempo
      - jaeger
    restart: "no"

The compose stack exposes:

Import the Chio dashboards from the repository root:

find deploy/dashboards -name '*.json' -print0 \
  | while IFS= read -r -d '' dashboard; do
      jq -n --argjson dashboard "$(cat "$dashboard")" \
        '{dashboard: $dashboard, overwrite: true}' \
      | curl -fsS -H 'Content-Type: application/json' \
          -X POST http://admin:admin@127.0.0.1:3000/api/dashboards/db -d @- >/dev/null
    done

Collector Pipeline

The collector receives OTLP on gRPC and HTTP, fans traces out to Tempo, Jaeger, and a debug exporter, and uses an attributes processor to strip high-cardinality keys from the metric pipeline. The receipt-id stays on the trace pipeline; the metric pipeline drops it so a Prometheus-shaped backend does not see it as a label:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}
  attributes/chio-metric-safety:
    actions:
      - key: gen_ai.tool.call.id
        action: delete
      - key: chio.receipt.id
        action: delete
      - key: chio.replay.run_id
        action: delete

exporters:
  debug:
    verbosity: basic
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo, otlp/jaeger, debug]
    metrics:
      receivers: [otlp]
      processors: [attributes/chio-metric-safety, batch]
      exporters: [debug]

Run the Contract Test

From the Chio workspace root:

$ cargo test -p otel-genai   # test-harness output only

running 1 test
test receipt_id_and_span_id_lookup_is_bidirectional ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

That run had no collector up: the test builds the span and drives the exporter in process, so the compose stack above is for looking at traces, not for passing this test. The prompt line is the command; the body is the test harness's own output, with cargo's build lines trimmed. The test also asserts the receipt id is 64 lowercase hex characters and differs from the source receipt id carried on the span.

The test constructs a decoded OTLP trace export with the locked GenAI tool-call attributes, exports it through chio-otel-receipt-exporter, verifies the signed receipt, and builds both lookup directions:

• receipt id -> span id (read provenance.otel.span_id out of the receipt metadata)
• span id -> receipt id (read chio.receipt.id attribute off the span)

It also holds the metric-safety line at the receipt: the two high-cardinality keys the collector deletes from the metric pipeline, gen_ai.tool.call.id and chio.receipt.id, are asserted absent from the attribute copy inside metadata.otel.attributes (tests/bidirectional_lookup.rs:59-64). The span's own receipt id is not lost, it moves: it lands at metadata.correlation.source_chio_receipt_id, while the receipt gets a fresh 64-hex id of its own.

Expected Span Shape

The contract's demo_span() helper (called from the test through export_demo_span()) builds a single span called gen_ai.tool.call with these attributes:

fn demo_span() -> OtlpSpan {
    OtlpSpan::new(TRACE_ID, SPAN_ID, GEN_AI_TOOL_CALL_SPAN_NAME)
        .with_attribute(ATTR_GEN_AI_SYSTEM, serde_json::json!("openai"))
        .with_attribute(
            ATTR_GEN_AI_OPERATION_NAME,
            serde_json::json!(GEN_AI_TOOL_CALL_OPERATION_NAME),
        )
        .with_attribute(ATTR_GEN_AI_REQUEST_MODEL, serde_json::json!("gpt-5"))
        .with_attribute(ATTR_GEN_AI_TOOL_CALL_ID, serde_json::json!("call-demo-1"))
        .with_attribute(ATTR_GEN_AI_TOOL_NAME, serde_json::json!(TOOL_NAME))
        .with_attribute(ATTR_GEN_AI_USAGE_INPUT_TOKENS, serde_json::json!(42))
        .with_attribute(ATTR_GEN_AI_USAGE_OUTPUT_TOKENS, serde_json::json!(7))
        .with_attribute(ATTR_CHIO_RECEIPT_ID, serde_json::json!(SOURCE_RECEIPT_ID))
        .with_attribute("chio.tenant.id", serde_json::json!(TENANT_ID))
        .with_attribute("chio.policy.ref", serde_json::json!(POLICY_HASH))
        .with_attribute("chio.verdict", serde_json::json!("allow"))
        .with_attribute("chio.tee.mode", serde_json::json!("shadow"))
        .with_attribute("chio.capability.id", serde_json::json!(CAPABILITY_ID))
        .with_attribute(ATTR_CHIO_SERVER_ID, serde_json::json!(TOOL_SERVER))
        .with_attribute(ATTR_CHIO_AGENT_ID, serde_json::json!("agent-demo"))
        .with_attribute(
            "redaction_pass_id",
            serde_json::json!("redactors@1.5.0+default"),
        )
        .with_attribute("redaction_elapsed_micros", serde_json::json!(12450_u64))

The ATTR_* names are not the example's: they are the kernel's own constants, imported from chio_kernel::otel (crates/kernel/chio-kernel/src/otel.rs:15-63), so the span the test builds carries the same attribute keys the kernel emits by construction rather than by agreement. GEN_AI_TOOL_CALL_SPAN_NAME is gen_ai.tool.call, GEN_AI_TOOL_CALL_OPERATION_NAME is tool.call, and ATTR_CHIO_RECEIPT_ID is chio.receipt.id. The values the example supplies are its own consts a few lines above the span (tests/support/mod.rs:19-27): SOURCE_RECEIPT_ID is a 64-hex placeholder, TOOL_NAME is customer_lookup, POLICY_HASH is policy-demo-otel, CAPABILITY_ID is cap-otel-demo, TOOL_SERVER is srv-openai-demo, and TENANT_ID is tenant-demo.

The two redaction_* attributes record the redaction pass that ran before the receipt was signed: redaction_pass_id pins the redactor set and version, redaction_elapsed_micros its wall-clock cost. Adapters that decorate spans with these attributes are compatible with the collector pipeline and the dashboards out of the box.

Sample Queries

With the collector running and traces flowing, slice the data from Tempo or Jaeger:

# TraceQL (Tempo): all denied tool calls in the last hour
{ span.chio.verdict = "deny" }

# TraceQL: lookup by receipt id
{ span.chio.receipt.id = "bb88956ef0fae73edcedebbea22f2c2f94a9fb0afd775160ebe9bc8d9024c415" }

# TraceQL: deny rate by policy
{ span.chio.verdict = "deny" } | by(span.chio.policy.ref)

# TraceQL: latency tail by tool
{ span.gen_ai.operation.name = "tool.call" } | quantile_over_time(span.duration, 0.99) by(span.gen_ai.tool.name)

# Jaeger: search by tag
chio.receipt.id="bb88956ef0fae73edcedebbea22f2c2f94a9fb0afd775160ebe9bc8d9024c415"
chio.verdict="deny"

Fuel and budget metrics flow on the metric pipeline (with receipt-id stripped). The chio dashboards under deploy/dashboards/ chart deny rate, fuel consumption per policy, and the p50/p99 latency tail per server.

On the metric side, the kernel histogram chio_guard_eval_duration_seconds is the most common Prometheus query. Its locked label set is guard_id and verdict. p99 by guard and verdict:

histogram_quantile(
  0.99,
  sum by (le, guard_id, verdict)(
    rate(chio_guard_eval_duration_seconds_bucket[5m])
  )
)

# Sample response (label set is guard_id, verdict):
# {guard_id="tool-gate",          verdict="allow"} 0.00042
# {guard_id="tool-gate",          verdict="deny"}  0.00061
# {guard_id="enriched-inspector", verdict="allow"} 0.00018

Wire It Into Your Service

To produce the same shape from a service running outside the kernel, pull chio-otel-receipt-exporter into your binary and emit spans with the attribute set above. The sink matches a span to its receipt through the ids on the span itself, so what matters is that chio.receipt.id and the trace and span ids are present and well formed. Nothing on this path deduplicates: the only span-id logic in the exporter is validate_span_id, a format check for 16 non-zero lowercase hex characters (crates/observability/chio-otel-receipt-exporter/src/sink.rs:452-459), so send each span once.