Trace a Call

One command runs four outcomes and verifies each trace

Start from chio new pricing-desk --example pricing-desk, or download the project, and run the comparison from pricing-desk. The project needs Rust and uv, listed in Builder Setup; rust-toolchain.toml selects Rust 1.94.1 and Cargo.lock pins the Chio crates.

uv run --locked trace.py --compare

complete: Allow; started=True; published=True; trace and receipt verified
refuse: Deny; started=False; published=False; trace and receipt verified
fail: Deny; started=True; published=False; trace and receipt verified
uncertain: Deny; started=True; published=True; trace and receipt verified
Records: state/trace-runs/f3c90265-9084-40ca-b023-f03a0914459e/comparison.json

You should see four lines, one per outcome, each ending in trace and receipt verified. Only the published sheet is Allow. The refused refund never started; the failed database started and published nothing; the lost acknowledgement started and published, and its receipt still records a denial.

The command builds the service, starts each case with its own authority and data, verifies its records, and stops its processes, with no collector account. Every run is retained under state/trace-runs. Choose your own discount with uv run --locked trace.py 25, from 0 to 100 percent, with port 4312 free.

Each repricing call leaves a decision, an effect, and a span

The kernel admits or refuses the repricing request and signs a receipt. The admitted tool writes the price sheet, and the service exports one span that points at the receipt.

The span is a search key and a duration. Its attributes are unsigned, so the verifier trusts the receipt's signature and checks the span against it. The price sheet is the effect, inspected separately from the signed decision, which matters most when a call fails part way.

The verifier pins the kernel key before it requests work

trace.py waits for the service, reads the public key of the kernel Larkfield runs, and only then submits the job.

            # Pin the public key from the kernel this operator owns, before requesting work.
            trusted_key = (state / "kernel-public-key.txt").read_text().strip()
            created = request("/jobs", token, {"percent": percent, "mode": mode})
            identity = created["job_id"]

Every retained record carries that key as trusted_kernel_key, and verify_receipt_with_trusted_signers accepts a receipt only if one of the trusted keys signed it. A receipt signed by any other key stops the run before any binding is compared.

The span and the receipt name each other in both directions

The verifier requires one request identity across the job, the span, and the signed receipt context, then compares the identifiers each record holds for the other.

    expected = record["job"]["request_id"]
    if not (expected == record["job"]["job_id"] == attributes["chio.request.id"] == metadata["receipt_context"]["request_id"]):
        raise ValueError("Trace, job and receipt name different requests")
    if attributes["chio.receipt.id"] != receipt["id"] or record["job"]["result"]["receipt_id"] != receipt["id"]:
        raise ValueError("Trace names another receipt")
    if span["traceId"] != metadata["trace_id"] or span["spanId"] != metadata["span_id"]:
        raise ValueError("Trace identifiers differ from their signed receipt metadata")

For the published run, the retained records line up field for field:

span     chio.request.id    a221a8a5-07b1-4f8b-97ef-3e81dcea8ef4
receipt  request_id         a221a8a5-07b1-4f8b-97ef-3e81dcea8ef4
span     chio.receipt.id    6cec4180f5626b7f5538d4911709c85125345edc6585e3376dd61f9a1d55b45a
receipt  id                 6cec4180f5626b7f5538d4911709c85125345edc6585e3376dd61f9a1d55b45a
span     traceId            62c68c5164854f4b85a6f637b768b1bc
receipt  trace_id           62c68c5164854f4b85a6f637b768b1bc
span     spanId             8f5329d991044eac
receipt  span_id            8f5329d991044eac

• chio.request.id. The submitted job and the receipt's signed request context.
• chio.receipt.id. The exported span and the exact retained receipt, a refusal included.
• trace_id and span_id. The receipt's signed metadata and its outgoing OTLP span.

The receipt binds the job input and the admitted output

The verdict on the span must match the retained outcome and the signed decision, and the receipt must describe the exact input the job submitted.

    if attributes["chio.verdict"] != record["job"]["result"]["verdict"]:
        raise ValueError("Trace verdict differs from the retained outcome")
    if record["job"]["result"]["verdict"].lower() != receipt["decision"]["verdict"]:
        raise ValueError("Displayed outcome differs from the signed decision")
    if record["input"] != receipt["action"]["parameters"]:
        raise ValueError("Receipt describes a different repricing input")
    if sha256_hex_utf8(canonicalize_json(record["input"])) != receipt["action"]["parameter_hash"]:
        raise ValueError("Request parameter hash changed")
    if receipt["decision"]["verdict"] == "allow":
        if sha256_hex_utf8(canonicalize_json(record["job"]["result"]["output"])) != receipt["content_hash"]:
            raise ValueError("Price sheet output differs from its signed content hash")

action.parameter_hash is the canonical hash of the job input, so a receipt for a different discount fails. For an allowed call, content_hash covers the returned price sheet, which ties the signed decision to the output the service kept. A span that ends before it starts fails the check as well.

The service retains each span before it sends one

The repricing service builds an OTLP/HTTP JSON batch with service.name larkfield-repricing-service and one reprice span carrying chio.request.id, chio.receipt.id, and chio.verdict. Status code 1 marks an Allow; every other verdict is 2.

use crate::*;

pub fn nanoseconds() -> String {
    std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .unwrap_or_default()
        .as_nanos()
        .to_string()
}

/// OTLP/HTTP JSON is useful for inspecting this small example's wire contract.
/// A larger application should use its usual batched OpenTelemetry exporter.
pub async fn export(
    path: &Path,
    id: &str,
    trace: &str,
    span: &str,
    started: &str,
    result: &Value,
) -> anyhow::Result<()> {
    let attributes = [
        ("chio.request.id", id.to_string()),
        (
            "chio.receipt.id",
            result["receipt_id"]
                .as_str()
                .unwrap_or("unavailable")
                .to_string(),
        ),
        (
            "chio.verdict",
            result["verdict"].as_str().unwrap_or("error").to_string(),
        ),
    ]
    .map(|(key, value)| json!({"key":key,"value":{"stringValue":value}}));
    let batch = json!({"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"larkfield-repricing-service"}}]},
    "scopeSpans":[{"scope":{"name":"larkfield-pricing-desk","version":"0.1.0"},"spans":[{
        "traceId":trace,"spanId":span,"name":"reprice","kind":2,
        "startTimeUnixNano":started,"endTimeUnixNano":nanoseconds(),"attributes":attributes,
        "status":{"code":if result["verdict"] == "Allow" {1} else {2}}
    }]}]}]});
    // Retain the exact export payload beside the job and signed receipt.
    private_file(
        &path.join("jobs").join(format!("{id}.trace.json")),
        &serde_json::to_vec_pretty(&batch)?,
    )?;
    if let Ok(endpoint) = std::env::var("OTEL_EXPORTER_OTLP_TRACES_ENDPOINT") {
        let response = reqwest::Client::builder()
            .timeout(std::time::Duration::from_secs(2))
            .build()?
            .post(endpoint)
            .json(&batch)
            .send()
            .await?
            .error_for_status()?;
        let acknowledgement: Value = response.json().await?;
        anyhow::ensure!(
            acknowledgement
                .get("partialSuccess")
                .is_none_or(
                    |partial| partial["rejectedSpans"].as_str().unwrap_or("0") == "0"
                        && partial["errorMessage"].as_str().unwrap_or("").is_empty()
                ),
            "Collector partially rejected the trace: {acknowledgement}"
        );
    }
    Ok(())
}

The payload is written to jobs/<job-id>.trace.json beside the job before any export. When OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set, the service posts the same batch with a 2-second timeout and fails the export if the collector reports rejected spans. A missing span in a collector means an export failed; the job, the receipt, and the payload on disk still say whether the tool ran.

Read the complete trace verifier.

The same runs reach an OpenTelemetry collector

The project's Compose file runs the OpenTelemetry Collector Contrib 0.115.1 image on 127.0.0.1:4318. Start it, point the exporter at it, and rerun the comparison.

docker compose up -d collector
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:4318/v1/traces uv run --locked trace.py --compare
docker compose logs collector
docker compose down

You should see four reprice spans in the collector log, received through the debug exporter at detailed verbosity. Each span's chio.receipt.id matches a receipt id in the run's comparison.json.

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch:
    timeout: 1s
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]

docker compose up -d collector
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:4318/v1/traces \
  cargo run --locked --bin repricing-service

chio --receipt-db state/repricing/receipts.db receipt explain RECEIPT_ID

A tool outside the grant is refused, signed, and traced

The refuse case asks for refund, a tool outside the job's grant. Run it alone with uv run --locked trace.py --mode refuse; the comparison already retained it.

{
  "started": false,
  "published": false,
  "artifact": null,
  "verdict": "Deny",
  "tool_name": "refund",
  "decision": {
    "guard": "kernel",
    "reason": "requested tool refund on server pricing is not in capability scope",
    "verdict": "deny"
  }
}

You should see no start, no publication, and no artifact, with a kernel decision naming the capability scope. The span carries the same receipt ID with verdict Deny and status code 2, and the verifier checks it like any other run. An error alone never says whether the agent changed anything, and the other two failures show why: the failed database leaves a one-row partial sheet, and the lost acknowledgement leaves a published sheet behind a denial receipt.

{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "larkfield-repricing-service"
            }
          }
        ]
      },
      "scopeSpans": [
        {
          "scope": {
            "name": "larkfield-pricing-desk",
            "version": "0.1.0"
          },
          "spans": [
            {
              "attributes": [
                {
                  "key": "chio.request.id",
                  "value": {
                    "stringValue": "a221a8a5-07b1-4f8b-97ef-3e81dcea8ef4"
                  }
                },
                {
                  "key": "chio.receipt.id",
                  "value": {
                    "stringValue": "6cec4180f5626b7f5538d4911709c85125345edc6585e3376dd61f9a1d55b45a"
                  }
                },
                {
                  "key": "chio.verdict",
                  "value": {
                    "stringValue": "Allow"
                  }
                }
              ],
              "endTimeUnixNano": "1791786890268675000",
              "kind": 2,
              "name": "reprice",
              "spanId": "8f5329d991044eac",
              "startTimeUnixNano": "1791786886855228000",
              "status": {
                "code": 1
              },
              "traceId": "62c68c5164854f4b85a6f637b768b1bc"
            }
          ]
        }
      ]
    }
  ]
}

Extend it: find receipts from Tempo and Jaeger

In production, Larkfield's collector fans traces out to Tempo and Jaeger instead of the debug exporter. Point the service's OTEL_EXPORTER_OTLP_TRACES_ENDPOINT at that collector; the span and the receipt are unchanged.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}
  attributes/chio-metric-safety:
    actions:
      - key: gen_ai.tool.call.id
        action: delete
      - key: chio.receipt.id
        action: delete
      - key: chio.request.id
        action: delete

exporters:
  otlp/tempo:
    endpoint: tempo.larkfield.example:4317
  otlp/jaeger:
    endpoint: jaeger.larkfield.example:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo, otlp/jaeger]
    metrics:
      receivers: [otlp]
      processors: [attributes/chio-metric-safety, batch]
      exporters: [otlp/tempo]

The metric pipeline deletes gen_ai.tool.call.id, chio.receipt.id, and chio.request.id: each is unique per evaluation, so as a metric label it would grow without bound. The trace pipeline keeps them, which is what makes the lookup work.

With traces in Tempo or Jaeger, search by the receipt ID from a retained record, or by verdict. Set the time range in the backend UI; rate queries need TraceQL metrics functions.

# TraceQL (Tempo): every refused repricing job
{ resource.service.name = "larkfield-repricing-service" && span.chio.verdict = "Deny" }

# TraceQL: the span for one receipt
{ span.chio.receipt.id = "6cec4180f5626b7f5538d4911709c85125345edc6585e3376dd61f9a1d55b45a" }

# TraceQL: refusal rate per span name
{ span.chio.verdict = "Deny" } | rate() by (name)

# Jaeger: search by tag
chio.receipt.id=6cec4180f5626b7f5538d4911709c85125345edc6585e3376dd61f9a1d55b45a
chio.verdict=Deny

Going the other way, the receipt's signed trace_id opens the trace directly: paste it into Jaeger's trace search or Tempo's trace ID field. Services that already emit GenAI tool-call spans can link chio-otel-receipt-exporter instead of writing OTLP by hand; it adds the same chio.receipt.id attribute and signs the matching receipt with trace_id and span_id in its metadata.

Next: route traces and receipts to their stores