Chio/Docs
LOGIN · JOIN

SIEM Export

Deliver every recorded decision to your SIEM at least once, resuming from a persisted cursor after an outage or restart.

Deployment chio@5f63e6e

On this page

Run chio-wall siem-export beside the receipt store, and every receipt the kernel records reaches your SIEM at least once, in log order.

The exporter is its own process. It opens the receipt database read-only and keeps its progress in a cursor database it owns, so delivery, retries, and backend outages stay outside the kernel's dispatch path.

Start
Export cursorChoose a moment.
Kkernel appends, signedreceipt log12345678910webhookacked 7splunk_hecacked 7elasticsearchacked 7read cursor = min(acked_seq)siem_dead_lettersKappendslog12345678910webhook 7splunk 7elastic 7read cursorsiem_dead_letters

Each exporter acknowledges its own mark

Every poll reads up to batch_size receipts past the read cursor (100 every 5 seconds by default), hands the batch to every exporter, and records each exporter’s acknowledged seq in the cursor store. Retries and redelivery.

Ten receipts and three exporters, illustrative. Each row is one exporter’s acknowledged high-water mark in the cursor store; rate limiting and the in-memory dead-letter queue are left out.

The manager scans by the receipt log's increasing seq, never by timestamp, so clock skew cannot reorder delivery. Each exporter acknowledges its own mark, and the read cursor waits for the slowest one, so a backend outage becomes a later redelivery of the same range.

1. Point the exporter at the receipt store

Configure at least one SOC export sink, then start the serve loop with the receipt database and a cursor database.

Terminalbash
export CHIO_SIEM_WEBHOOK_URL="https://siem.larkfield.example/chio/receipts"
export CHIO_SIEM_WEBHOOK_BEARER_TOKEN="$SIEM_WEBHOOK_TOKEN"
chio-wall siem-export \
  --receipt-db /var/lib/chio/receipts.sqlite3 \
  --cursor-db /var/lib/chio/chio-wall-siem-cursor.sqlite3

Both flags are required. --receipt-db is the kernel's receipt log, opened with one read-only connection that every poll reuses. --cursor-db is the exporter's read/write store for each exporter's high-water mark. Every 5 seconds by default, the exporter reads up to 100 receipts past the read cursor and hands them to every configured sink.

Run the exporter on the host that writes the receipt store; a remote file system works with a longer poll interval.

2. Verify that receipts are flowing

The exporter serves a Prometheus scrape on CHIO_SIEM_METRICS_ADDR, 127.0.0.1:9090 by default. On a host where a sidecar already listens on 9090, set it to another port. Read the export counter and the dead-letter depth.

Terminalbash
curl -s http://127.0.0.1:9090/metrics \
  | grep -E '^chio_(soc_export_total|dlq_depth)'
Output
chio_soc_export_total{exporter="webhook",outcome="success"} 1284
chio_dlq_depth{exporter="webhook"} 0

You should see the success count rise as the kernel records decisions, and a dead-letter depth of 0. The alert rule pack pages on the same families when export errors burn their budget or the counter goes missing.

3. Write detection content against the verified fields

Each receipt arrives wrapped in a SiemEvent that carries verification results beside the receipt. authorized is true only for an authoritative, Chio-mediated allow at a prevent boundary, with the signature, receipt id, parameter hash, and signer trust all verified. receipt_kind and boundary_class keep a trace or advisory observation from reading as an enforced decision.

A deny names the guard that refused it. This Splunk search counts refusals by the egress allowlist, per tool.

Splunk searchspl
sourcetype="chio:receipt" event.decision.verdict="deny" event.decision.guard="egress-allowlist"
| stats count by event.tool_server, event.tool_name
| sort - count

You should see one row per tool server and tool the egress allowlist refused. Guard names on deny receipts come from a fixed kebab-case set, listed with the OCSF mapping below, so detection content can key on them directly.

A failing sink is retried, then redelivered

Each exporter gets up to max_retries attempts per batch, 3 by default, with backoff doubling from 500 ms to 1 s to 2 s. Transient 5xx responses and network errors take the same loop.

  • Retries exhausted. The batch enters the in-memory dead-letter queue, the failed exporter keeps its high-water mark, and later polls redeliver its source range.
  • Partial acceptance. Elasticsearch can accept part of a batch; ExportError::PartialFailure marks it, and the unacknowledged part is retried.
  • Malformed row. A row that does not parse goes to the durable siem_dead_letters table in the cursor database before the cursor moves past it.
  • Diagnostic queue full. The in-memory dead-letter queue holds 1,000 entries by default; when full it drops the oldest and logs a tracing::error.

Inspect the failure class and the sink's deduplication before re-feeding data by hand.

A redelivered range reaches every sink again

At-least-once delivery means a redelivered range reaches exporters that already acknowledged it. Elasticsearch indexes each receipt with its id as the document _id, so a repeat overwrites the same document. Splunk HEC receives the id as event data; deduplicate downstream.

Each exporter keeps its own mark, so a credential rotation can run a second exporter with the new HEC token or API key beside the first, and Elasticsearch still writes each document once. After any restart, confirm that no duplicate documents land in the index and that your index template keeps the receipt id.

A rate limit delays a burst and drops nothing

With rate_limit configured, each exporter gets its own token bucket, keyed by exporter name. When a bucket is empty, the manager waits for capacity before it sends the next batch, so burst traffic arrives late and whole.

chio-siem.yamlyaml
siem:
  db_path: /var/lib/chio/receipts.sqlite
  poll_interval_ms: 5000
  batch_size: 100
  max_retries: 3
  base_backoff_ms: 500
  dlq_capacity: 1000

  rate_limit:
    splunk_hec:
      capacity: 500      # burst ceiling in receipts
      refill_per_sec: 50 # sustained rate
    elasticsearch:
      capacity: 1000
      refill_per_sec: 200

Three symptoms locate an export failure

The SIEM stream and the audit chain both read the kernel's append-only receipt log. Never drop or truncate it; that breaks both at once.

Dead-letter depth stays above zero

The downstream SIEM is failing. Page on any sustained nonzero dlq_len, then check the sink's health and credentials; the cursor holds the range until the sink accepts it.

The exporter refuses to start

No SOC export sink is configured; an alert backend alone does not count, because alerting and export are separate obligations. Set CHIO_SIEM_WEBHOOK_URL and its bearer token, or embed another exporter.

A row lands in siem_dead_letters

The row did not parse into a SiemEvent, usually from schema skew or a corrupted raw_json. Fix the cause and replay the row from the table; the receipt itself is still in the store.

Exporters, events, and configuration

Exporter modules

The chio-siem crate ships seven exporter modules under crates/observability/chio-siem/src/exporters/. Each registers through manager.add_exporter; the manager lives in crates/observability/chio-siem/src/manager.rs.

ExporterModuleSink
Splunk HEC
splunk.rs
Splunk HTTP Event Collector
Elasticsearch
elastic.rs
Elasticsearch _bulk NDJSON
OCSF
ocsf_exporter.rs
Any OCSF 1.3.0 Authorization sink
CEF
cef.rs
ArcSight and CEF syslog collectors
Datadog
datadog.rs
Datadog logs intake
Sumo Logic
sumo_logic.rs
Sumo Logic HTTP source
Webhook
webhook.rs
Generic bearer-auth webhook

Two more modules sit beside them: alerting.rs carries the PagerDuty and OpsGenie alert backends, and metrics_sink.rs exposes the Prometheus scrape. chio-wall siem-export is the product binary (crates/products/chio-wall/, public_entrypoint = true) that drives the ExporterManager serve loop and wires these from environment variables.

Environment variables

VariableEffect
CHIO_SIEM_WEBHOOK_URL, CHIO_SIEM_WEBHOOK_BEARER_TOKEN
Generic webhook SOC export sink
CHIO_SIEM_ALERT_PAGERDUTY_ROUTING_KEY, CHIO_SIEM_ALERT_PAGERDUTY_ENDPOINT
PagerDuty alert backend
CHIO_SIEM_ALERT_OPSGENIE_API_KEY, CHIO_SIEM_ALERT_OPSGENIE_ENDPOINT
OpsGenie alert backend
CHIO_SIEM_METRICS_ADDR
Prometheus scrape bind, default 127.0.0.1:9090

SiemEvent fields

The wrapper carries the raw ChioReceipt plus the verification and semantic fields a SIEM needs to tell a kernel-mediated allow from a trace or advisory observation, and the hoisted financial metadata.

crates/observability/chio-siem/src/event.rs15-39rust
pub struct SiemEvent {
    /// The full ChioReceipt as stored in the kernel receipt database.
    pub receipt: ChioReceipt,
    /// Semantic receipt class used to prevent trace/advisory observations from
    /// being rendered as authorization decisions.
    pub receipt_kind: String,
    /// Runtime mediation boundary for this receipt.
    pub boundary_class: String,
    /// Human-facing semantic result label.
    pub result: String,
    /// True when the receipt id, receipt signature, and action parameter hash verify.
    pub authoritative: bool,
    /// True when the embedded receipt signature verifies against the embedded kernel key.
    pub signature_valid: bool,
    /// True when the receipt id matches the canonical receipt body.
    pub receipt_id_valid: bool,
    /// True when the action parameter hash matches the canonical action parameters.
    pub parameter_hash_valid: bool,
    /// True when the receipt signer is pinned as a trusted kernel signer.
    pub signer_trusted: bool,
    /// True only for authoritative Chio-mediated allow receipts at a prevent boundary.
    pub authorized: bool,
    /// Financial metadata extracted from `receipt.metadata["financial"]`, if present.
    pub financial: Option<FinancialReceiptMetadata>,
}

authorized holds only when authoritative, signature_valid, receipt_id_valid, parameter_hash_valid, and signer_trusted all hold, for a Chio-mediated allow at a prevent boundary.

financial is extracted from receipt.metadata["financial"]. It exposes grant_index, cost_charged, currency, budget_remaining, budget_total, delegation_depth, root_budget_holder, and settlement_status, plus four fields omitted from the wire when absent: payment_reference, cost_breakdown, oracle_evidence, and attempted_cost, which only denial receipts populate.

Embedding the exporter manager

A Rust host can run the manager itself. It takes a SiemConfig, whose doc comments carry the defaults.

crates/observability/chio-siem/src/manager.rs35-61rust
pub struct SiemConfig {
    /// Path to the Chio kernel receipt SQLite database.
    pub db_path: PathBuf,
    /// Interval between polls for new receipts. Default: 5 seconds.
    pub poll_interval: Duration,
    /// Maximum number of receipts to read per poll cycle. Default: 100.
    pub batch_size: usize,
    /// Maximum number of retry attempts per exporter before DLQ. Default: 3.
    pub max_retries: u32,
    /// Base backoff in milliseconds for exponential retry (actual: base * 2^attempt). Default: 500.
    pub base_backoff_ms: u64,
    /// Maximum capacity of the dead-letter queue. Default: 1000.
    pub dlq_capacity: usize,
    /// Optional per-exporter batch rate limit. None means unlimited.
    pub rate_limit: Option<RateLimitConfig>,
    /// Kernel public keys trusted to produce authoritative receipts.
    pub trusted_kernel_keys: BTreeSet<String>,
    /// Explicit read authority for local receipt polling. SIEM polling is an
    /// operator surface and must not run with tenant-scoped authority.
    pub read_context: ReceiptReadContext,
    /// Optional path to the SIEM-owned RW cursor store (per-exporter high-water
    /// mark). When set, delivery is at-least-once: the read cursor resumes at
    /// min(acked_seq) so a failed exporter forces bounded redelivery instead of
    /// a silent skip. None keeps the legacy advance-regardless behavior. Distinct
    /// from the read-only receipt DB.
    pub cursor_db_path: Option<PathBuf>,
}
  1. Reuse the connection. The read-only SQLite connection opened at construction (SQLITE_OPEN_READ_ONLY | SQLITE_OPEN_NO_MUTEX) serves every tick, so the exporter can never mutate the receipt store.
  2. Read past the cursor. SELECT seq, raw_json FROM chio_tool_receipts WHERE seq > cursor ORDER BY seq ASC LIMIT batch_size.
  3. Wrap. Each row parses into a SiemEvent.
  4. Fan out. export_batch runs on every registered exporter.
  5. Advance. Each exporter's acknowledged high-water mark persists to cursor_db_path when set.

With cursor_db_path set, the read cursor resumes at the slowest exporter's mark, so an unacknowledged range is redelivered. Without it, an exhausted batch advances the cursor and a restart replays from zero.

src/main.rs · in mainrust
let (cancel_tx, cancel_rx) = tokio::sync::watch::channel(false);
manager.run(cancel_rx).await;

// To stop gracefully:
let _ = cancel_tx.send(true);

// Inspect DLQ depth from operator tooling:
let dlq_len = manager.dlq_len();

Splunk HEC exporter

The exporter POSTs newline-separated JSON envelopes to {endpoint}/services/collector/event. Each envelope wraps the full ChioReceipt under event, with time, sourcetype, and optional index and host. It lifts four wrapper fields into an indexed Splunk fields object: receipt_kind, boundary_class, result, and authorized, so a search can filter on authorized without parsing the receipt.

src/main.rs · in mainrust
use chio_egress_contract::HttpEgressContract;
use chio_siem::exporters::splunk::{SplunkConfig, SplunkHecExporter};
use std::collections::BTreeSet;
use std::time::Duration;

// Pin the exact scheme and authority the exporter may reach.
// SplunkHecExporter::new rejects a missing contract with ExportError::HttpError.
let egress_contract = HttpEgressContract {
    tenant_egress_namespace: "siem:splunk:splunk.larkfield.example:8088".to_string(),
    allowed_schemes: BTreeSet::from(["https".to_string()]),
    allowed_authority_set: BTreeSet::from(["splunk.larkfield.example:8088".to_string()]),
    deny_loopback: true,
    deny_link_local: true,
    deny_ipv6_ula: true,
    max_redirect_chain: 3,
    max_response_bytes: 1024 * 1024,
};

let config = SplunkConfig {
    endpoint: "https://splunk.larkfield.example:8088".to_string(),
    hec_token: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx".to_string(),
    sourcetype: "chio:receipt".to_string(),
    index: Some("chio_audit".to_string()),
    host: Some("chio-node-01".to_string()),
    timeout: Duration::from_secs(30),
    egress_contract: Some(egress_contract),
};

let exporter = SplunkHecExporter::new(config)?;
manager.add_exporter(Box::new(exporter));

The Authorization header is Splunk {hec_token}. reqwest handles TLS against the system native certificate store. Every dispatch and every redirect hop runs through the HttpEgressContract; a config without one compiles and fails at new() with "Splunk HEC exporter requires an HttpEgressContract". The timeout caps a stalled collector so it cannot block the poll loop.

Sample SPL

Decision is an internally tagged enum, #[serde(tag = "verdict", rename_all = "snake_case")]. A deny is {"verdict":"deny","reason":...,"guard":...}, with the guard at event.decision.guard, one level up from where an externally tagged enum would put it. Budget denials carry "guard":"kernel": the kernel refuses the charge in its own deny path, ahead of any pipeline guard, and metadata.financial.attempted_cost is populated only on denial receipts.

Splunk searchspl
sourcetype="chio:receipt" event.decision.verdict="deny" event.decision.guard="kernel"
| stats sum(event.metadata.financial.attempted_cost) as total_attempted
        by event.capability_id
| sort - total_attempted

Elasticsearch bulk exporter

The exporter POSTs NDJSON to {endpoint}/_bulk, two lines per receipt: an index action keyed on receipt.id as _id, which makes the write idempotent, and the full receipt document. A partial failure (HTTP 200 with errors: true) surfaces as ExportError::PartialFailure.

src/main.rs · in mainrust
use chio_egress_contract::HttpEgressContract;
use chio_siem::exporters::elastic::{
    ElasticAuthConfig, ElasticConfig, ElasticsearchExporter,
};
use std::collections::BTreeSet;
use std::time::Duration;

let egress_contract = HttpEgressContract {
    tenant_egress_namespace: "siem:elastic:es.larkfield.example:9200".to_string(),
    allowed_schemes: BTreeSet::from(["https".to_string()]),
    allowed_authority_set: BTreeSet::from(["es.larkfield.example:9200".to_string()]),
    deny_loopback: true,
    deny_link_local: true,
    deny_ipv6_ula: true,
    max_redirect_chain: 3,
    max_response_bytes: 1024 * 1024,
};

let config = ElasticConfig {
    endpoint: "https://es.larkfield.example:9200".to_string(),
    index_name: "chio-receipts".to_string(),
    auth: ElasticAuthConfig::ApiKey("base64encodedkey==".to_string()),
    // or Basic { username, password }
    timeout: Duration::from_secs(30),
    egress_contract: Some(egress_contract),
};

let exporter = ElasticsearchExporter::new(config)?;
manager.add_exporter(Box::new(exporter));

ElasticsearchExporter::new rejects a missing egress_contract with "Elasticsearch exporter requires an HttpEgressContract".

Sample Elasticsearch DSL

es/deny-by-guard.jsonjson
POST chio-receipts/_search
{
  "query": {
    "bool": {
      "filter": [
        { "term": { "decision.verdict": "deny" } },
        { "range": { "timestamp": { "gte": "now-24h/h" } } }
      ]
    }
  },
  "aggs": {
    "by_guard": {
      "terms": { "field": "decision.guard", "size": 20 }
    }
  }
}
es/top-spenders.jsonjson
POST chio-receipts/_search
{
  "size": 0,
  "query": { "term": { "decision.verdict": "allow" } },
  "aggs": {
    "by_subject": {
      "terms": {
        "field": "metadata.financial.root_budget_holder",
        "size": 10
      },
      "aggs": {
        "spend": {
          "sum": { "field": "metadata.financial.cost_charged" }
        }
      }
    }
  }
}

OCSF mapping

The OCSF exporter (exporters/ocsf_exporter.rs) uses siem_event_to_ocsf to normalize each SiemEvent to the OCSF 1.3.0 Authorization event class (category 3, Identity & Access Management; class_uid 3002). It keeps the wrapper's signer-trust and verification state and computes no authorization from the receipt alone.

Chio receipt fieldOCSF fieldNotes
id
metadata.uid
UUIDv7, directly usable for pivoting
timestamp
time
Unix seconds, multiplied by 1000 for OCSF milliseconds
tool_server
dst_endpoint.name
Tool server handling the call
tool_name
api.operation
Tool invoked
action.parameters
api.request.data
Redacted tool-call parameters
decision (verdict)
activity_id, status_id, severity_id
Drives activity, status, severity, and type_uid
decision.reason (deny)
status_detail
Human-facing deny reason
decision.guard (deny)
unmapped.chio.guard
Kebab-case guard name on deny
policy_hash
policy.uid
Policy the decision resolved against
capability_id
observables[*], unmapped.chio.capability_id
Capability exercised
evidence[]
enrichments[*]
One enrichment per guard
tenant_id
unmapped.chio.tenant_id
Present when the receipt is tenant-scoped
full canonical JSON
raw_data
A serialize failure falls back to an Unknown event that keeps class_uid 3002

Guard names on deny receipts come from a fixed kebab-case set: forbidden-path, path-allowlist, shell-command, egress-allowlist, mcp-tool, secret-leak, patch-integrity, velocity.

Next: watch the node that writes the receipts