SIEM Export
Deliver every recorded decision to your SIEM at least once, resuming from a persisted cursor after an outage or restart.
On this page
Run chio-wall siem-export beside the receipt store, and every receipt the kernel records reaches your SIEM at least once, in log order.
The exporter is its own process. It opens the receipt database read-only and keeps its progress in a cursor database it owns, so delivery, retries, and backend outages stay outside the kernel's dispatch path.
StartEach exporter acknowledges its own mark
Every poll reads up to batch_size receipts past the read cursor (100 every 5 seconds by default), hands the batch to every exporter, and records each exporter’s acknowledged seq in the cursor store. Retries and redelivery.
The manager scans by the receipt log's increasing seq, never by timestamp, so clock skew cannot reorder delivery. Each exporter acknowledges its own mark, and the read cursor waits for the slowest one, so a backend outage becomes a later redelivery of the same range.
1. Point the exporter at the receipt store
Configure at least one SOC export sink, then start the serve loop with the receipt database and a cursor database.
export CHIO_SIEM_WEBHOOK_URL="https://siem.larkfield.example/chio/receipts"
export CHIO_SIEM_WEBHOOK_BEARER_TOKEN="$SIEM_WEBHOOK_TOKEN"
chio-wall siem-export \
--receipt-db /var/lib/chio/receipts.sqlite3 \
--cursor-db /var/lib/chio/chio-wall-siem-cursor.sqlite3Both flags are required. --receipt-db is the kernel's receipt log, opened with one read-only connection that every poll reuses. --cursor-db is the exporter's read/write store for each exporter's high-water mark. Every 5 seconds by default, the exporter reads up to 100 receipts past the read cursor and hands them to every configured sink.
Run the exporter on the host that writes the receipt store; a remote file system works with a longer poll interval.
2. Verify that receipts are flowing
The exporter serves a Prometheus scrape on CHIO_SIEM_METRICS_ADDR, 127.0.0.1:9090 by default. On a host where a sidecar already listens on 9090, set it to another port. Read the export counter and the dead-letter depth.
curl -s http://127.0.0.1:9090/metrics \
| grep -E '^chio_(soc_export_total|dlq_depth)'chio_soc_export_total{exporter="webhook",outcome="success"} 1284
chio_dlq_depth{exporter="webhook"} 0You should see the success count rise as the kernel records decisions, and a dead-letter depth of 0. The alert rule pack pages on the same families when export errors burn their budget or the counter goes missing.
3. Write detection content against the verified fields
Each receipt arrives wrapped in a SiemEvent that carries verification results beside the receipt. authorized is true only for an authoritative, Chio-mediated allow at a prevent boundary, with the signature, receipt id, parameter hash, and signer trust all verified. receipt_kind and boundary_class keep a trace or advisory observation from reading as an enforced decision.
A deny names the guard that refused it. This Splunk search counts refusals by the egress allowlist, per tool.
sourcetype="chio:receipt" event.decision.verdict="deny" event.decision.guard="egress-allowlist"
| stats count by event.tool_server, event.tool_name
| sort - countYou should see one row per tool server and tool the egress allowlist refused. Guard names on deny receipts come from a fixed kebab-case set, listed with the OCSF mapping below, so detection content can key on them directly.
A failing sink is retried, then redelivered
Each exporter gets up to max_retries attempts per batch, 3 by default, with backoff doubling from 500 ms to 1 s to 2 s. Transient 5xx responses and network errors take the same loop.
- Retries exhausted. The batch enters the in-memory dead-letter queue, the failed exporter keeps its high-water mark, and later polls redeliver its source range.
- Partial acceptance. Elasticsearch can accept part of a batch;
ExportError::PartialFailuremarks it, and the unacknowledged part is retried. - Malformed row. A row that does not parse goes to the durable
siem_dead_letterstable in the cursor database before the cursor moves past it. - Diagnostic queue full. The in-memory dead-letter queue holds 1,000 entries by default; when full it drops the oldest and logs a
tracing::error.
Inspect the failure class and the sink's deduplication before re-feeding data by hand.
A redelivered range reaches every sink again
At-least-once delivery means a redelivered range reaches exporters that already acknowledged it. Elasticsearch indexes each receipt with its id as the document _id, so a repeat overwrites the same document. Splunk HEC receives the id as event data; deduplicate downstream.
Each exporter keeps its own mark, so a credential rotation can run a second exporter with the new HEC token or API key beside the first, and Elasticsearch still writes each document once. After any restart, confirm that no duplicate documents land in the index and that your index template keeps the receipt id.
A rate limit delays a burst and drops nothing
With rate_limit configured, each exporter gets its own token bucket, keyed by exporter name. When a bucket is empty, the manager waits for capacity before it sends the next batch, so burst traffic arrives late and whole.
siem:
db_path: /var/lib/chio/receipts.sqlite
poll_interval_ms: 5000
batch_size: 100
max_retries: 3
base_backoff_ms: 500
dlq_capacity: 1000
rate_limit:
splunk_hec:
capacity: 500 # burst ceiling in receipts
refill_per_sec: 50 # sustained rate
elasticsearch:
capacity: 1000
refill_per_sec: 200Three symptoms locate an export failure
The SIEM stream and the audit chain both read the kernel's append-only receipt log. Never drop or truncate it; that breaks both at once.
Dead-letter depth stays above zero
The downstream SIEM is failing. Page on any sustained nonzero dlq_len, then check the sink's health and credentials; the cursor holds the range until the sink accepts it.
The exporter refuses to start
No SOC export sink is configured; an alert backend alone does not count, because alerting and export are separate obligations. Set CHIO_SIEM_WEBHOOK_URL and its bearer token, or embed another exporter.
A row lands in siem_dead_letters
The row did not parse into a SiemEvent, usually from schema skew or a corrupted raw_json. Fix the cause and replay the row from the table; the receipt itself is still in the store.
Exporters, events, and configuration
Search for a variable, an exporter, a field, or an OCSF mapping.
Exporter modules
The chio-siem crate ships seven exporter modules under crates/observability/chio-siem/src/exporters/. Each registers through manager.add_exporter; the manager lives in crates/observability/chio-siem/src/manager.rs.
| Exporter | Module | Sink |
|---|---|---|
Splunk HEC | splunk.rs | Splunk HTTP Event Collector |
Elasticsearch | elastic.rs | Elasticsearch _bulk NDJSON |
OCSF | ocsf_ | Any OCSF 1.3.0 Authorization sink |
CEF | cef.rs | ArcSight and CEF syslog collectors |
Datadog | datadog.rs | Datadog logs intake |
Sumo Logic | sumo_logic.rs | Sumo Logic HTTP source |
Webhook | webhook.rs | Generic bearer-auth webhook |
Two more modules sit beside them: alerting.rs carries the PagerDuty and OpsGenie alert backends, and metrics_sink.rs exposes the Prometheus scrape. chio-wall siem-export is the product binary (crates/products/chio-wall/, public_entrypoint = true) that drives the ExporterManager serve loop and wires these from environment variables.
Environment variables
| Variable | Effect |
|---|---|
CHIO_, CHIO_ | Generic webhook SOC export sink |
CHIO_, CHIO_ | PagerDuty alert backend |
CHIO_, CHIO_ | OpsGenie alert backend |
CHIO_ | Prometheus scrape bind, default 127.0.0.1:9090 |
SiemEvent fields
The wrapper carries the raw ChioReceipt plus the verification and semantic fields a SIEM needs to tell a kernel-mediated allow from a trace or advisory observation, and the hoisted financial metadata.
pub struct SiemEvent {
/// The full ChioReceipt as stored in the kernel receipt database.
pub receipt: ChioReceipt,
/// Semantic receipt class used to prevent trace/advisory observations from
/// being rendered as authorization decisions.
pub receipt_kind: String,
/// Runtime mediation boundary for this receipt.
pub boundary_class: String,
/// Human-facing semantic result label.
pub result: String,
/// True when the receipt id, receipt signature, and action parameter hash verify.
pub authoritative: bool,
/// True when the embedded receipt signature verifies against the embedded kernel key.
pub signature_valid: bool,
/// True when the receipt id matches the canonical receipt body.
pub receipt_id_valid: bool,
/// True when the action parameter hash matches the canonical action parameters.
pub parameter_hash_valid: bool,
/// True when the receipt signer is pinned as a trusted kernel signer.
pub signer_trusted: bool,
/// True only for authoritative Chio-mediated allow receipts at a prevent boundary.
pub authorized: bool,
/// Financial metadata extracted from `receipt.metadata["financial"]`, if present.
pub financial: Option<FinancialReceiptMetadata>,
}authorized holds only when authoritative, signature_valid, receipt_id_valid, parameter_hash_valid, and signer_trusted all hold, for a Chio-mediated allow at a prevent boundary.
financial is extracted from receipt.metadata["financial"]. It exposes grant_index, cost_charged, currency, budget_remaining, budget_total, delegation_depth, root_budget_holder, and settlement_status, plus four fields omitted from the wire when absent: payment_reference, cost_breakdown, oracle_evidence, and attempted_cost, which only denial receipts populate.
Embedding the exporter manager
A Rust host can run the manager itself. It takes a SiemConfig, whose doc comments carry the defaults.
pub struct SiemConfig {
/// Path to the Chio kernel receipt SQLite database.
pub db_path: PathBuf,
/// Interval between polls for new receipts. Default: 5 seconds.
pub poll_interval: Duration,
/// Maximum number of receipts to read per poll cycle. Default: 100.
pub batch_size: usize,
/// Maximum number of retry attempts per exporter before DLQ. Default: 3.
pub max_retries: u32,
/// Base backoff in milliseconds for exponential retry (actual: base * 2^attempt). Default: 500.
pub base_backoff_ms: u64,
/// Maximum capacity of the dead-letter queue. Default: 1000.
pub dlq_capacity: usize,
/// Optional per-exporter batch rate limit. None means unlimited.
pub rate_limit: Option<RateLimitConfig>,
/// Kernel public keys trusted to produce authoritative receipts.
pub trusted_kernel_keys: BTreeSet<String>,
/// Explicit read authority for local receipt polling. SIEM polling is an
/// operator surface and must not run with tenant-scoped authority.
pub read_context: ReceiptReadContext,
/// Optional path to the SIEM-owned RW cursor store (per-exporter high-water
/// mark). When set, delivery is at-least-once: the read cursor resumes at
/// min(acked_seq) so a failed exporter forces bounded redelivery instead of
/// a silent skip. None keeps the legacy advance-regardless behavior. Distinct
/// from the read-only receipt DB.
pub cursor_db_path: Option<PathBuf>,
}- Reuse the connection. The read-only SQLite connection opened at construction (
SQLITE_OPEN_READ_ONLY | SQLITE_OPEN_NO_MUTEX) serves every tick, so the exporter can never mutate the receipt store. - Read past the cursor.
SELECT seq, raw_json FROM chio_tool_receipts WHERE seq > cursor ORDER BY seq ASC LIMIT batch_size. - Wrap. Each row parses into a
SiemEvent. - Fan out.
export_batchruns on every registered exporter. - Advance. Each exporter's acknowledged high-water mark persists to
cursor_db_pathwhen set.
With cursor_db_path set, the read cursor resumes at the slowest exporter's mark, so an unacknowledged range is redelivered. Without it, an exhausted batch advances the cursor and a restart replays from zero.
let (cancel_tx, cancel_rx) = tokio::sync::watch::channel(false);
manager.run(cancel_rx).await;
// To stop gracefully:
let _ = cancel_tx.send(true);
// Inspect DLQ depth from operator tooling:
let dlq_len = manager.dlq_len();Splunk HEC exporter
The exporter POSTs newline-separated JSON envelopes to {endpoint}/services/collector/event. Each envelope wraps the full ChioReceipt under event, with time, sourcetype, and optional index and host. It lifts four wrapper fields into an indexed Splunk fields object: receipt_kind, boundary_class, result, and authorized, so a search can filter on authorized without parsing the receipt.
use chio_egress_contract::HttpEgressContract;
use chio_siem::exporters::splunk::{SplunkConfig, SplunkHecExporter};
use std::collections::BTreeSet;
use std::time::Duration;
// Pin the exact scheme and authority the exporter may reach.
// SplunkHecExporter::new rejects a missing contract with ExportError::HttpError.
let egress_contract = HttpEgressContract {
tenant_egress_namespace: "siem:splunk:splunk.larkfield.example:8088".to_string(),
allowed_schemes: BTreeSet::from(["https".to_string()]),
allowed_authority_set: BTreeSet::from(["splunk.larkfield.example:8088".to_string()]),
deny_loopback: true,
deny_link_local: true,
deny_ipv6_ula: true,
max_redirect_chain: 3,
max_response_bytes: 1024 * 1024,
};
let config = SplunkConfig {
endpoint: "https://splunk.larkfield.example:8088".to_string(),
hec_token: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx".to_string(),
sourcetype: "chio:receipt".to_string(),
index: Some("chio_audit".to_string()),
host: Some("chio-node-01".to_string()),
timeout: Duration::from_secs(30),
egress_contract: Some(egress_contract),
};
let exporter = SplunkHecExporter::new(config)?;
manager.add_exporter(Box::new(exporter));The Authorization header is Splunk {hec_token}. reqwest handles TLS against the system native certificate store. Every dispatch and every redirect hop runs through the HttpEgressContract; a config without one compiles and fails at new() with "Splunk HEC exporter requires an HttpEgressContract". The timeout caps a stalled collector so it cannot block the poll loop.
Sample SPL
Decision is an internally tagged enum, #[serde(tag = "verdict", rename_all = "snake_case")]. A deny is {"verdict":"deny","reason":...,"guard":...}, with the guard at event.decision.guard, one level up from where an externally tagged enum would put it. Budget denials carry "guard":"kernel": the kernel refuses the charge in its own deny path, ahead of any pipeline guard, and metadata.financial.attempted_cost is populated only on denial receipts.
sourcetype="chio:receipt" event.decision.verdict="deny" event.decision.guard="kernel"
| stats sum(event.metadata.financial.attempted_cost) as total_attempted
by event.capability_id
| sort - total_attemptedElasticsearch bulk exporter
The exporter POSTs NDJSON to {endpoint}/_bulk, two lines per receipt: an index action keyed on receipt.id as _id, which makes the write idempotent, and the full receipt document. A partial failure (HTTP 200 with errors: true) surfaces as ExportError::PartialFailure.
use chio_egress_contract::HttpEgressContract;
use chio_siem::exporters::elastic::{
ElasticAuthConfig, ElasticConfig, ElasticsearchExporter,
};
use std::collections::BTreeSet;
use std::time::Duration;
let egress_contract = HttpEgressContract {
tenant_egress_namespace: "siem:elastic:es.larkfield.example:9200".to_string(),
allowed_schemes: BTreeSet::from(["https".to_string()]),
allowed_authority_set: BTreeSet::from(["es.larkfield.example:9200".to_string()]),
deny_loopback: true,
deny_link_local: true,
deny_ipv6_ula: true,
max_redirect_chain: 3,
max_response_bytes: 1024 * 1024,
};
let config = ElasticConfig {
endpoint: "https://es.larkfield.example:9200".to_string(),
index_name: "chio-receipts".to_string(),
auth: ElasticAuthConfig::ApiKey("base64encodedkey==".to_string()),
// or Basic { username, password }
timeout: Duration::from_secs(30),
egress_contract: Some(egress_contract),
};
let exporter = ElasticsearchExporter::new(config)?;
manager.add_exporter(Box::new(exporter));ElasticsearchExporter::new rejects a missing egress_contract with "Elasticsearch exporter requires an HttpEgressContract".
Sample Elasticsearch DSL
POST chio-receipts/_search
{
"query": {
"bool": {
"filter": [
{ "term": { "decision.verdict": "deny" } },
{ "range": { "timestamp": { "gte": "now-24h/h" } } }
]
}
},
"aggs": {
"by_guard": {
"terms": { "field": "decision.guard", "size": 20 }
}
}
}POST chio-receipts/_search
{
"size": 0,
"query": { "term": { "decision.verdict": "allow" } },
"aggs": {
"by_subject": {
"terms": {
"field": "metadata.financial.root_budget_holder",
"size": 10
},
"aggs": {
"spend": {
"sum": { "field": "metadata.financial.cost_charged" }
}
}
}
}
}OCSF mapping
The OCSF exporter (exporters/ocsf_exporter.rs) uses siem_event_to_ocsf to normalize each SiemEvent to the OCSF 1.3.0 Authorization event class (category 3, Identity & Access Management; class_uid 3002). It keeps the wrapper's signer-trust and verification state and computes no authorization from the receipt alone.
| Chio receipt field | OCSF field | Notes |
|---|---|---|
id | metadata.uid | UUIDv7, directly usable for pivoting |
timestamp | time | Unix seconds, multiplied by 1000 for OCSF milliseconds |
tool_server | dst_ | Tool server handling the call |
tool_name | api.operation | Tool invoked |
action. | api. | Redacted tool-call parameters |
decision (verdict) | activity_id, status_id, severity_id | Drives activity, status, severity, and type_uid |
decision.reason (deny) | status_detail | Human-facing deny reason |
decision.guard (deny) | unmapped. | Kebab-case guard name on deny |
policy_hash | policy.uid | Policy the decision resolved against |
capability_id | observables[*], unmapped. | Capability exercised |
evidence[] | enrichments[*] | One enrichment per guard |
tenant_id | unmapped. | Present when the receipt is tenant-scoped |
full canonical JSON | raw_data | A serialize failure falls back to an Unknown event that keeps class_uid 3002 |
Guard names on deny receipts come from a fixed kebab-case set: forbidden-path, path-allowlist, shell-command, egress-allowlist, mcp-tool, secret-leak, patch-integrity, velocity.