Chio/Docs
LOGIN · JOIN

Relay Operations

Run a signals relay between two operators, deliver queued batches on a schedule, and read each delivery outcome.

Deployment chio@5f63e6e

On this page

Two operators exchange signed signal deposits through a relay. The sender keeps a durable outbox and delivers on a schedule; the receiver admits each frame under its own policy and reports back.

Signals are the advisory observations agents deposit to rank work, and a deposit grants nothing; action execution stays with each kernel. The relay runs as chio pheromone relay: a long-running listener on each side and a separate scheduled tick that drains the sender’s outbox.

Start
Relay crossingChoose an outcome.
sending operatorreceiving operatorenqueuedue rowssigned POSTframesadmitsreceive reportdeliveredsignsverifiesaagentqoutboxσtickdlistenerKruntime admissionsadmission storesending operatorreceiving operatorenqueuedue rowssigned POSTframesadmitsreceive reportdeliveredsignsverifiesaagentqoutboxσtickdlistenerKruntime admissionsadmission store

The receiver admits the batch and its report closes the row

The tick leases a due row, signs an HTTP envelope with the sender’s key, and posts the batch. The listener checks the envelope, the nonce, and the sender’s directory scope, then hands each frame to the receiver’s runtime admission. The receive report travels back, and an accepted report marks the row delivered. Deliver one batch.

One relay tick over HTTPS between two operators, with the sender’s retry schedule folded into one label. Catch-up, the observability routes, and the optional iroh lane are left out.

The operator boundary runs between the tick and the listener. The sender signs each request with its own key, the receiver verifies it against its pinned peer directory before runtime admission sees a frame, and the receive report closes the sender’s outbox row.

1. Prepare a sender and start its receiver

Complete the Rust project setup and install the Chio CLI. In a fresh project directory, the setup command creates independent sender, receiver, and passport keys, two peer directories, and one signed observation in the sender’s SQLite outbox.

Terminalbash
cargo run --locked --features relay --bin relay -- setup
Output
relay-state: fresh keys, two directories, one signed observation queued

You should see the fresh keys, the two directories, and one queued observation. Setup refuses to overwrite an existing relay-state directory, and on Unix it writes the private key files with owner-only permissions. Start the receiver in the same directory and leave it running:

Terminal · receiverbash
cargo run --locked --features relay --bin relay-serve -- relay-state

This receiver installs the documented one-frame receiver on the relay library. It validates the admitted passport’s signature, nonce, and observation capacity, and its local peer directories pin both identities. The CLI receiver adds the proof-package and transit-policy contract on top of these checks.

2. Deliver one batch with a scheduler tick

In a second terminal, check the receiver’s readiness, then run one tick and read its report.

Terminalbash
curl --fail http://127.0.0.1:18080/v1/chio/pheromone/ready
Output
{"schema":"chio.pheromone.relay-health-report.v1","accepted":true,"code":"accepted","detail":"relay health evaluated from durable store state","localKernelId":"did:chio:buyer-kernel","generatedAtUnixMs":1789054258421,"peerDirectoryVersion":null,"queueDepth":0,"oldestPendingAgeMs":null,"retryCount":0,"deadLetterCount":0,"inboxCount":0,"cursorCount":0,"staleLeaseCount":0,"checks":[{"code":"store.connected","accepted":true,"detail":"SQLite relay store is reachable"},{"code":"outbox.pressure","accepted":true,"detail":"queue_depth=0"},{"code":"leases.fresh","accepted":true,"detail":"stale_lease_count=0"}]}
Terminalbash
chio pheromone relay tick \
  --store relay-state/sender.sqlite \
  --peer-directory relay-state/sender-directory.json \
  --signing-key relay-state/sender-key.json \
  --profile local-dev \
  --max-batches 1 \
  --report relay-state/tick.json
cat relay-state/tick.json
Output
{
  "schema": "chio.pheromone.relay-tick-report.v1",
  "accepted": true,
  "delivered": 1,
  "retried": 0,
  "deadLettered": 0,
  "duplicateIdempotent": 0,
  "failures": []
}

You should see accepted true and one batch delivered, with no retries or failures. A completed tick writes its report and exits zero even when accepted is false. A zero process exit therefore does not mean every batch was delivered: read failures and the per-outcome counts. Configuration, storage, and report-writing errors stop the command before that path.

Terminal · second tickbash
chio pheromone relay tick \
  --store relay-state/sender.sqlite \
  --peer-directory relay-state/sender-directory.json \
  --signing-key relay-state/sender-key.json \
  --profile local-dev \
  --max-batches 1 \
  --report relay-state/tick.json
cat relay-state/tick.json
Output
{
  "schema": "chio.pheromone.relay-tick-report.v1",
  "accepted": true,
  "delivered": 0,
  "retried": 0,
  "deadLettered": 0,
  "duplicateIdempotent": 0,
  "failures": []
}

You should see a second tick that finds no due work. A tick uses one evaluation time for its lease, request timestamps, and retry schedule; --now-unix-ms sets it and defaults to the local clock. A past time suits controlled fixtures and fails freshness checks against a live receiver.

Production ticks take the active peer-directory state, its trusted issuer document, and --profile production; see Production flags and transport.

3. Confirm the receiver admitted the observation

Open relay-state/reports and find the receiver’s admission event for the batch the tick delivered. Then stop the receiver with Ctrl-C; keep the private key files in the operator’s directory.

You should see one admitted observation on the receiving side, matching the sender’s delivered row. The downloadable example below also runs the outage path with an injected scheduler clock: it queues while the receiver is offline and delivers after the retry deadline.

Worked Rust example: Deliver a signed observation between two local peers

Deliver a signed observation between two local peers

Create distinct peer keys and directories, persist a batch in SQLite, and tick the scheduler while the receiver is offline. Start the receiver and tick past the retry deadline; verify that one passport-signed observation reached its admission store.

Use the Rust example setup, or download the complete source project. The files below belong to that project; shared constructors and configuration are included in the download.

Run it

bash
cargo run --locked --features relay --bin relay

Implementation

relay.rsrust
use swarm_authority_examples::observation;
#[path = "../relay_directory.rs"]
mod relay_directory;
use chio_core_types::Keypair;
use chio_federation::pheromone_gossip::*;
use chio_pheromone::*;
use chio_pheromone_relay::*;

use std::sync::Arc;
use swarm_authority_examples::now_ms;

#[path = "../relay_receiver.rs"]
mod relay_receiver;
use relay_receiver::Receiver;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    if std::env::args().nth(1).as_deref() == Some("setup") {
        return setup();
    }
    let sender = Keypair::generate();
    let recipient = Keypair::generate();
    let passport = Keypair::generate();
    // Reserve an available loopback address, then close it for the outage case.
    let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await?;
    let address = listener.local_addr()?;
    drop(listener);
    let endpoint = format!("http://{address}");
    let receiver_document = relay_directory::directory(&sender, endpoint.clone());
    let sender_document = relay_directory::client_directory(&recipient, endpoint);
    let receiver_directory = PeerDirectory::from_document(receiver_document.clone(), now_ms())?;
    let sender_directory = PeerDirectory::from_document(sender_document.clone(), now_ms())?;
    let run_dir = format!("relay-run-{}", now_ms());
    std::fs::create_dir(&run_dir)?;
    std::fs::write(
        format!("{run_dir}/sender-directory.json"),
        serde_json::to_vec_pretty(&sender_document)?,
    )?;
    std::fs::write(
        format!("{run_dir}/receiver-directory.json"),
        serde_json::to_vec_pretty(&receiver_document)?,
    )?;
    let outbox = SqlitePheromoneRelayStore::open(format!("{run_dir}/sender.sqlite"))?;
    let inbox = Arc::new(SqlitePheromoneRelayStore::open(format!(
        "{run_dir}/receiver.sqlite"
    ))?);
    let deposit = sign_deposit(observation::body(&passport), &passport)?;
    let batch = PheromoneGossipBatch {
        schema: PHEROMONE_GOSSIP_BATCH_SCHEMA.into(),
        recipient_kernel_id: "did:chio:buyer-kernel".into(),
        treaty_id: "treaty:buyer-llamaworks:support-ops".into(),
        flushed_at_unix_ms: now_ms(),
        frames: vec![PheromoneDepositGossip {
            schema: PHEROMONE_GOSSIP_SCHEMA.into(),
            deposit,
            origin_kernel_id: "did:chio:llamaworks".into(),
            gossiping_peer_kernel_id: "did:chio:llamaworks".into(),
            treaty_id: "treaty:buyer-llamaworks:support-ops".into(),
            ts_unix_ms: now_ms(),
            transit_chain: None,
        }],
    };
    outbox.enqueue_batch(
        "did:chio:llamaworks",
        "did:chio:buyer-kernel",
        &batch.treaty_id,
        &batch,
        now_ms(),
    )?;
    let unavailable = deliver_due_batches(
        &outbox,
        sender_directory.clone(),
        sender.clone(),
        "did:chio:llamaworks",
        now_ms(),
        1,
    )
    .await?;
    assert_eq!(unavailable.retried, 1);
    println!("receiver offline: {} retry queued", unavailable.retried);
    // Advance the injected scheduler clock past backoff. This does not sleep or
    // change the machine clock; production schedulers wait until work is due.
    let retry_at = now_ms() + 180_000;
    let mut context = observation::live_context(&passport, &recipient);
    for policy in &mut context.scarcity_policies {
        policy.window_end_unix_ms = now_ms() + 3_600_000;
        policy.window_id = scarcity_window_id(policy, &batch.treaty_id)?;
        policy.policy_sha256 = scarcity_policy_sha256(policy)?;
    }
    let observations = Arc::new(InMemoryPheromoneSubstrate::new());
    let receiver = Arc::new(Receiver {
        observations: observations.clone(),
        context,
    });
    let service = PheromoneRelayService::new(
        PheromoneRelayConfig {
            local_kernel_id: "did:chio:buyer-kernel".into(),
            profile: RelayProfile::LocalDev,
            now_unix_ms: retry_at,
            freshness_window_ms: 60_000,
            max_body_bytes: 256_000,
            use_system_clock: false,
            operator_token: None,
            report_dir: None,
        },
        receiver_directory,
        receiver,
        inbox,
    );
    let listener = tokio::net::TcpListener::bind(address).await?;
    let server = tokio::spawn(service.serve(listener));
    let delivered = deliver_due_batches(
        &outbox,
        sender_directory,
        sender,
        "did:chio:llamaworks",
        retry_at,
        1,
    )
    .await?;
    server.abort();
    assert_eq!(delivered.delivered, 1, "{delivered:?}");
    let admitted = observations.query_deposits(&DepositQuery {
        subject_class: None,
        treaty_id: None,
    })?;
    assert_eq!(admitted.len(), 1);
    println!(
        "receiver online: {} batch delivered; {} signed observation stored",
        delivered.delivered,
        admitted.len()
    );
    println!("evidence: {run_dir}");
    Ok(())
}

fn setup() -> Result<(), Box<dyn std::error::Error>> {
    use std::fs::{self, OpenOptions};
    use std::io::Write;
    let directory = std::path::Path::new("relay-state");
    fs::create_dir(directory)?;
    let sender = Keypair::generate();
    let receiver = Keypair::generate();
    let passport = Keypair::generate();
    let endpoint = "http://127.0.0.1:18080".to_string();
    let sender_directory = relay_directory::client_directory(&receiver, endpoint.clone());
    let receiver_directory = relay_directory::directory(&sender, endpoint);
    for (name, key) in [
        ("sender", &sender),
        ("receiver", &receiver),
        ("passport", &passport),
    ] {
        let mut options = OpenOptions::new();
        options.write(true).create_new(true);
        #[cfg(unix)]
        {
            use std::os::unix::fs::OpenOptionsExt;
            options.mode(0o600);
        }
        let mut file = options.open(directory.join(format!("{name}-key.json")))?;
        file.write_all(&serde_json::to_vec(&serde_json::json!({"kernelId": if name == "receiver" { "did:chio:buyer-kernel" } else { "did:chio:llamaworks" }, "seedHex":key.seed_hex()}))?)?;
    }
    fs::write(
        directory.join("sender-directory.json"),
        serde_json::to_vec_pretty(&sender_directory)?,
    )?;
    fs::write(
        directory.join("receiver-directory.json"),
        serde_json::to_vec_pretty(&receiver_directory)?,
    )?;
    let body = observation::body(&passport);
    let batch = PheromoneGossipBatch {
        schema: PHEROMONE_GOSSIP_BATCH_SCHEMA.into(),
        recipient_kernel_id: "did:chio:buyer-kernel".into(),
        treaty_id: body.treaty_scope[0].clone(),
        flushed_at_unix_ms: now_ms(),
        frames: vec![PheromoneDepositGossip {
            schema: PHEROMONE_GOSSIP_SCHEMA.into(),
            treaty_id: body.treaty_scope[0].clone(),
            origin_kernel_id: body.kernel_id.clone(),
            gossiping_peer_kernel_id: body.kernel_id.clone(),
            deposit: sign_deposit(body, &passport)?,
            ts_unix_ms: now_ms(),
            transit_chain: None,
        }],
    };
    fs::write(
        directory.join("batch.json"),
        serde_json::to_vec_pretty(&batch)?,
    )?;
    let store = SqlitePheromoneRelayStore::open(directory.join("sender.sqlite"))?;
    store.enqueue_batch(
        "did:chio:llamaworks",
        "did:chio:buyer-kernel",
        &batch.treaty_id,
        &batch,
        now_ms(),
    )?;
    println!("relay-state: fresh keys, two directories, one signed observation queued");
    Ok(())
}

Open src/bin/relay.rs

relay_directory.rsrust
use chio_core_types::Keypair;
use chio_pheromone_relay::*;
use swarm_authority_examples::now_ms;
pub fn directory(sender: &Keypair, endpoint: String) -> PeerDirectoryDocument {
    PeerDirectoryDocument {
        schema: PHEROMONE_PEER_DIRECTORY_SCHEMA.to_string(),
        local_kernel_id: "did:chio:buyer-kernel".to_string(),
        issued_at_unix_ms: now_ms() - 1_000,
        expires_at_unix_ms: now_ms() + 3_600_000,
        peers: vec![PeerDirectoryEntry {
            kernel_id: "did:chio:llamaworks".to_string(),
            public_key: sender.public_key(),
            endpoint,
            treaty_subscriptions: vec!["treaty:buyer-llamaworks:support-ops".to_string()],
            relay_role: RelayRole::Origin,
            allowed_subject_class_namespaces: vec!["dev.chio.support".to_string()],
            accepted_ladder_refs: vec![RelayLadderRef {
                ladder_manifest_id: "ladder:llamaworks:support:v1".to_string(),
                ladder_manifest_sha256: "a".repeat(64),
                expires_at_unix_ms: now_ms() + 3_600_000,
            }],
            max_batch_frames: 8,
            max_catchup_frames: 16,
            max_catchup_bytes: 64_000,
        }],
    }
}

pub fn client_directory(recipient: &Keypair, endpoint: String) -> PeerDirectoryDocument {
    PeerDirectoryDocument {
        schema: PHEROMONE_PEER_DIRECTORY_SCHEMA.to_string(),
        local_kernel_id: "did:chio:llamaworks".to_string(),
        issued_at_unix_ms: now_ms() - 1_000,
        expires_at_unix_ms: now_ms() + 3_600_000,
        peers: vec![PeerDirectoryEntry {
            kernel_id: "did:chio:buyer-kernel".to_string(),
            public_key: recipient.public_key(),
            endpoint,
            treaty_subscriptions: vec!["treaty:buyer-llamaworks:support-ops".to_string()],
            relay_role: RelayRole::Receiver,
            allowed_subject_class_namespaces: vec!["dev.chio.support".to_string()],
            accepted_ladder_refs: vec![RelayLadderRef {
                ladder_manifest_id: "ladder:buyer:support:v1".to_string(),
                ladder_manifest_sha256: "b".repeat(64),
                expires_at_unix_ms: now_ms() + 3_600_000,
            }],
            max_batch_frames: 8,
            max_catchup_frames: 16,
            max_catchup_bytes: 64_000,
        }],
    }
}

Open src/relay_directory.rs

relay_receiver.rsrust
use async_trait::async_trait;
use chio_core_types::crypto::{canonical_json_bytes, sha256_hex};
use chio_federation::pheromone_gossip::*;
use chio_pheromone::*;
use chio_pheromone_relay::*;
use chio_pheromone_runtime::{
    PheromoneBatchOutcome, PheromoneFrameReport, PheromoneReceiveReport,
    PHEROMONE_RECEIVE_REPORT_SCHEMA,
};
use std::sync::Arc;
// A bounded local origin-to-receiver host: one frame per request, no forwarding.
// It authenticates the relay request through PheromoneRelayService, then validates
// the passport signature, receiver policy, observation nonce, and capacity here.
pub struct Receiver {
    pub observations: Arc<InMemoryPheromoneSubstrate>,
    pub context: PheromoneValidationContext,
}
#[async_trait]
impl RelayBatchReceiver for Receiver {
    async fn receive_batch(
        &self,
        batch: PheromoneGossipBatch,
        sender: String,
        now: u64,
    ) -> Result<PheromoneReceiveReport, PheromoneRelayError> {
        if batch.frames.len() != 1 {
            return Err(PheromoneRelayError::SenderMismatch(
                "this host requires exactly one origin frame".into(),
            ));
        }
        let frame = &batch.frames[0];
        if frame.origin_kernel_id != sender
            || frame.gossiping_peer_kernel_id != sender
            || frame.transit_chain.is_some()
        {
            return Err(PheromoneRelayError::SenderMismatch(
                "origin-only receiver".into(),
            ));
        }
        let mut context = self.context.clone();
        context.now_unix_ms = now;
        let result = self.observations.deposit(frame.deposit.clone(), &context);
        let accepted = result.is_ok();
        let (code, detail) = match result {
            Ok(()) => ("accepted".into(), "passport observation admitted".into()),
            Err(error) => (error.code().into(), error.to_string()),
        };
        Ok(PheromoneReceiveReport {
            schema: PHEROMONE_RECEIVE_REPORT_SCHEMA.into(),
            accepted,
            batch_outcome: if accepted {
                PheromoneBatchOutcome::Accepted
            } else {
                PheromoneBatchOutcome::Rejected
            },
            accepted_frame_count: u64::from(accepted),
            rejected_frame_count: u64::from(!accepted),
            batch_sha256: sha256_hex(
                &canonical_json_bytes(&batch)
                    .map_err(|e| PheromoneRelayError::SenderMismatch(e.to_string()))?,
            ),
            recipient_kernel_id: batch.recipient_kernel_id.clone(),
            authenticated_sender_kernel_id: sender,
            received_at_unix_ms: now,
            frames: vec![PheromoneFrameReport {
                frame_index: 0,
                accepted,
                code,
                detail,
                deposit_nonce: Some(frame.deposit.body.nonce.clone()),
            }],
        })
    }
}

Open src/relay_receiver.rs

relay-serve.rsrust
use swarm_authority_examples::observation;
#[path = "../relay_receiver.rs"]
mod relay_receiver;
use chio_core_types::Keypair;
use chio_pheromone::*;
use chio_pheromone_relay::*;
use std::{path::Path, sync::Arc};
use swarm_authority_examples::now_ms;
fn key(root: &Path, name: &str) -> Result<Keypair, Box<dyn std::error::Error>> {
    let json: serde_json::Value =
        serde_json::from_slice(&std::fs::read(root.join(format!("{name}-key.json")))?)?;
    Ok(Keypair::from_seed_hex(
        json["seedHex"].as_str().ok_or("missing seedHex")?,
    )?)
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let root = std::path::PathBuf::from(
        std::env::args()
            .nth(1)
            .unwrap_or_else(|| "relay-state".into()),
    );
    let passport = key(&root, "passport")?;
    let receiver = key(&root, "receiver")?;
    let document: PeerDirectoryDocument =
        serde_json::from_slice(&std::fs::read(root.join("receiver-directory.json"))?)?;
    let directory = PeerDirectory::from_document(document, now_ms())?;
    let mut context = observation::live_context(&passport, &receiver);
    // This receiver owns the window and passport-admission configuration.
    for policy in &mut context.scarcity_policies {
        policy.window_start_unix_ms = now_ms() - 3_600_000;
        policy.window_end_unix_ms = now_ms() + 3_600_000;
        policy.window_id = scarcity_window_id(policy, &policy.treaty_scope[0])?;
        policy.policy_sha256 = scarcity_policy_sha256(policy)?;
    }
    let service = PheromoneRelayService::new(
        PheromoneRelayConfig {
            local_kernel_id: "did:chio:buyer-kernel".into(),
            profile: RelayProfile::LocalDev,
            now_unix_ms: now_ms(),
            freshness_window_ms: 60_000,
            max_body_bytes: 256_000,
            use_system_clock: true,
            operator_token: None,
            report_dir: Some(root.join("reports")),
        },
        directory,
        Arc::new(relay_receiver::Receiver {
            observations: Arc::new(InMemoryPheromoneSubstrate::new()),
            context,
        }),
        Arc::new(SqlitePheromoneRelayStore::open(
            root.join("receiver.sqlite"),
        )?),
    );
    let listener = tokio::net::TcpListener::bind("127.0.0.1:18080").await?;
    println!("receiver listening: http://127.0.0.1:18080");
    service.serve(listener).await?;
    Ok(())
}

Open src/bin/relay-serve.rs

Captured result and refusal

Captured executable outputtext
receiver offline: 1 retry queued
receiver online: 1 batch delivered; 1 signed observation stored
evidence: relay-run-1789230069693

This executable embeds the relay library and a one-frame origin-only receiver over real loopback HTTP. It advances an injected scheduler clock instead of waiting through backoff. Observation storage is in memory; production CLI proof-package admission, crash recovery and directory rotation are separate integration work.

Exit 0 indicates a successful example run. Expected refusals shown in the output are checked by the program; invalid command inputs exit nonzero. Keys, digests, timestamps and run-directory names vary.

Each outbox row is leased, retried twice, then dead-lettered

Enqueue creates a pending row with attempts 0, identified by a hash of sender, recipient, treaty, batch, and queued-at time. Re-enqueuing exactly that tuple is idempotent; a new queue time makes a new row.

Relay outboxChoose an outcome.
outbox row0 s60 s120 s180 sdeliveredattempts 0due0 s60 s120 s180 sdelivered

An accepted report finishes the row

Enqueue stores a pending row with attempts 0. A scheduled tick leases it for 30 seconds, rechecks the recipient’s scope, and sends a signed HTTP request; the receiving operator admits the batch under its own runtime admission and returns a receive report. An accepted report marks the row delivered. Relay Operations.

One outbox row on an illustrative time axis with the relay’s 30-second lease and its 60- and 120-second retry schedule; the times are due times, not delivery latency. The receiving operator’s admission and the relay’s metrics are left out.

A lease call first moves expired leased rows back to retry, then selects due pending and retry rows by next-attempt time and outbox id and leases them for 30 seconds. Ordinary failures retry at +60 and then +120 seconds from the tick’s time, and the third stores dead_letter. A sender_mismatch retries at +60 seconds every time and never reaches that threshold, though its attempts still count toward it. An expired lease returns as stale_lease_recovered with attempts and next-attempt time unchanged.

Run one scheduler per store. The store serializes calls through one shared connection lock, while calls through separately opened connections can select the same rows. A lease is not renewed during delivery, and the HTTP client’s 60-second timeout can outlast the 30-second lease. Restarting the listener recovers nothing; the next lease call does.

Read more in Retry decisions.

A request nonce stops replays at the door, and admission records each frame

The signed HTTP envelope binds schema, sender, recipient, method, path, payload hash, nonce, and sent-at time. The listener checks the sender’s key from its loaded directory, the canonical payload hash, and clock skew, then records the sender and nonce pair. Directory admission checks role, frame cap, treaty subscription, and transit-ladder pins: the inbound peer is an origin or hub, and the outbound recipient a receiver or hub.

A reused sender and nonce pair is refused as relay_nonce_replay before the receiver runs, and a signed time outside the freshness window as relay_request_stale. The sender records either as an ordinary failure. Admission and delivery bookkeeping are separate durable writes, so a lost response can leave admitted deposits while the sender records a failure.

Ordinary retries use relay-tick:{outbox id}:{attempts + 1} as the request nonce. A new nonce re-enters the runtime deposit replay check, which can report replay_window_exceeded for a deposit already admitted; reusing the old nonce fails at the envelope. After a timeout, compare the original request with the receiver’s records before classifying a replay refusal.

Read more in Receive and recovery boundaries.

A peer resumes from a cursor it holds

Catch-up returns outbox rows for the authenticated requester and treaty after a decimal rowid cursor. The requester keeps nextCursor and sends it back as afterCursor; the relay keeps no per-requester progress. The response field frames holds gossip batches, and their inner frames count against the requested limit, so two two-frame batches under a limit of 3 arrive one per page.

A first batch that exceeds a bound refuses the request; resolve the bound or the underlying work before advancing application progress.

Read more in Catch-up bounds.

Readiness reports local state, and observability names what to inspect

The ready route returns HTTP 200 when the store is reachable, fewer than 10,000 rows are pending, retrying, or leased, and no leased row has expired; it returns 503 otherwise. These predicates describe local state, separate from peer reachability and completed deliveries.

The observability route adds directory information, queue age, recent failures, and recommendations, and any recommendation marks that report degraded, including a single retry. A ready relay can therefore report degraded observability. The core relay emits five metric families, and peer identities, treaty ids, and nonces stay in diagnostic records instead of metric labels.

Read more in Observability recommendations and Metric families.

Each record proves one step of a delivery

Start with the queue and recent failures, then the matching event and the receiver’s report. Each record establishes one thing:

  • Enqueue report. Local work is queued under the validated inputs; remote admission comes later.
  • Delivered outbox row. The tick recorded an accepted receive report; the receiver’s evidence interprets the observations.
  • Tick exit 0. The command completed its report-writing path; the report’s counts and failures say what was delivered.
  • Ready response. The local health predicates passed.
  • Supervisor lint. A submitted JSON declaration of the deployment passed its checks, including its singleWriter flag and proxy settings. The running processes and proxy files are inspected separately.

Preserve the original request when delivery is uncertain. Requeueing an observation the receiver already admitted meets replay protection, and the receiver’s first effect stands. A missing event is weak evidence: the rejection-reporting path discards an event emission error, while normal batch and catch-up responses require event recording to succeed.

Read more in Failure codes.

Incident evidence outlives the live listener

Reporting tools carry an incident from observability into signed archives reviewed without the listener:

  1. Alert evaluation. Each recommendation maps through an operator-owned rule and route, with required event evidence and a suppression policy. accepted means every produced alert is suppressed, or there are none.
  2. Handoff dry run. Route coverage, target and runbook references, and severity and escalation are checked without sending notifications.
  3. Delivery and review. Supplied downstream delivery, acknowledgement, drift, and review records are bound into one chain.
  4. Assurance package. Source-report hashes are bound and outstanding action codes listed; acceptance requires the six upstream reports to pass and the delivery and acknowledgement attention counts to be zero.
  5. Signed export and archive. Manifests verify against caller-supplied trusted exporter or packager keys and their validity constraints; an embedded signing key is never its own trust root.

Human receipt of a notification is established by the downstream system. The relay runbook covers the export, archive, and recovery-drill sequence.

Relay routes, limits, and codes

HTTP routes

RouteCredentialResult
POST /v1/chio/pheromone/batches
Signed peer envelope, directory scope
Receive report; a 2xx can carry accepted=false or a partial batch
POST /v1/chio/pheromone/catchup
Signed envelope, requester role, treaty subscription, bounds
Batches and nextCursor
GET /v1/chio/pheromone/health
None
Store-derived health report; HTTP 200 even when accepted is false
GET /v1/chio/pheromone/ready
None
HTTP 200 when checks pass, 503 when degraded
GET /v1/chio/pheromone/observability
Bearer operator token
Directory, queue, recent failures, recommendations
GET /v1/chio/pheromone/metrics
Bearer operator token
Prometheus text, plus any transport hook’s families

Production serve requires a nonempty operator token supplied through --operator-token-env. A library caller can configure no token, which leaves the observability and metrics routes open. Relay handler errors use HTTP 400, and operator authentication failure uses 401; router parsing, body limits, timeouts, and overload can produce other statuses before a relay report exists.

Retry decisions

ConditionStored resultNext attempt
Accepted receive report
delivered
None; the row leaves the due queue
Ordinary failure, attempts 0
retry; attempts 1
Tick time +60 s
Ordinary failure, attempts 1
retry; attempts 2
Tick time +120 s
Ordinary failure, attempts 2 or more
dead_letter
None; correct the cause, then requeue
sender_mismatch
retry; attempts increase
Tick time +60 s, every time
Expired lease
retry with stale_lease_recovered
Unchanged, as are attempts

Sender-mismatch retries also increase attempts, so a later ordinary failure reaches the threshold sooner. Starting at attempts 0 at time 0, ordinary failures at 0 and 60 seconds make the third attempt due at 180 seconds, and a third failure dead-letters the row. These are due times, and actual timing follows your scheduler.

crates/trust/chio-pheromone-relay/src/service.rs844-866rust
pub(crate) fn mark_delivery_failure(
    store: &(impl PheromoneRelayStore + ?Sized),
    entry: &RelayOutboxBatch,
    code: &str,
    now_unix_ms: u64,
    report: &mut RelayTickReport,
) -> Result<(), PheromoneRelayError> {
    report.accepted = false;
    report.failures.push(format!("{}: {code}", entry.outbox_id));
    if entry.attempts.saturating_add(1) >= 3 {
        store.mark_dead_letter(&entry.outbox_id, code)?;
        report.dead_lettered = report.dead_lettered.saturating_add(1);
    } else {
        let backoff_ms = 60_000u64.saturating_mul(entry.attempts.saturating_add(1));
        store.mark_retry(
            &entry.outbox_id,
            code,
            now_unix_ms.saturating_add(backoff_ms),
        )?;
        report.retried = report.retried.saturating_add(1);
    }
    Ok(())
}

Receive and recovery boundaries

  • HTTP request nonce. A reused sender and nonce pair is refused before the receiver runs. relay_nonce_replay is request replay protection and returns no prior success report.
  • HTTP inbox. The handler calls the receiver, records the inbox verdict, then emits its event, with no inbox slot reserved before receiving.
  • Runtime admission. Each frame gets its own verdict, so valid frames can commit while siblings are rejected. Read the frame results and batch_outcome; a rejected batch-level flag can coexist with stored frames.
  • Iroh lane. A deterministic batch nonce, an inbox lookup, and a reservation precede receive, and recovery can adopt a runtime verdict bound to both the batch hash and the authenticated sender. These recovery mechanics belong to that transport.
  • Directory reload. The HTTP listener holds the peer directory it loaded at startup, and each tick loads its directory again. The iroh transport has its own reload facility.
  • Subject classes. The directory type declares allowed_subject_class_namespaces; the receiver’s runtime policy admits subject classes.

Catch-up bounds

Input or conditionBehavior
afterCursor
Empty or whitespace starts at zero; a malformed unsigned integer is refused; a page with no batch leaves the cursor unchanged.
limit
Positive and at most the peer’s maxCatchupFrames; both the SQL row limit and the inner-frame budget.
maxCatchupBytes
Bounds the sum of stored batch JSON lengths, separate from the full HTTP response size.
First batch exceeds a bound
The request is refused. A later batch that would exceed a bound ends the page after the batches already included.
Final outbox states
Delivered and dead-lettered rows remain eligible; the query has no status filter.
Current peer scope
Requester identity, role, treaty, and limits are checked; returned transit hops must match the loaded directory’s pins.

Observability recommendations

CodeInspect next
directory_unknown
The directory summary has no expiry; check which directory and state inputs produced the report.
directory_expiring
The summarized expiry is within five minutes or past; prepare valid successor state and load it.
dead_letters_present
The final failure code and the recipient’s report, before requeueing.
stale_leases_present
Scheduler ownership, then a controlled tick to recover expired leases.
retries_pending
Next-attempt times against scheduler activity, endpoint reachability, and sender identity.

Metric families

FamilyBacking value
chio_pheromone_relay_queue_depth
Current outbox count per status: pending, retry, leased, delivered, dead_letter.
chio_pheromone_relay_oldest_pending_age_seconds
Age of the oldest pending, retry, or leased row; zero when none exists.
chio_pheromone_relay_stale_leases
Leased rows whose expiry is at or before the evaluation time.
chio_pheromone_relay_dead_letters_total
Current dead-letter row count from stored state, labelled reason=observed.
chio_pheromone_relay_rejections_total
The top 32 merged failure-code counts from attempts, dead letters, and rejected events.

The failure list is bounded per scrape, so an absent series is weak evidence that a code never occurred. A rejected 2xx receive report contributes the remote peer’s first rejected-frame code, so distinct reason values accumulate over time. The observability route requests 25 failure summaries, and the offline observe command accepts --limit.

crates/trust/chio-pheromone-relay/src/store.rs708-715rust
pub fn relay_metrics_snapshot(
    &self,
    local_kernel_id: &str,
    generated_at_unix_ms: u64,
) -> Result<RelayMetricsSnapshot, PheromoneRelayError> {
    let conn = self.conn.lock()?;
    let queue = relay_queue_summary(&conn, generated_at_unix_ms)?;
    let failures = recent_failure_summaries(&conn, 32)?;

Production flags and transport

For a production deployment, pass the active peer-directory state with --peer-directory-state, its trusted issuer document with --trusted-issuers, and --profile production. The signing-key document must match the queued work’s sender identity.

HTTPS is the default transport. Iroh is an optional lane on self-hosted relays with its own transport keys, signed directory, and discovery inputs, and serve can mount its signals lane beside HTTP. Each tick selects one transport, so schedule one drain per set of rows.

Production limits

LimitDefault and scope
Request freshness
60,000 milliseconds of absolute clock skew
HTTP request body
256,000 bytes in the production profile; 1,048,576 bytes in local-dev
Batch frames
Per-peer positive limit, at most 128
Catch-up frames and bytes
Per peer, at most 256 frames and 1,048,576 stored JSON bytes; a batch-frame cap stays within the catch-up frame cap
Server request and shutdown
20-second request timeout, 25-second drain; the systemd serve template allows 35 seconds before forced termination
Concurrency and connections
1,024 in-flight requests and 2,048 accepted TCP connections

Production peer endpoints require HTTPS, and local-dev permits plain HTTP only at a loopback host. Both reject URL credentials, queries, and fragments, and the HTTP client disables redirects. Network egress policy belongs to your deployment’s network controls at the operating boundary.

The deployment templates provide a long-running listener and a separate scheduler: the systemd timer requests a tick every 30 seconds, and the oneshot template leases up to 32 rows. Adapt paths, credentials, process ownership, TLS termination, and proxy limits, and keep request signatures mandatory behind TLS.

Failure codes

Code or symptomInspect next
sender_mismatch
Match the signing key’s kernel id to the outbox sender. This branch retries without ordinary dead-lettering.
transport_error
The peer endpoint, TLS and proxy path, response status, and response loss. A non-2xx response also reports this code.
relay_profile_denied
The loaded directory’s role, treaty subscription, frame cap, and transit-ladder references.
relay_request_stale
Signed sentAtUnixMs against receiver time and its freshness window, including time spent on rows processed before it in the same tick.
relay_nonce_replay
Sender, request nonce, and prior records. A lease recovered without an attempt increment can reuse the prior HTTP nonce.
replay_window_exceeded
Runtime deposit age and nonce state. After a lost response, this can reflect a deposit the receiver already admitted.
body_hash_mismatch, signature_invalid
Canonical payload bytes, the signed hash, and the sender’s directory key; look for changes after signing.
catchup_denied
Requester and responder identity, role, treaty, cursor, first-batch size, and loaded ladder pins.
operator_auth_required
The configured operator token on observability and metrics requests; peer signing keys are a separate credential.

Record retention

The store keeps nonce, attempt, event, and final outbox records, and its reservation cleanup covers reservations only. Plan retention deliberately, and preserve the records that interpret delivery and replay before changing the store.

Next: what the relay carries

Relay Operations · Chio Docs