Relay Operations

One queue, two directions, two halves

The relay moves PheromoneGossipBatch frames between kernels over a signed HTTP envelope, durably queued on both sides. Everything an operator touches reduces to one SQLite file with seven tables and one command family, chio pheromone relay. The wire format the frames carry is not this page’s subject; that is Pheromone Store.

The crate has two halves that share nothing but a store. The live relay (directory, http_signing, client, store, service, metrics) is the transport. The alert assurance chain (alerts, delivery, assurance, archive) is an offline reporting pipeline built on the transport’s own observability output. The first runs inside the serving process; the second runs as a sequence of CLI invocations against files on disk and never opens the listener.

Two profiles bound everything. RelayProfile::Production requires HTTPS peer endpoints and carries the ceilings below; RelayProfile::LocalDev permits HTTP only to a loopback host (localhost, 127.0.0.1, ::1). Every peer endpoint is rejected for embedded credentials, a query string, or a fragment, in both profiles.

pub fn production_defaults() -> Self {
    Self {
        freshness_window_ms: 60_000,
        max_body_bytes: 256_000,
        max_batch_frames: 128,
        max_catchup_frames: 256,
        max_catchup_bytes: 1_048_576,
    }
}

Those are ceilings on the signed peer directory, not the operating values. Each PeerDirectoryEntry carries its own max_batch_frames, max_catchup_frames, and max_catchup_bytes, and validate_peer_directory_profile rejects a directory whose per-peer numbers are zero, exceed the profile ceiling, or set a batch-frame cap above the same peer’s catch-up frame cap.

There is no local-dev counterpart to that function. RelayProfileLimits exposes production_defaults() and nothing else, and the crate never varies max_body_bytes by profile. The 1 MiB body limit a local-dev relay serves under comes from relay_service_limits_for_profile in the CLI, which takes the production defaults and overrides that one field. A library caller that builds PheromoneRelayConfig directly gets whatever body limit it sets, under either profile.

Six routes behind one path prefix

The router registers exactly six routes, all under PHEROMONE_RELAY_PATH_PREFIX, /v1/chio/pheromone. The constants are pinned three ways at once by public_relay_http_paths_are_chio_native: against the literals, against the two-value path enum in spec/schemas/chio-pheromone/v1/relay-http-request.schema.json, and against the deployment fixture’s health, ready, and pinned proxy prefix.

Above the routes sit two layers of bounds. The route-local DefaultBodyLimit::max(max_body_bytes) comes from the profile, 256,000 bytes in production. Then apply_server_hygiene from chio-http-serve adds the defaults every serve site in the workspace shares: a 20 second per-request timeout that denies with 408, a 1024 in-flight concurrency limit that sheds with 503 rather than queueing, a 2048 accepted connection cap, and a 25 second graceful drain. The relay commits synchronously inside its handlers, so the drain has nothing to flush beyond completing in-flight requests.

The outbox: lease, retry, dead letter

Sending is not a request path. A batch is enqueued by chio pheromone relay enqueue into a durable outbox, and a separate periodic invocation, chio pheromone relay tick, drains it. The two run as different units under different lifetimes: the serve unit is Type=simple, the tick unit is Type=oneshot fired by a timer. Nothing in the serving process drains the outbox.

The row identity is content-derived. enqueue_batch hashes sender, recipient, treaty, the batch itself, and the queue timestamp into an outbox_id with RFC 8785 canonical JSON plus SHA-256, then inserts ON CONFLICT(outbox_id) DO NOTHING. Enqueueing the identical batch twice at the identical millisecond is a silent no-op; one millisecond later it is a second row.

The lease is 30 seconds and recovers on read

lease_due_batches does three things in order, and the first is the recovery step. Before selecting anything it re-lists every row whose lease has expired.

UPDATE chio_pheromone_relay_outbox
SET status = 'retry',
    lease_expires_unix_ms = NULL,
    last_error_code = 'stale_lease_recovered'
WHERE status = 'leased' AND lease_expires_unix_ms <= ?1

It then selects status IN ('pending', 'retry') with next_attempt_unix_ms <= now, ordered by due time and then outbox_id, capped at max_batches, and marks each selected row leased with lease_expires_unix_ms = now + 30_000. The 30 second lease is a literal in that function and is not configurable. The shipped timer fires every 30 seconds (OnUnitActiveSec=30s, AccuracySec=5s), so a tick that outlives its own lease is a real possibility on a slow peer, and the next tick will re-list its rows underneath it.

Three attempts, then the dead-letter status

Every failure inside a tick funnels through one function, and it is where both the backoff and the dead-letter threshold live.

pub(crate) fn mark_delivery_failure(
    store: &(impl PheromoneRelayStore + ?Sized),
    entry: &RelayOutboxBatch,
    code: &str,
    now_unix_ms: u64,
    report: &mut RelayTickReport,
) -> Result<(), PheromoneRelayError> {
    report.accepted = false;
    report.failures.push(format!("{}: {code}", entry.outbox_id));
    if entry.attempts.saturating_add(1) >= 3 {
        store.mark_dead_letter(&entry.outbox_id, code)?;
        report.dead_lettered = report.dead_lettered.saturating_add(1);
    } else {
        let backoff_ms = 60_000u64.saturating_mul(entry.attempts.saturating_add(1));
        store.mark_retry(
            &entry.outbox_id,
            code,
            now_unix_ms.saturating_add(backoff_ms),
        )?;
        report.retried = report.retried.saturating_add(1);
    }
    Ok(())
}

Read the arithmetic literally. A row that has never been attempted (attempts = 0) retries in 60 seconds; at attempts = 1 it retries in 120 seconds; at attempts = 2 the predicate is true and it is dead-lettered. Three failed deliveries, two retries, one dead letter. The backoff is linear, not exponential, and it is not jittered.

mark_retry also writes a row into chio_pheromone_relay_attempts carrying the bounded failure code, which is what the failure histogram later reads. mark_dead_letter writes no attempt row; a dead letter is counted through the outbox row’s own last_error_code instead.

What a tick checks before it dials

For each leased row the tick applies enforce_outbound_peer_batch_directory_scope against the current signed directory, before any socket is opened. Four conditions, all fail-closed: the queued batch’s own recipient_kernel_id must equal the outbox recipient, the recipient must hold RelayRole::Receiver or RelayRole::Hub, the frame count must be within that peer’s max_batch_frames, and the peer must be subscribed to the batch treaty. Then every transit hop naming that peer must reference a ladder manifest pinned in the peer’s accepted_ladder_refs by id, SHA-256, and expiry together.

This is a re-check, not a first check, and it exists for directory rotation: a batch enqueued against directory version 4 is delivered under whatever version the tick loads. Two drills prove the refusal against a live queue. relay_tick_rechecks_recipient_scope_before_delivery strips the recipient’s treaty subscriptions and asserts delivered: 0, retried: 1, and a relay_profile_denied failure line. relay_tick_rejects_recipient_without_receiver_role_before_delivery does the same by demoting the recipient to Origin.

The dial itself signs a fresh envelope per attempt with the nonce relay-tick:{outbox_id}:{attempts+1} and POSTs it through a reqwest client built with redirect::Policy::none() and a timeout equal to the freshness window, 60_000 ms as deliver_due_batches constructs it. A non-2xx response is folded into TransportError carrying the status and the body text. A 2xx response whose PheromoneReceiveReport is not accepted is a failure too, and the recorded code is the first rejected frame’s code, falling back to receiver_rejected.

That client dials reqwest directly rather than through the workspace egress contract. The builder and the send() call both carry a CHIO_EGRESS_LINT_ALLOW_DIRECT_REQWEST escape hatch. What bounds an outbound relay dial is the signed directory and the endpoint checks in validation.rs: scheme, no userinfo, no query, no fragment, HTTPS under production, loopback only under local-dev. Do not expect an allowed_authority_set evaluation, a resolved-address denial, or an HttpEgressError on this path.

The tick writes chio.pheromone.relay-tick-report.v1 to --report, and with --report-dir also emits a bounded outbound_delivery event report whose code is the first failure’s code. Six fields: accepted, delivered, retried, deadLettered, duplicateIdempotent, and failures, each failure line formatted {outbox_id}: {code}. Do not build a gate on duplicateIdempotent: it is required by the schema and initialized to zero, and no code path in either the HTTP tick or the iroh drain ever increments it.

The inbox, and the two dedup mechanisms that are not the same

Inbound is the mirror image, with the same directory scope applied by enforce_peer_batch_directory_scope and the roles inverted: a submitting peer must be Origin or Hub. Ahead of that, PheromoneRelayHttpRequest::verify_payload checks seven things and rejects on any one: the envelope schema, the recipient against this node’s local_kernel_id, the method, the path, the peer’s Ed25519 signature over an eight-field signing body, the canonical SHA-256 of the payload against the signed bodySha256, and clock skew against the freshness window. Only then is the nonce recorded.

The signing body deliberately excludes the payload and carries its hash instead, so a tampered body fails as body_hash_mismatch rather than as a signature failure. signed_relay_request_verifies_payload_hash_sender_and_replay_nonce pins both codes and the replay code in one test.

Nonces live in chio_pheromone_relay_nonces keyed by (sender_kernel_id, nonce). The insert is ON CONFLICT DO NOTHING and a zero-row insert raises relay_nonce_replay. The table carries an expires_at_unix_ms column that is written on every insert and read by nothing: no code in the workspace deletes from that table. Size the volume accordingly, or expect the nonce table to grow for the life of the store.

The reservation protocol itself is worth reading even where it is not wired, because it is what the store is built around. reserve_inbox_slot inserts into chio_pheromone_relay_inbox_reservations and returns won: true to exactly one caller; losers must read the winner’s durable verdict rather than re-receive. mark_inbox_reservation_committed marks the instant the receive has committed its deposits but before the verdict is recorded, and the store’s clear-at-open step reclaims only committed = 0 rows. A crash inside that window therefore leaves a surviving reservation, and a redelivery fails closed rather than re-entering the replay window. Five unit tests in store.rs pin it, including an eight-thread barrier test asserting exactly one winner and a restart test asserting that a committed reservation survives a reopen while a pre-commit one does not.

Catch-up is a caller-held cursor over the outbox

A peer that missed frames pulls them with POST /v1/chio/pheromone/catchup carrying a chio.pheromone.catchup-request.v1 body: requester, responder, treaty, afterCursor, and limit. The response returns the frames and a nextCursor the requester is expected to hold and send back next time.

The cursor is a SQLite rowid rendered as a decimal string. An empty or whitespace cursor parses to 0; anything that is not a u64 is catchup_denied with "catch-up cursor is invalid". The query pages forward from it:

SELECT rowid, batch_json
FROM chio_pheromone_relay_outbox
WHERE recipient_kernel_id = ?1 AND treaty_id = ?2 AND rowid > ?3
ORDER BY rowid
LIMIT ?4

Two bounds are then applied per row, and the first-row case differs from the rest. Bytes are measured as the stored JSON length against the peer’s max_catchup_bytes; frames are counted as batch.frames.len() summed across included batches against the request’s limit. Exceeding either bound after at least one batch is a clean page break. Exceeding either bound on the very first batch is a refusal: catch-up byte limit exceeded before first frame or catch-up frame limit exceeded before first batch. A single batch larger than a peer’s bound is unservable by catch-up at any cursor, permanently, until the bound is raised in a new signed directory.

The request’s limit is bound twice: once as the SQL row limit in the statement above, and again as the frame budget applied while iterating those rows. A call with limit 3 never reads more than three outbox rows, so a page of single-frame batches returns three frames and a page of three-frame batches returns one. Size a page by the number of batches you are willing to read, not by the number of frames you want back, or you will under-fetch on every call.

relay_store_catchup_limit_counts_frames_not_batches is the drill: two two-frame batches under a limit of 3 return one batch and a non-zero cursor, and the second call from that cursor returns the other. relay_store_catchup_denies_first_batch_above_frame_limit pins the refusal. When nothing is included, next_cursor stays at the input value, so a caller that keeps polling an exhausted stream sends the same cursor forever and gets an empty page each time.

The route gate is narrower than the store call. validate_catchup_request requires the request schema, requires requesterKernelId to equal the authenticated sender, requires responderKernelId to equal this relay’s local kernel id, requires the peer to hold Receiver or Hub (the inverse of the batch-submission role), requires a limit in 1..=max_catchup_frames, and requires the treaty subscription. After the store returns, enforce_catchup_response_directory_scope re-checks every returned frame’s transit hops against the same ladder pins, so a frame that was legal when queued but whose ladder reference has since rotated out of the directory is denied on the way out.

What the relay renders

relay_metrics_snapshot reads the store and emits a chio.pheromone.relay-metrics-snapshot.v1 list of samples. RelayMetricsSnapshot::render turns it into either pretty JSON or Prometheus text; the route always renders Prometheus, and chio pheromone relay metrics --format selects between them offline against the same file. Five families, and that is the whole set.

The failure histogram is a union of three sources, computed by recent_failure_summaries: rows in chio_pheromone_relay_attempts grouped by code, dead-lettered outbox rows grouped by last_error_code, and non-accepted rows in chio_pheromone_relay_events grouped by code. Counts for the same code from different sources are summed. The result sorts by count descending, then code ascending, then truncates to the caller’s limit: 32 for /metrics, 25 for /observability, and --limit for relay observe. A rare code can fall off the end of that list; it does not fall out of the store.

pub fn relay_metrics_snapshot(
    &self,
    local_kernel_id: &str,
    generated_at_unix_ms: u64,
) -> Result<RelayMetricsSnapshot, PheromoneRelayError> {
    let conn = self.conn.lock()?;
    let queue = relay_queue_summary(&conn, generated_at_unix_ms)?;
    let failures = recent_failure_summaries(&conn, 32)?;

Label cardinality is bounded per scrape, not by the type system. Most label values are fixed: the five status strings, the constant observed, error codes drawn from PheromoneRelayError::code(), which is a fixed match arm per variant, and the two literals the tick writes itself, sender_mismatch and receiver_rejected. One source is not fixed. When a peer answers 2xx with a report that is not accepted, deliver_due_batches records that peer’s first rejected frame code verbatim, a String deserialized from the remote response body, and mark_retry writes it into the attempts table where recent_failure_summaries later groups it. The limit the scrape renderer passes is a literal in the call, not a config value, so any single scrape is bounded while a peer that returns novel frame codes still creates new chio_pheromone_relay_rejections_total series over time. Treat the reason label as bounded by the receiving peer’s code vocabulary, not by this crate. relay_observability_report_summarizes_directory_store_and_bounded_failures asserts the rendered text contains status="retry" and does not contain the peer kernel id that produced the failure. The deployment note states the rule the other way round: no peer ids, treaty ids, hashes, nonces, cursors, or outbox ids as labels.

/metrics renders one # HELP and one # TYPE line per family name, deduplicated through a BTreeSet, then one sample line each with backslash, quote, and newline escaped inside label values. Integral values print with no decimal places and fractional values with six. An optional ExtraMetricsHook appends a second block of Prometheus text to the same body; it exists so the iroh federation transport can publish its families on the relay’s scrape endpoint without a dependency cycle, and relay_metrics_endpoint_appends_extra_metrics_hook_only_when_set asserts the body is unchanged when no hook is attached.

Health, readiness, and the recommendation codes

health_report composes seven counters and three checks, plus the peer-directory version its caller hands it. store.connected is hard-coded true, which is honest only in the sense that a failed store open would have returned an error before reaching it. outbox.pressure is queue_depth < 10_000, a literal. leases.fresh is stale_lease_count == 0. The report is accepted only when all three pass, and /ready converts a non-accepted report into 503 while /health returns the same body with 200 either way. Point a load balancer at /ready and a liveness probe at /health.

/observability adds the signed directory summary and a list of bounded recommendation codes. Five exist: directory_unknown and directory_expiring (warning, the second firing when the active bundle expires within 300,000 ms), dead_letters_present and stale_leases_present (warning), and retries_pending (info). The report is accepted only when the list is empty, so a single row in retry makes an otherwise healthy relay report degraded. Those five codes are also the entire input alphabet of the alert stage below.

Every rejection the service produces also emits an event. relay_http_status_error calls emit_event_report with request_rejected and the stable code, which writes a row into chio_pheromone_relay_events and, when --report-dir is set, a file named {generated_at}-{kind}-{hash12}.json. The emit is best-effort. The result is discarded with let _ =, so a failed insert or a failed report-directory write drops the event and the rejection response is returned anyway. Read the status code as the reliable signal and the event row as corroboration. Every handler rejection is HTTP 400 except the operator-token failure, which is 401.

The alert assurance chain

The second half of the crate turns those recommendation codes into routed alerts, then into evidence that the alerts were handed off, delivered, acknowledged, reviewed, packaged, signed, archived, and drilled. Every stage is a separate CLI invocation reading the previous stage’s file, and every stage re-derives the canonical SHA-256 of its inputs before trusting them. No stage opens a socket.

observability + events -> RelayAlertReport -> RelayTrendReport
  -> RelayAlertHandoffReport
  -> RelayAlertNormalizationReport -> RelayAlertDeliveryReport
  -> RelayAlertAcknowledgementReport
  -> (drift report, route-review packet)
  -> RelayAlertAssurancePackage -> RelayAlertAssuranceExportBundle (signed)
  -> RelayAlertAssuranceArchiveReport / closeout report
  -> RelayAlertAssuranceArchivePackage (signed)
  -> retention / restore-drill / physical-drill / retention-handoff reports

evaluate_relay_alerts is the entry. It walks the observability report’s recommendations, and for each one it demands a rule in the operator-owned routing profile and a route that rule names. A recommendation with no rule is alert_routing_invalid; a rule marked require_event_evidence with no matching bounded event is alert_source_invalid. Each alert carries a dedupe key built as chio-relay:{local_kernel_id}:{alert_code}:{route_id}, and a rule may be marked unsuppressible, in which case suppression state cannot silence it. The report is accepted only when every alert is suppressed, which is the inverse of the usual reading: an accepted alert report means nothing is firing.

evaluate_relay_alert_handoff is a dry run and says so in its own check detail: "all firing alerts are routeable without sending notifications". It proves every route in the profile has a downstream receiver whose target and runbook reference match, that every firing alert’s severity is at or above its receiver’s floor and at or below its escalation mapping, and that the label set is bounded. It never opens a connection to Alertmanager, PagerDuty, OpsGenie, Slack, or a webhook. Its returned accepted field is the literal true: every condition it checks is an error rather than a soft failure, so a handoff report that exists is a handoff report that passed.

Delivery evidence flows the other way. Local Alertmanager-style or SIEM-style drops are converted by relay alert normalize, which rejects inline secrets, URLs, unknown receivers, ambiguous mappings, unbounded labels, and missing source hashes; then delivery import binds them to the handoff report by hash, and delivery acknowledge and delivery drift-window compare directories of reports so a later delivery result cannot mask an earlier missing handoff. The assurance package is accepted only when six upstream reports are accepted and both the delivery-attention and acknowledgement-pending counts are zero; otherwise it emits bounded operator action codes such as delivery_attention_required or delivery_drift_detected.

The whole family is schema-gated in CI. ci-gates/pheromone.toml is the single manifest for fifteen facets, and only two of them, runtime and transit, are not the relay’s. Twelve are the relay’s own: the bare relay facet, plus eleven carrying the relay- prefix: relay-ops, relay-observability, relay-alert-routing, relay-alert-handoff, relay-alert-delivery, and relay-alert-assurance with its export, external-retention, archive, archive-package, and archive-hardening descendants. The thirteenth, directory-lifecycle, gates the same relay fixture directory. Each facet pins schema ids to files, validates fixtures against them, runs a named cargo test filter, and carries a negative fixture corpus.

Units, timers, and the proxy in front

docs/release/chio-pheromone-relay/ ships three systemd units, two launchd plists, and one nginx server block. They are templates, not a package: the README says so, and the properties they preserve are the point rather than the paths.

The serve unit binds 127.0.0.1:18080 and the proxy holds 443. That split is the deployment boundary the runbook states: terminate TLS at a proxy owned by the same operator boundary as the relay, pin the upstream path to /v1/chio/pheromone, disable redirects, and keep the relay request signatures mandatory even with TLS in front. The signature is not a substitute for TLS and TLS is not a substitute for the signature; the envelope check runs regardless of what terminated the connection.

chio pheromone relay supervisor lint checks a JSON description of that deployment rather than the files themselves. Eight checks, all of which must pass or the report code is supervisor_profile_invalid: the profile schema, the health path, the ready path, a singleWriter: true declaration, the proxy’s pinned prefix, redirects disabled, a proxy body limit at or below the production ceiling, and a scheme that is HTTPS under the production profile.

Four incidents, in reading order

The runbook’s rule is to read canonical reports before raw SQLite, in a fixed order: observability report, alert report, trend report, then the bounded event files under --report-dir, then the store. Four shapes cover most of what goes wrong.

1. Stuck outbox. Confirm whether retries_pending or dead_letters_present is firing, read the trend report for direction, then run one tick with the correct signing key. If staleLeaseCount is non-zero, restart the relay and run one tick: expired leases are recovered into retry on the way into lease_due_batches. If attempts climb with transport failures, the peer endpoint in the active directory or the proxy route is wrong.
2. Dead-letter triage. Group by bounded failure code in the observability report. endpoint_denied means the active directory fails the profile, so lint it. sender_mismatch means the signing key’s kernel id does not match the outbox sender, and per the caveat above it will show as unbounded retries rather than as a dead letter. A receiver rejection means re-running the runtime gate with the same proof package, trust bundle, and context. Requeue only after the cause is fixed, with a fresh relay nonce.
3. Replay storm. The shipped Prometheus rule is increase(chio_pheromone_relay_rejections_total{reason="relay_nonce_replay"}[5m]) > 10 at severity p1. Separate exact idempotent redelivery from a genuine nonce conflict before blocking anything, preserve the signed request and the event report as evidence, and rotate the peer directory only if a key is suspected compromised.
4. Catch-up overload. Check the requested frame and byte limits against the peer’s directory entry first, then the treaty subscription and requester identity. The fix for sustained pressure is a lower per-peer bound in the next signed bundle, and the standing instruction is not to advance cursors for poison frames or unauthorized attempts. Cursors are the requester’s state, so that instruction lands on the requester.

Four alert rules ship in deploy/prometheus/chio-pheromone-relay-observability-rules.yml, each annotated back to the runbook: dead letters above zero for 5m, retry depth above 100 for 10m, stale leases above zero for 5m, and the replay-storm rule above. The first three are p2 and the last is p1.

Guarantees and limits

Next Steps

• Pheromone Store · the signed deposit and batch formats this process moves, and the decay and diversity rules the receiving runtime applies
• Swarm Denial Codes · the wider code vocabulary the relay’s bounded reasons sit inside, and what a caller may branch on
• Federation · the treaty and transit-ladder objects the directory pins each peer against
• Node Observability · what a node exports on its own metrics route, which shares no family with the five above
• Health & Readiness · the same liveness and readiness split asked of a process that does host a kernel