Run a Cognition Market

Replay host

The replay sandbox runs seller-controlled code with no network and bounded storage, output, memory, CPU, and descendant processes. A version command that succeeds or cgroup files that exist prove little: namespace permissions and a usable delegation or user scope must both work.

The probes and the cgroup preparation are in verified_fix_sandbox.rs (lines 1005 to 1059).

pub(super) fn require_sandbox() -> Result<(), CliError> {
    for (command, message) in [
        (
            "bwrap",
            "verified-fix packaging requires bubblewrap for network isolation",
        ),
        (
            "prlimit",
            "verified-fix packaging requires prlimit for resource isolation",
        ),
    ] {
        let output = Command::new(command)
            .arg("--version")
            .output()
            .map_err(|_| CliError::cli_other_error(message.to_owned()))?;
        if !output.status.success() {
            return Err(CliError::cli_other_error(message.to_owned()));
        }
    }
    let cgroup = SandboxCgroup::prepare(TestSandboxLimits::production())?;

Both probes are hard requirements, and the cgroup preparation after them is a third. On the UserScope branch the function runs systemd-run --user --scope with MemoryMax, MemorySwapMax=0, and TasksMax, so a host with neither a delegated cgroup nor a systemd user manager fails closed with verified-fix packaging requires a delegated cgroup v2 or systemd user scope (crates/products/chio-cli/src/cli/dispatch/finding/verified_fix_sandbox.rs:1026-1046). The sandbox mounts toolchain executables but not the operator account's Cargo registry or Git dependency caches, so a Rust repository with third-party crates commits a vendored source tree and its offline Cargo configuration.

Profiles and credentials

Initialization writes the operator profile, the buyer and seller client profiles, the listen address, and the repository root that authenticated submissions are bound to; operator ingress canonicalizes every submitted path against that absolute root. operator-profile.json, buyer-client.json, and seller-client.json each hold a bearer token and a signing seed, at mode 0600. Hand out only the two client files, never the profile, and publish none of them. Keep the deployment directory's stores, role profiles, and working directories together for operation and backup.

Initialization is resumable: the same command completes or checks the same deployment without rotating its identity. A retry with a different listen, buyer, seller, or payout value fails closed, so a buyer that pinned the venue's identity finds the same identity on every return.

Service unit

Serving is one long-lived process reading one profile. It composes the purchase service, the seller admission routes, the venue index, and the status reads behind the trust control plane. The shipped unit keeps operator state writable in one place and seller repositories read-only:

[Service]
Type=simple
User=chio
Group=chio
UMask=0077
ExecStart=/usr/local/bin/chio finding operator serve --profile /var/lib/chio/cognition-market/operator-profile.json
Restart=on-failure
RestartSec=2
Delegate=memory pids
MemoryMax=8G
MemorySwapMax=0
TasksMax=768
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadOnlyPaths=/srv/chio/cognition-market-repositories
ReadWritePaths=/var/lib/chio/cognition-market

Delegate=memory pids gives packaging its cgroup branch; ReadOnlyPaths keeps a seller-controlled Git read away from operator state. Prepare both roots before starting, and stage each seller repository under the read-only root in place of pointing --repository-root at a working checkout.

One writer per profile

Run one serving owner per single-operator profile. The authority store's serving lease refuses a second owner, and the first keeps the lease and keeps answering. The hosted profile has a different concurrency model.

$ chio finding operator serve --profile ./market/operator-profile.json

error[CHIO-CLI-STORE-LOCKED]: the authority store ~/chio/market/authority.db is held by a running operator
  help: stop the running chio finding operator serve, or give this operator its own profile and store
  code: urn:chio:error:cli:store-locked
  docs: https://chio.computer/docs/reference/errors#error-cli-store-locked

Health checks

The reconcile pass is idempotent, so a service timer or cron entry can run it on a cadence. On a new venue every counter is zero; on an active one, read failed_admission_jobs and any missing terminal or capture against the specific operation. The second check goes over HTTP: the venue index needs a topic prefix and answers with a header row and a count even when empty, and a count of zero with an empty cursor is healthy. The Cognition Market follows one sale through these counters.

Status epoch and cadence

A deployment that requires current status proofs publishes and retains the feed its buyers trust, and a buyer pinning that feed refuses a finding whose status it cannot establish under the configured freshness and rollback rules. The epoch publisher runs on the trusted operator plane, from an operator cron, timer, or scheduler holding a single-writer lease for the feed. No public HTTP route advances an epoch; the HTTP routes serve only the exact durable current epoch and its proofs.

The feed binds fixed protocol values, and a different nonce or backend version is a different protocol that must reject:

• Status domain. chio.finding.status.v1
• Key-domain nonce. 3318287169837494
• Sparse-map depth. 256
• Epoch artifact. chio.finding.status-epoch.v1
• Portable proof input. chio.finding.status-proof-input.v1

Each cadence run reads the durable feed floor and the finalized anchor set, collects the bounded candidate batches, inserts every eligible key in one transactional sparse-map update, advances the map epoch exactly once, signs the complete epoch, and stores its exact canonical bytes before making it current. After the floor advances it re-queries the refresh candidates, because the new epoch can displace proofs current when the run started. Failed items stay retryable; a conflicting identity or an ambiguous finality is quarantined, never rewritten.

Two read routes preserve the exact canonical epoch and proof bytes in bounded base64 fields, so a caller verifies them locally instead of rebuilding signed bytes from copied JSON. Both fail closed on the feed identity:

pub(crate) const FINDING_STATUS_ROOT_PATH: &str = "/v1/findings/status/{feed}/root";
pub(crate) const FINDING_STATUS_PROOF_PATH: &str = "/v1/findings/status/{feed}/proof/{finding_id}";

A venue without a governance-pinned feed answers the root route with a refusal instead of an empty root:

$ curl -s http://127.0.0.1:7143/v1/findings/status/local-status/root

{"error":"urn:chio:error:economy:status-feed-mismatch [economy:error]: finding-market status feed does not match the configured feed (request the status feed named in the operator profile)"}

Operators and buyers verify the same bytes through the CLI. The command requires the governance-pinned feed, the operator authorization, the current service bond, a durable rollback floor, and a freshness limit, and advances that floor atomically on every verified observation.

chio finding status \
    --id <finding-id> \
    --feed <governance-pinned-feed-id> \
    --operator-authorization <governance-pinned-authorization.json> \
    --service-bond <governance-pinned-current-service-bond.json> \
    --rollback-floor <durable-status-floor.json> \
    --max-epoch-age-secs <deployment-freshness-limit>

Reuse one rollback-floor path for every query against that feed and operator identity, and back up the sibling <rollback-floor>.retractions/ directory with it. The floor makes a lower epoch detectable; the tombstones keep a retraction sticky.

Single-operator ceilings

These bounds shape capacity and recovery. A busy lane refuses or waits, and a full retained store refuses new work while keeping qualifying exact retries replayable. Respond to the resource a bound protects before you raise a constant.

Sources: the read connection in chio-store-sqlite/src/serving_owner.rs (lines 788 to 792) and its lock in finding_market_status_read.rs (54 to 59); the serving-owner lease, serving_owner.rs:46-59; the cross-process admission-job lock, verified_fix_admission_lock.rs:7-8; VERIFIED_FIX_MAXIMUM_SALE_EXPOSURE_UNITS, crates/platform/chio-control-plane/src/trust_control/finding_verified_fix.rs:80; MAX_RETAINED_SELLER_JOBS, crates/products/chio-cli/src/cli/dispatch/finding/operator.rs:63.

Discovery reads use a separate read-only WAL connection: they skip the authority writer's lock but serialize with other reads on their own mutex, so a discovery snapshot can trail an in-flight write while purchase admission rechecks current authority. Live seller admission and operator tick share the admission-job lock, so one seller submission or retraction runs at a time and an overlapping request receives HTTP 503 with the same durable identity to retry. The lock is a file in the deployment directory with its own deadline:

const ADMISSION_JOB_LOCK_FILE: &str = ".finding-admission-jobs.lock";
const ADMISSION_JOB_LOCK_TIMEOUT: Duration = Duration::from_secs(300);

The price cap is in minor units of the profile's currency, which initialization sets to USD. The example seller defaults to 300 units ($3.00); a price of zero or above the cap is rejected with price_units exceeds the verified-fix sale exposure (crates/platform/chio-control-plane/src/trust_control/finding_operator_seller_routes.rs:113-115).

Operator database retention

Each store is bounded by row count and by bytes. At capacity, new work fails closed while exact retries stay replayable, so a full store never drops the record a retry depends on.

const MAX_RETAINED_CHALLENGE_POLICIES: i64 = 100_000;
const MAX_RETAINED_AUDIT_ROUNDS: i64 = 10_000;
const MAX_RETAINED_AUDIT_ROUND_BYTES: usize = 16 * 1024 * 1024;
const MAX_TERMINAL_BYTES: usize = 16 * 1024 * 1024;
const MAX_PROOF_BYTES: usize = 24 * 1024 * 1024;
const MAX_PURCHASE_JOB_BYTES: usize = 2 * 1024 * 1024;
const MAX_RETAINED_BUNDLES: i64 = 10_000;
const MAX_RETAINED_PURCHASE_JOBS: i64 = 10_000;
const MAX_RETAINED_TERMINAL_BYTES: i64 = 256 * 1024 * 1024;
const MAX_RETAINED_SELLER_ARTIFACT_CLAIMS: i64 = 256;
const MAX_SELLER_ARTIFACT_BYTES: i64 = (8 + 4 + 24) * 1024 * 1024 + 256 * 1024;

MAX_RETAINED_PURCHASE_JOBS is the one an operator meets first: at ten thousand durable purchase jobs, new purchases are refused and exact retries still replay. MAX_RETAINED_TERMINAL_BYTES bounds retained purchase-result bodies at 256 MiB with the same behavior.

Seller replay limits

Every test the seller names runs in a private size-capped tmpfs with no network, a cleared environment, bounded output, and hard memory, CPU, process, descriptor, and file-size limits. cgroup v2 enforces memory, swap, and process limits across the descendant tree, with process-local rlimits as defense in depth.

const MAX_COMMAND_OUTPUT_BYTES: usize = 4 * 1024 * 1024;
const TEST_COMMAND_TIMEOUT: Duration = Duration::from_secs(300);
pub(super) const PACKAGE_WORK_TIMEOUT: Duration = Duration::from_secs(300);
const TEST_SANDBOX_ADDRESS_SPACE_BYTES: u64 = 6 * 1024 * 1024 * 1024;
const TEST_SANDBOX_FILE_BYTES: u64 = 1024 * 1024 * 1024;
const TEST_SANDBOX_TMPFS_BYTES: u64 = 4 * 1024 * 1024 * 1024;
const TEST_SANDBOX_PROCESS_LIMIT: u64 = 512;
const TEST_SANDBOX_OPEN_FILE_LIMIT: u64 = 1024;
const TEST_SANDBOX_CPU_SECS: u64 = 300;

All seller-controlled Git reads, staging, baseline and candidate commands, and patch generation share one five-minute aggregate package deadline. Clone and checkout staging has its own five-minute deadline, a 1 GiB aggregate storage ceiling, a 75,000-entry ceiling, and a per-file size limit, and the operator reserves the full staging budget plus publication headroom inside an 8 GiB, 100,000-entry package and report storage ceiling.

Hosted profile ceilings

A deployment that needs more than one concurrent writer runs the hosted PostgreSQL profile. Its ceilings are per tenant and mostly configured:

Sources: chio-control-plane/src/trust_control/finding_hosted_profile.rs:135 (bounded at :747); chio-finding-market-store-postgres/src/auth.rs:17 and :25, tenant.rs:11-31, lib.rs:143-144; chio-finding-market-server/src/main.rs:30-31; chio-finding-hosted-edge/src/server.rs:484.

The monthly spend ceiling is enforced inside the statement that inserts a reservation, by a trigger the runtime role cannot bypass: the runtime holds no write privilege on the accumulator, and a charged reservation's units are immutable once written. At a tenant's ceiling, inspect its reservations and configured limit, and use the hosted tenant configuration and retention controls; a process-wide constant leaves a tenant's spending authority unchanged.

Bubblewrap cannot create its namespaces

loopback: Failed RTM_NEWADDR: Operation not permitted on Ubuntu means AppArmor denies capabilities inside the unprivileged user namespace. Grant an application-scoped permission to a dedicated Bubblewrap executable and keep the global user-namespace restriction enabled; Ubuntu documents the restriction. In a container with cgroup v2 mounted but no usable delegation or systemd user scope, packaging refuses before replay: check scope creation, not filesystem presence. A Linux VM with a working systemd user manager satisfies it.

Initialization does not match the deployment

A re-run with a changed listen address, principal, or payout destination is refused, and the deployment keeps the identity it published. To move a venue, stand up a new deployment directory and migrate buyers to its pinned identity.

$ chio finding operator init --directory ./market --repository-root ./repos \
    --listen 127.0.0.1:7999

error[CHIO-CLI-PROFILE-MISMATCH]: an operator profile already exists in this directory with different settings
  help: run chio finding operator init with the existing profile's settings, or pass a new --directory
  code: urn:chio:error:cli:profile-mismatch
  docs: https://chio.computer/docs/reference/errors#error-cli-profile-mismatch

HTTP 503 from a busy lane

Purchase execution uses one non-queued blocking lane, and public proof reads a separate one-response lane with 64 KiB streaming chunks and a 30-second absolute egress deadline. A busy lane returns HTTP 503 and the caller retries the same durable identity; overlapping seller submissions get the same answer from the admission lock. A 503 is back-pressure, and a sustained one is a sizing signal.

An expired purchase ask

A prepared ask is checked against current operator time before its first reservation; an expired ask reserves nothing, and the buyer prepares a fresh purchase request. An open reservation that expires during a restart is durably expired and returns the same rejection on every retry, so a retry loop converges.

A finding stuck in publication_pending

Inspect the stages in order before touching the store. Confirm the liability is finalizing, not appealed, reversed, or only evaluated; that the seller-impair and enforcement-root intents are the exact identities the signed enforcement artifact bound; that the impairment receipt is final and unambiguous under the pinned chain-finality policy; that the retraction intent is the original durable item; and that the authorization, service bond, signing key, feed floor, and sparse-map store are available. Then retry the same item. Never mint a replacement intent to bypass a conflict, and keep purchases denied through the investigation.

Two roots at one map epoch

A second epoch id or root for an observed map epoch is equivocation. Freeze status publication and status-gated purchases for the feed, keep both exact signed byte sequences with their authorization snapshots and observation times, open the objective service-bond penalty path, and audit every buyer and kernel floor for the conflicting epoch. Resume with a strictly greater map epoch, the same feed id, the fixed nonce, and the complete retracted-key set; never pick a root by local timestamp or reset the epoch to zero.

Restart and restore

On restart the service loads and verifies the durable feed floor, the exact current signed epoch, the sparse leaves, the sticky status rows, and the outbox before it serves proofs or allows purchases. Missing or inconsistent state keeps the feed unavailable and denies status-gated reads and purchases. An established feed that starts without a floor is a fail-closed recovery incident, not an empty feed. Restore only from a backup holding at least the last externally observed map epoch; a lower floor is unsafe to serve even inside its epoch's validity window.

Packaging refuses on a workstation

Three host conditions fail seller packaging for reasons the seller cannot act on. A version-manager shim on python3 (mise, pyenv, and asdf install one) resolves to the manager instead of an interpreter, and admission returns failed to inspect Python standard library. A runtime tree past 20,000 visited entries aborts with sandbox runtime tree exceeded its entry bound, which an interpreter under a version manager can reach. The sandbox's optional tool list omits awk, so a test command using it fails with the seller's own exit code and reads as a broken fix. Point the operator at a system interpreter and keep the test inside the admitted tool set.

What to alert on

For a single-operator venue: an epoch publication that misses its cadence; an established feed starting without a floor; a lower-epoch replay or same-epoch equivocation; an authorization, key epoch, rotation, or revocation mismatch; a service bond missing, expired, revoked, or below its allocation; a pending intent past its inclusion SLA; and an outbox retry count or age past the operator threshold. Each keeps the affected operation fail-closed until the exact durable evidence is reconciled.