The Hosted Profile
Run the multi-tenant market venue: validate the profile, bring up edge, store, and workers, and qualify each release with a canary.
On this page
The hosted profile runs a multi-tenant market venue: stateless edge replicas behind a TLS proxy sidecar, one PostgreSQL database that keeps tenants apart with forced row-level security, and jobs in Firecracker guests on dedicated Linux KVM hosts.
It serves the catalog, finding publication, market events, and job infrastructure for many tenants. The single-operator profile runs one venue on SQLite inside the trust control plane, in one process; choose the hosted profile when tenants share infrastructure.
StartEdge, store, and workers each run under their own credential
Tenants share the database, the edge replicas, and the worker hosts. Row-level security and per-tenant ceilings hold them apart.
The edge binds one tenant to each transaction
A caller names its tenant in Chio-Tenant-ID and presents a capability token with a Chio-DPoP proof, or an API key pair. Each query then runs with chio.tenant_id set for that transaction, and forced row-level security returns only that tenant's rows. Hosted tenancy.
Purchases run through the coordinator. Purchase records, purchase results, and purchase terminals are coordinator-produced and have no public ingress on the edge, which carries no bid, ask, or reveal route.
The hosted journal retains each status epoch as a signed sparse-map root without per-finding proofs. Once a tenant has a status epoch, catalog_non_live_finding_ids answers with every finding ID it was asked about: a single read returns not_found and a catalog page comes back with those rows removed. A status-gated purchase runs on the single-operator venue, which holds the feed's sparse map and produces the portable non-inclusion proof; the single-operator walkthrough reproduces one.
1. Validate the profile offline
validate-hosted reads a strict canonical private JSON profile and refuses before it touches a listener, a pool, a rail, or a remote signer.
chio finding operator validate-hosted --profile /etc/chio/hosted.jsonhosted_profile: valid
You should see hosted_profile: valid followed by the deployment ID, endpoint, tenant count, and signer count. With --json, the same facts arrive as a chio.finding.hosted-profile-validation.v1 report. Profile validation lists everything the command opens.
2. Migrate the schema, then start the runtimes
The binaries divide by credential. Bring them up in this order; the container, systemd, and Kubernetes contracts live under deploy/cognition-market/.
chio-finding-market-migrate. Applies schema changes as its own Job under a migration-only role. Runtime pods never receive that credential.chio-finding-market-server. Serves the authenticated loopback API from stateless replicas, each behind a TLS proxy sidecar in its pod.chio-finding-worker. Runs under systemd on each dedicated Linux KVM host and executes jobs under fenced leases in Firecracker isolation. Neither production role mounts or admits SQLite.
At startup the server compares the candidate commit and artifact digest in its environment with the profile's release block and fails closed unless both match. Every store connection verifies the embedded migration ledger and refuses to serve schema drift, and the store audits its own credential before it serves anything.
3. Confirm readiness and the release binding
Query the edge through its TLS proxy.
curl -fsS "$MARKET_URL/health/ready"
curl -fsS "$MARKET_URL/v1/release"You should see a ready answer, and a release document naming the deployment ID, candidate commit, artifact digest, and configuration revision the process is bound to. A readiness answer reaches the backend at most once per READINESS_ANSWER_LIFETIME, one second, and a probe that arrives after the answer has aged out while a check is already in flight reports not ready.
4. Promote or roll back with the canary
chio-finding-market-canary qualifies a deployment against an exact release candidate from an attested ephemeral runner. evaluate-canary judges one audit-authority-signed observation against the same profile.
chio finding operator evaluate-canary \
--profile /etc/chio/hosted.json \
--observation /run/chio/canary-observation.jsoncanary_decision: promote
You should see canary_decision: promote, or canary_decision: rollback with one reason: binding, observation_window, freshness, availability, error_rate, latency, queue_age, or security_invariant. A rollback decision exits non-zero, so a promotion script can branch on the exit code alone.
Each transaction sees one tenant's rows
A caller picks its tenant with the Chio-Tenant-ID header and authenticates with a capability token plus a Chio-DPoP proof, or an API key pair, as the tenant's policy admits. Every tenant-scoped operation then sets chio.tenant_id inside its own transaction before it touches a row, so the binding never reaches a pooled connection's next user.
Forced row-level security on every table in TENANT_SCOPED_TABLES enforces the binding, for the table owner too. A statement that runs without the setting reads and writes nothing, and a security-definer function called for another tenant raises 42501. An unknown tenant returns TenantNotFound; a disabled tenant returns TenantDisabled, keeping durable state while every later scoped operation fails closed.
The monthly spend accumulator is written only by a security-definer trigger, and the runtime role can read it but never write it, so a compromised runtime cannot forge a low total. Hosted tenancy documents the same predicate and startup audit from the boundary side.
Read more in Tenancy enforcement, which gives the exact policy expression and the credential audit.
Storage authority moves only through signed transitions
Each tenant carries one row of authority state: which store is authoritative, an epoch, a mode, a mutations flag, the last replication and outbox sequences, and the digest of the transition that produced the row. A new tenant starts at authority sqlite, epoch 1, mode shadow, with mutations enabled.
chio_finding_market_apply_authority_transition applies; filled states admit domain writes. Every transition also requires a matching epoch, sequence, and configuration revision, and a signed replication check reporting zero lag and equal projection digests.Freeze stops domain writes under either authority. Cutover moves a frozen SQLite tenant to PostgreSQL with a rollback window of exactly 604800 seconds; rollback returns a frozen PostgreSQL tenant to SQLite before that deadline, and retire_sqlite closes the window after it. A domain write reaches durable state only when the authority is postgres, mutations_enabled is true, and the mode is rollback_window, authoritative, or retired, checked under a row lock so a freeze that lands mid-flight refuses the write.
Each transition is one signed chio.finding.hosted-authority-transition.v1 envelope applied under a tenant advisory lock, raising the epoch by exactly one. The replication credential that applies these barriers is separate from the runtime credential and cannot invoke public market commands. Market boundary documents the same four operations from the boundary side.
Ceilings bound work without weakening a guarantee
Each ceiling is an enforced constant or a setting the profile validates before the process starts. A ceiling that refuses work fails closed, and none degrades a guarantee to keep working. Admission, spend, and job ceilings are per tenant; request ceilings bound one edge replica. Capacity refusals reach the caller as authentication_capacity_unavailable or request_capacity_unavailable, HTTP 503, retryable.
/health/metrics exports chio_finding_market_edge_requests_total labelled accepted, denied, and shed. The route is absent from the Service, and its network policy admits only namespaces labelled chio.world/market-metrics-scraper, so traffic volume reaches a scraper without becoming publicly routable.
A single-operator venue serves discovery from a companion pool bounded by MAX_READ_COMPANIONS, and its serving-owner lease makes exactly one process the writer. A deployment that needs more concurrent writers uses the PostgreSQL profile.
Read more in Capacity ceilings, which lists each value, its constant or setting, and when it binds.
A failed job retries with capped backoff until its budget runs out
One worker tick claims a bounded batch for one tenant under a fenced lease, renews the lease on a heartbeat while the guest runs, and sorts each job into Completed, GuestRejected, Retried, Exhausted, or Cancelled. A failed execution with attempts left returns to failed and comes due after this delay.
delay = retry_base_secs * 2^(attempt_count - 1)
exponent capped at 10, delay capped at 3600 secondsThe attempt budget is the smaller of the profile's maxAttempts and the attempt limit inside the signed job capability, and a capability whose limit fails to verify exhausts the job at once. A canceled execution relinquishes its lease so another worker can claim the job immediately.
Profile validation refuses unless requireDefaultSeccomp is true and networkEnabled is false, and refuses a lease duration shorter than the execution timeout, so a lease cannot expire under a running guest.
A refusing runtime names what drifted
The server or a store connection refuses to start
The release block, the migration ledger, or a credential audit disagrees with the deployment. A role audit failure returns HostedMarketStoreError::Configuration. Correct the named drift, then rerun validate-hosted before starting again.
A request fails with a retryable code
Only rate_limited, authentication_capacity_unavailable, request_capacity_unavailable, and authentication_dependency_unavailable carry retryable: true. Retry those with backoff; every other chio.finding.hosted-error.v1 code carries retryable: false.
The canary decides rollback
The decision names one reason, and the non-zero exit stops the promotion script. Correct the named condition, then evaluate a fresh observation.
Hosted profile reference
Search for MAX_CONCURRENT_REQUESTS, retire_sqlite, or Chio-DPoP.
Components and binaries
Four crates carry the hosted profile. The edge and the store link into one server process; the worker library links into a daemon that runs on hosts of its own.
| Part | Crate | Where it runs |
|---|---|---|
PostgreSQL journal, tenancy, storage authority, retention | crates/platform/chio-finding-market-store-postgres | Every process that opens the database, each under its own least-privilege role |
HTTP routing, credential authentication, request admission, load shedding | crates/platform/chio-finding-hosted-edge | The market server, on loopback behind a TLS proxy sidecar in the same pod |
Firecracker- | crates/platform/chio-finding-worker | The worker daemon on a dedicated Linux KVM host |
Worker daemon: preflight, tenant rotation, tick reporting | crates/products/chio-finding-worker | systemd on the KVM host |
The platform crate's package is named chio-finding-worker and lives at crates/platform/chio-finding-worker. The products crate is packaged as chio-finding-worker-daemon, sits at crates/products/chio-finding-worker, and ships a binary also named chio-finding-worker. A profile validates against the library; systemd starts the binary. chio-finding-market-migrate comes from crates/products/chio-finding-market-migrator.
Profile validation
validate-hosted closes the deployment boundary offline. It requires the profile to be a private regular file inside its byte bound and byte-identical to its own canonical form, validates the profile, then opens every referenced path and resolves every named secret environment variable.
It builds the authenticator, loads the API-key pepper, TLS, and the trusted proxy, loads a signer for every role in FindingHostedSigningRole::ALL, loads the bond observer and the impairment publisher, and on Linux builds the worker executor. Only a run that survives all of that prints hosted_profile: valid.
Edge routes
The router carries three health routes outside the concurrency limiter and five guarded routes inside it. Liveness and readiness stay outside because the proxy probes them on the same pod, and shedding a probe would restart a live sidecar exactly while the edge is saturated. The router is defined in crates/platform/chio-finding-hosted-edge/src/lib.rs.
GET /health/live,GET /health/ready: liveness, and a readiness answer that reaches the backend at most once perREADINESS_ANSWER_LIFETIME.GET /health/metrics:chio_finding_market_edge_requests_totalin Prometheus exposition format, labelledaccepted,denied, andshed.GET /v1/release: the deployment ID, candidate commit, artifact digest, and configuration revision the process is bound to.GET /v1/findings,GET /v1/findings/{finding_id}: one bounded page, or one finding, for the buyer role. Each row is re-verified against its own signature and digest, and expired, voluntarily retracted, and enforcement-retracted findings drop out.POST /v1/findings/publish: one signedchio.finding.v1artifact from the seller role.POST /v1/findings/events/{operation}: exactly six operations,listing,delivery,challenge,verified-fix,retraction, andpenalty.
Each operation fixes its own domain event kind, artifact schema, governed action name, and writing role, and anything outside that list resolves to no operation. Coordinator-produced records, which include admission, purchase, purchase terminals, adjudication, enforcement, settlement, status, and audit, have no public route. The challenge route accepts only the buyer_submission authorization branch, so a venue audit cannot enter through it, and a penalty write must be signed by the profile's pinned penalty authority and name itself in both issuedBy and governingOperatorId.
Error body
Every failure returns the same body: chio.finding.hosted-error.v1 with a stable code, a fixed message, the caller's request ID, and a retryable flag that is true only for rate_limited, authentication_capacity_unavailable, request_capacity_unavailable, and authentication_dependency_unavailable. A request ID that is empty, over its byte bound, padded, or carrying a control character is replaced by invalid-request-id instead of being echoed.
Tenancy enforcement
Every tenant-scoped operation opens a transaction and runs SELECT set_config('chio.tenant_id', $1, TRUE) before it touches a row. Catalog reads take the same setting under SET TRANSACTION ISOLATION LEVEL REPEATABLE READ READ ONLY, so a page is answered from one snapshot.
Every table in TENANT_SCOPED_TABLES carries ENABLE ROW LEVEL SECURITY and FORCE ROW LEVEL SECURITY, plus one permissive policy for all commands, applying to PUBLIC, whose USING and WITH CHECK expressions are both exactly (tenant_id = NULLIF(current_setting('chio.tenant_id'::text, true), ''::text)). Without the setting that predicate evaluates to NULL for every row. The security-definer functions compare the requested tenant with the session setting first and raise 42501 with a message naming the mismatch, such as tenant context does not match authority transition.
verify_runtime_role refuses a role that is a superuser or carries BYPASSRLS, CREATEROLE, CREATEDB, REPLICATION, INHERIT, or any role membership, then checks its database privileges, per-table privileges, function grants, and the row-level security of each table; the startup audit also requires that the runtime role own none of those tables. The migrator, worker, retention, and replicator credentials each get their own audit, the migrator alone may bypass row-level security or create schema objects, and every pool connects with PgSslMode::VerifyFull against the certificate authority the profile names, whatever the DSN asked for. The runtime role holds SELECT on the spend accumulator and no write privilege.
Authority modes
The constraint chio_finding_market_authority_state_consistency_v1 pairs sqlite only with shadow or frozen, and postgres only with rollback_window, authoritative, retired, or frozen.
shadow: SQLite holds authority. PostgreSQL accepts signed replication events from it and appends signed replication checks, and admits no domain write of its own.frozen: the barrier state, reachable under either authority.mutations_enabledis false, replication events still land while SQLite holds authority, and it is the only modecutoverandrollbackaccept.rollback_window: PostgreSQL holds authority and admits domain writes, while a stored deadline still allows a return to SQLite.authoritative: admitted by the domain-write guard alongsiderollback_windowandretired, and accepted byfreezeas a source mode. No transition sets it.retired: PostgreSQL holds authority and admits domain writes, and the rollback deadline has passed.
Authority transitions
chio_finding_market_apply_authority_transition applies four operations, defined in crates/platform/chio-finding-market-store-postgres/migrations/0012_authority_replication.sql, lines 1035 to 1088. Every row also requires a matching epoch, sequence, and configuration revision, and a signed replication check reporting zero lag and equal projection digests.
| Operation | From | To | Also required |
|---|---|---|---|
freeze | Either authority; mode shadow, rollback_window, or authoritative | Same authority; mode frozen; mutations_ false | The request carries no rollback window |
cutover | sqlite, mode frozen | postgres, mode rollback_window, mutations enabled | The window ends exactly 604800 seconds after the transition's creation time |
rollback | postgres, mode frozen | sqlite, mode shadow, mutations enabled, stored window cleared | A stored window exists and the transition is created no later than its deadline |
retire_sqlite | postgres, mode rollback_window | postgres, mode retired, mutations enabled | The stored deadline has passed and the request carries no new window |
Presenting the identical envelope twice returns the replay outcome instead of moving the epoch again; a failed precondition returns a rejection and leaves the row untouched. A table constraint and the function both require the one-step epoch increase. Each applied transition is inserted into chio_finding_market_authority_transitions, where a trigger rejects any later UPDATE or DELETE, so the authority history is append-only.
Capacity ceilings
| What it bounds | Value | Constant or setting | When it binds |
|---|---|---|---|
Unexpired proof nonces one tenant holds at once | 1_000..=10_000_000 | identity. | A sweep runs first; if the count still stands, admission returns Capacity and the caller reads authentication_, HTTP 503, retryable |
Retained request admissions per live-proof slot | 64 | RETAINED_ | The same capacity refusal; a recorded admission is never evicted, because its retry depends on it |
Interval after which admission sweeps expired proof state | 3_600 seconds | DPOP_ | No refusal; the sweep runs inside the admitting transaction, and capacity pressure triggers it early |
Monthly spend units per tenant | 1..=MAX_ | Hosted | The accumulator trigger raises chio_ inside the inserting statement, mapped to Capacity |
Concurrent jobs per tenant | 1..=1_024 | Hosted | No refusal; caps the batch one worker tick claims, with the tick's job budget and the host-wide instance count |
Queued jobs per tenant: pending, leased, and failed | 1..=database. | Hosted | Job creation returns Capacity; a tenant value above the global one refuses to start instead of being clamped |
Retained jobs per tenant, in every state | 1..=10_000_000 | database., bounded by MAX_TENANT_JOBS | Job creation returns Capacity, checked under a tenant advisory lock so the count and the insert cannot interleave |
Rows in one catalog page, and finding IDs in one liveness query | 100, default 50 | MAX_ | A larger limit returns Invalid("catalog limit"), read as invalid_request, HTTP 400 |
In-flight requests per edge replica | 1_024 | MAX_ | The request sheds before a handler runs, with request_, HTTP 503, retryable, and its own request ID |
Request body | 4 MiB | MAX_ | The router's body-limit layer stops the request before a handler sees it, and the canonical-body check refuses the same bound again |
How long one readiness answer is reused | 1 second | READINESS_ | No refusal; one backend check runs at a time |
Live-proof capacity applies to every tenant's admissions with no per-tenant override, and both the profile and the authenticator refuse a value below 1_000, so the store's wider acceptance is unreachable through configuration.
Job states
A job is a row in chio_finding_market_jobs holding a canonical JSON payload, its digest, and a job_kind. The kind is an operator-chosen identifier bounded by MAX_JOB_KIND_BYTES, constrained by the signed worker capability, which must name the same job_kind as the job it authorizes. The state column is closed at pending, leased, completed, failed, and exhausted, with table constraints tying the lease columns, the result columns, and the error code to it.
A job that has used its attempt budget goes to exhaust_job with its error code and stops. Otherwise fail_job records the same code and the retry delay, and the row returns to failed. A guest result whose exit classification is other than Succeeded still completes the job durably, and the attested result records the refusal. A lease already lost to expiry leaves the row where the next claim finds it.
The worker stages digest-pinned guest assets into a unique jail, starts Firecracker only through its jailer, and exchanges bounded canonical JSON frames over virtio-vsock; the guest supervisor owns the vsock endpoint and runs the untrusted workload in a cgroup with a hard file-descriptor limit.