Hosted Tenancy

The authenticated tenant stays bound to work that runs later

A hosted task begins in an authenticated request and often finishes in a queue or a worker, after the HTTP context is gone. The tenant travels with the retained work, and every later owner derives it from trusted state.

A row identifier or a payload field never selects a tenant: guessing another tenant's job id or naming another tenant in a payload reaches no other partition. Both harbor-goods and ostrander can hold a job called quote-118, and each request carries its own tenant from the edge into the store, so the same identifier names two different rows.

Isolation covers availability as well as data. A full queue for harbor-goods leaves ostrander's capacity untouched, because per-tenant limits stop one tenant's queued or active work from exhausting what the others share.

Every tenant table refuses rows outside the session's tenant

Each of the 31 tables in TENANT_SCOPED_TABLES carries one row-level security predicate, checked against the chio.tenant_id session variable a connection sets before it touches a row:

(tenant_id = NULLIF(current_setting('chio.tenant_id'::text, true), ''::text))

The NULLIF turns an unset or empty tenant into NULL, so a connection without a tenant context reads, updates, and deletes no rows, and the policy's WITH CHECK refuses its inserts. Above the row level, an unknown tenant returns TenantNotFound and a tenant the operator disabled returns TenantDisabled; both refuse before the store touches tenant data, and the public edge answers both with one generic code. Tenancy in the hosted profile covers the database roles that enforce the predicate.

At startup, verify_runtime_rls_surface rereads Postgres's own catalogs, pg_class and pg_policy. It confirms that every scoped table has row-level security enabled and forced, that the runtime role owns none of them, and that each policy applies to every command and every role with exactly the predicate above. Any drift, such as a missing table, an unforced policy, or a policy that reads differently, returns HostedMarketStoreError::Configuration and the process does not start. The result is an unsigned local self-check, so a remote verifier runs its own query against the same database.

One durable row bounds each tenant's queue, concurrency, and spend

A tenant's ceiling is one HostedTenantLimits row: a maximum concurrent job count, a maximum queue depth, and a maximum monthly spend, all bound to a single hosted configuration revision.

pub struct HostedTenantLimits {
    pub(super) max_concurrent_jobs: u32,
    pub(super) max_queued_jobs: u64,
    pub(super) max_monthly_spend_units: u64,
    pub(super) configuration_revision: String,
}

The concurrency ceiling holds under contention because claiming a batch of jobs for one tenant runs under pg_advisory_xact_lock(hashtextextended(tenant_id, 2)). Two workers racing to claim work for the same tenant serialize on that lock before either counts the tenant's active leases, so concurrent claims stay within max_concurrent_jobs. A signed chio.finding.hosted-quota-alert.v1 record, appended through the store's append_quota_alert, reports a tenant ceiling reached or exceeded; the store refuses one whose observed value is below its limit.

Each job runs in its own jailed microVM with no network

A FirecrackerWorkerConfig pins every binary and image a guest can touch by sha256 digest: the worker, Firecracker, the jailer, the guest kernel, and the rootfs. A worker runs only the code it was configured to run.

jailer_args puts each guest in its own PID namespace under memory, PID-count, and CPU cgroup ceilings, with open-file and output-size resource limits, and launches Firecracker with --no-api so nothing outside the jail can drive it. The plan keeps Firecracker's default seccomp filter, passes no --netns, and the guest's VM configuration declares no network interface; the tests jailer_plan_enforces_namespaces_cgroups_and_default_seccomp and machine_config_has_no_network_and_keeps_default_seccomp hold both. The hosted worker layers lease, retry, and outcome accounting on top of the jail.

The edge reads client identity from one configured proxy header

HostedTrustedProxy reconstructs a client's IP, scheme, and host from a single Forwarded header, sent by a peer whose IP is in an exact configured set, carrying a shared authentication token the proxy compares in constant time. X-Forwarded-For, X-Forwarded-Host, and X-Forwarded-Proto are refused outright, as is a Forwarded header that repeats a parameter, carries more than one hop, or fails to parse.

The test proxy_spoofing_and_ambiguous_headers_fail_closed covers both paths: a well-formed header from an untrusted peer and an ambiguous mix of X-Forwarded-* and Forwarded headers from a trusted one are refused, and a wrong shared token is refused as HostedEdgeError::AuthenticationFailed. The tenant binding, chio.finding.hosted-tenant-binding.v1, comes from the tenant header before any credential is inspected; the edge then authenticates the credential and proof against that tenant, because header text alone authenticates nothing.

Two tenants' quote-118 jobs stay apart, and quote-204 waits for capacity

Start from a provisioned hosted operator profile and a non-owning runtime database role. Validate the profile first; replace /etc/chio/hosted.json with your own, a private regular file in canonical JSON.

chio finding operator validate-hosted --profile /etc/chio/hosted.json

validate-hosted binds no listener and opens no database pool. Its profile checks run before it loads anything, and it refuses a worker configuration unless requireDefaultSeccomp is true and networkEnabled is false, the two invariants the running jailer plan holds. A passing profile then loads TLS, the trusted proxy, and a signer for every role, and a remote signer is preflighted against its signing service. The profile itself follows chio.finding.hosted-operator-profile.v1; operator commands lists what a full run prints.

The comparison runs against PostgresFindingMarketStore with two empty, disposable tenants, each holding the one-job hosted-v17 limits, and a store retention ceiling of at least two. Pass only tenant ids the edge has authenticated: a HostedTenantId constructor validates an identifier and authenticates nothing. Download the complete source.

use chio_core_types::sha256_hex;
use chio_finding_market_store_postgres::{
    HostedJobWriteOutcome, HostedMarketStoreError, HostedTenantId, HostedTenantLimits,
    PostgresFindingMarketStore,
};

pub fn reviewed_limits() -> Result<HostedTenantLimits, HostedMarketStoreError> {
    HostedTenantLimits::new(1, 1, 5_000, "hosted-v17")
}

pub fn compare_tenant_limit_shapes() -> Result<(), HostedMarketStoreError> {
    let limits = reviewed_limits()?;
    assert_eq!(limits.max_concurrent_jobs(), 1);
    assert_eq!(limits.max_queued_jobs(), 1);
    assert_eq!(limits.configuration_revision(), "hosted-v17");
    for (active, queued, spend) in [(0, 1, 5_000), (1, 0, 5_000), (1, 1, 0), (1_025, 1, 5_000)] {
        assert!(HostedTenantLimits::new(active, queued, spend, "hosted-v17").is_err());
    }
    println!("CONFIGURE original hosted-v17 limits; refuse zero or oversized declarations");
    Ok(())
}

/// The edge has already authenticated each supplied tenant. This function is
/// an owner-run test on empty, pre-provisioned disposable tenant partitions.
/// Both have reviewed_limits(); store retention capacity is at least two.
pub async fn qualify_tenant_partitions(
    runtime: &PostgresFindingMarketStore,
    harbor_goods: &HostedTenantId,
    ostrander: &HostedTenantId,
    now: u64,
) -> Result<(), Box<dyn std::error::Error>> {
    if harbor_goods == ostrander { return Err("qualification requires two independent tenants".into()); }
    let limits = reviewed_limits()?;
    for tenant in [harbor_goods, ostrander] {
        runtime.verify_tenant_limits(tenant, &limits).await?;
        if runtime.job_count(tenant).await? != 0 { return Err("disposable partition is not empty".into()); }
    }
    let payload = br#"{"sku":"sku-118","percent":10}"#;
    let request = sha256_hex(payload);
    let put = |tenant, id| runtime.put_job(tenant, id, "quote", &request, payload, now, now);
    assert_eq!(put(harbor_goods, "quote-118").await?, HostedJobWriteOutcome::Inserted);
    assert_eq!(put(harbor_goods, "quote-118").await?, HostedJobWriteOutcome::ExactReplay);
    assert!(runtime.get_job(ostrander, "quote-118").await?.is_none());
    // Same job ID in the other tenant denotes its own row and quota.
    assert_eq!(put(ostrander, "quote-118").await?, HostedJobWriteOutcome::Inserted);
    assert!(matches!(put(harbor_goods, "quote-204").await, Err(HostedMarketStoreError::Capacity)));
    let original = runtime.get_job(harbor_goods, "quote-118").await?.ok_or("original job disappeared")?;
    if original.payload_json != payload { return Err("original job payload changed".into()); }
    println!("ISOLATE quote-118 lookup; ostrander has its own same-ID row");
    println!("REPLAY original without another queue entry; refuse harbor-goods' next job at capacity");
    Ok(())
}

CONFIGURE original hosted-v17 limits; refuse zero or oversized declarations

You should see the declaration line: the limit constructor accepts the one-job declaration and refuses zero or oversized ones. That comparison runs anywhere. qualify_tenant_partitions compiles against the store API and runs its database assertions in your configured operator environment, where each operation must observe:

1. harbor-goods writes quote-118. Inserted, under the tenant-scoped transaction and queue admission.
2. harbor-goods repeats it exactly. ExactReplay: the original kind, request digest, and canonical payload match before any new quota admission.
3. ostrander looks up quote-118. None, through the scoped lookup and forced row-level security.
4. ostrander writes its own quote-118. Inserted in its own partition, against its own capacity.
5. harbor-goods submits quote-204. Capacity, from the per-tenant queue count serialized under the tenant advisory lock; quote-118 is unchanged.

The public edge answers with generic codes; a store caller gets typed errors

A direct store caller, such as the worker, receives a typed HostedMarketStoreError. The public edge authenticates every guarded request before it reaches the store and maps store and request reasons onto a small fixed set of wire codes, so several causes share one code.

The wire bodies are "Authentication failed." (HostedEdgeError::AuthenticationFailed), "The request is invalid.", and "The service is temporarily unable to accept this request." A replayed proof (replay_rejected), an exhausted capability budget (authorization_failed), and a store at capacity or unavailable keep their own codes. A tenant caller sees one reason for every authentication_failed cause, by design, so the check reveals nothing about another tenant's state.

A privileged operator reads the cause from the service records: the bound tenant, the configured proxy peer and token, the credential proof, quota state, the database isolation check, and the worker preflight. Repair the failing configuration or the authorized capacity policy, then retry, and keep the generic public error. Outbound HTTP from the host is a separate boundary with its own egress contract.

Next: keep authority exact as the market changes