Cloud Run

Every request to the service lands on the Chio sidecar first. The sidecar admits or refuses it, writes the receipt, and only then forwards an admitted call to your application over loopback.

Cloud Run sends external traffic only to the container that declares a containerPort. In this Knative service that is chio-sidecar on 9090, running chio api protect against your OpenAPI document; the app listens on 8080 with no port declared. Secret Manager mounts the spec and the signing seed as files, and one instance per revision keeps receipts in memory.

1. Create the runtime identity and its three secrets

The revision runs as chio-sidecar@PROJECT_ID, which reads three secrets: the OpenAPI spec and the authority seed mount as files, and the sidecar control token arrives as an environment variable.

gcloud iam service-accounts create chio-sidecar --project="$PROJECT_ID"

# OpenAPI spec for the upstream app (mounted at /etc/chio/spec)
gcloud secrets create chio-openapi-spec --replication-policy=automatic
gcloud secrets versions add chio-openapi-spec --data-file=./openapi.yaml

# Authority signing seed (64 hex characters encoding 32 bytes)
gcloud secrets create chio-authority-seed --replication-policy=automatic
gcloud secrets versions add chio-authority-seed --data-file=./authority.seed

# Sidecar control token (bearer token gating /metrics + control endpoints)
gcloud secrets create chio-sidecar-control-token --replication-policy=automatic
openssl rand -hex 32 | \
    gcloud secrets versions add chio-sidecar-control-token --data-file=-

for s in chio-openapi-spec chio-authority-seed \
           chio-sidecar-control-token; do
    gcloud secrets add-iam-policy-binding "$s" \
      --member="serviceAccount:chio-sidecar@$PROJECT_ID.iam.gserviceaccount.com" \
      --role="roles/secretmanager.secretAccessor"
  done

Created service account [chio-sidecar].
Created secret [chio-openapi-spec].
Created version [1] of the secret [chio-openapi-spec].
Created secret [chio-authority-seed].
Created version [1] of the secret [chio-authority-seed].
Created secret [chio-sidecar-control-token].
Created version [1] of the secret [chio-sidecar-control-token].
Updated IAM policy for secret [chio-openapi-spec].
Updated IAM policy for secret [chio-authority-seed].
Updated IAM policy for secret [chio-sidecar-control-token].

You should see each secret created at version 1 and the accessor role bound on all three. The runtime account needs no project-wide roles/run.invoker grant to serve requests: these examples keep Cloud Run IAM authentication, so grant invocation rights to the caller separately.

2. Pin the manifest to your images and secret versions

Export digest-pinned APP_IMAGE and CHIO_SIDECAR_IMAGE and the numeric CHIO_SPEC_VERSION, CHIO_SEED_VERSION, and CHIO_CONTROL_VERSION. The edit also sets the hop ceiling to 4 seconds and names the log variable RUST_LOG, which the sidecar reads.

curl -fsS https://raw.githubusercontent.com/backbay-labs/chio/fe5657020cb669d7f9cce5efcae17ff3bdcde3bb/deploy/cloud-run/service.yaml -o cloud-run.yaml

# Requires yq v4. Apply your digest-pinned images and numeric secret versions.
yq -i '
  .spec.template.spec.serviceAccountName = "chio-sidecar@" + strenv(PROJECT_ID) + ".iam.gserviceaccount.com" |
  (.spec.template.spec.containers[] | select(.name == "app").image) = strenv(APP_IMAGE) |
  (.spec.template.spec.containers[] | select(.name == "chio-sidecar").image) = strenv(CHIO_SIDECAR_IMAGE) |
  (.spec.template.spec.containers[] | select(.name == "chio-sidecar").args) += ["--upstream-timeout-secs", "4"] |
  (.spec.template.spec.containers[] | select(.name == "chio-sidecar").env[] | select(.name == "CHIO_LOG_LEVEL").name) = "RUST_LOG" |
  (.spec.template.spec.containers[] | select(.name == "chio-sidecar").env[] | select(.name == "CHIO_SIDECAR_CONTROL_TOKEN").valueFrom.secretKeyRef.key) = strenv(CHIO_CONTROL_VERSION) |
  (.spec.template.spec.volumes[] | select(.name == "chio-openapi-spec").secret.items[0].key) = strenv(CHIO_SPEC_VERSION) |
  (.spec.template.spec.volumes[] | select(.name == "chio-authority-seed").secret.items[0].key) = strenv(CHIO_SEED_VERSION)
' cloud-run.yaml

yq '(.spec.template.spec.containers[] | select(.name == "chio-sidecar").args)[-2:]' cloud-run.yaml

- --upstream-timeout-secs
- "4"

You should see the two arguments that close the drain gap described below. The deployer needs permission to act as the service account and to pull both images.

3. Deploy the service and read its health

gcloud run services replace cloud-run.yaml --region="$REGION" --project="$PROJECT_ID"
URL=$(gcloud run services describe agent-tool-server --region="$REGION" \
    --project="$PROJECT_ID" --format='value(status.url)')
INVOKER_TOKEN=$(gcloud auth print-identity-token)
curl -fsS "$URL/chio/health" -H "Authorization: Bearer $INVOKER_TOKEN" \
    | jq '{status, receipt_backend}'

Applying new configuration to Cloud Run service [agent-tool-server]...
Done.
{
  "status": "healthy",
  "receipt_backend": "ephemeral"
}

You should see healthy with an ephemeral receipt backend, the value --allow-ephemeral-receipts selects. The CLI identity making these requests needs service-level invocation permission.

4. Watch the sidecar refuse, then admit, a protected call

With the pricing-service image as APP_IMAGE and its openapi.yaml as the spec secret, POST /v1/quote is a capability-protected route. CHIO_CAPABILITY_TOKEN holds the outside buyer's quote grant from Quickstart part 2.

curl -sS -D - -o /dev/null "$URL/v1/quote" \
    -H "Authorization: Bearer $INVOKER_TOKEN" \
    -H "Content-Type: application/json" -d '{"sku":"sku-118","percent":10}'
curl -sS -D - -o /dev/null "$URL/v1/quote" \
    -H "Authorization: Bearer $INVOKER_TOKEN" \
    -H "X-Chio-Capability: $CHIO_CAPABILITY_TOKEN" \
    -H "Content-Type: application/json" -d '{"sku":"sku-118","percent":10}'

HTTP/2 403
content-type: application/json
x-chio-receipt-id: 5e1d…a07c

HTTP/2 200
content-type: application/json
x-chio-receipt-id: 9b62…31fd

You should see a 403 for the uncredentialed request and a 200 for the scoped one, each with an x-chio-receipt-id. The admitted quote reached the pricing handler; the refused one never did. Use each receipt id to locate the evidence for the call.

A 4-second hop ceiling fits the drain inside the fixed 10-second grace

On scale-in or revision rollover, Cloud Run sends SIGTERM and SIGKILL 10 seconds later, and no service-level field raises that grace. The sidecar stops ingress and lets in-flight hops finish; each handler writes its receipt synchronously, so nothing is queued for a flush after the drain.

The sidecar runs no generic request timeout. It derives its drain from the upstream hop ceiling plus 5 seconds: at the default --upstream-timeout-secs 20 it asks for 25 seconds, and SIGKILL takes the last 15. The reference manifest passes no hop ceiling, so step 2 adds two entries to the sidecar's args, and a 4-second ceiling derives a 9-second drain:

# Two entries to add to the chio-sidecar args array. A 4s hop ceiling
# plus the 5s margin derives a 9s drain, inside Cloud Run's fixed 10s
# grace. Raising it without a platform that can hold a longer drain severs
# in-flight calls before they are receipted.
- "--upstream-timeout-secs"
- "4"

CPU stays allocated, and billed, through the shutdown period, so the wall clock is the binding limit. The service's timeoutSeconds: 300 is a separate, looser budget: a request admitted at second 299 cannot finish inside a 10-second grace, so the hop ceiling bounds each call. Checkpoint longer work, or run it on an orchestrator whose grace holds the operation. The app should mirror the contract: stop accepting requests, finish in-flight ones, exit.

Read more in Sidecar shutdown.

The app starts only after the sidecar's startup probe passes

The run.googleapis.com/container-dependencies annotation starts the app container once chio-sidecar's startupProbe succeeds. That probe polls /chio/health every second for up to 30 attempts. Readiness polls the same route and liveness polls /chio/live, each every 10 seconds with a failure threshold of 3.

/chio/health is dependency-aware and returns 503 when the receipt store can no longer persist; /chio/live answers for the process alone. A startup probe on /chio/live satisfies the annotation while the kernel cannot record, and the app starts serving into a gap in the audit trail. Declare a containerPort only on the sidecar: a port on the app container routes ingress around the kernel.

Read more in Sidecar health routes.

One instance per revision holds the receipt log in memory

A Cloud Run container gets an in-memory filesystem. Its one local volume type, emptyDir with medium: Memory and a sizeLimit, bills writes against the container's memory limit and vanishes with the instance. Cloud Storage FUSE and NFS mounts are network filesystems, and SQLite keeps its WAL index in shared memory that processes on separate hosts cannot share, so a WAL receipt database runs on neither.

The manifest therefore declares the log ephemeral with --allow-ephemeral-receipts and sets autoscaling.knative.dev/maxScale: "1", which limits instances per revision and retains nothing. Restarts, replacement, and overlapping revisions lose or split that history. Export receipts before an instance is replaced, and configure export, retention, and aggregation before you raise maxScale; a per-instance volume on another platform still holds an independent log.

Read more in Receipt store mechanics.

A new revision takes a slice of traffic before it takes all of it

gcloud run services replace applies the manifest's traffic configuration. For an existing service, keep traffic pinned to the current revision in spec.traffic before you replace, then export the two deployed names as PREVIOUS_REVISION and NEW_REVISION. --no-traffic belongs to gcloud run deploy, not to the replace command.

# Canary: send 10 percent of requests to the new revision.
gcloud run services update-traffic agent-tool-server \
    --region="$REGION" \
    --to-revisions="$NEW_REVISION=10,$PREVIOUS_REVISION=90"

# Rollback: return every request to the previous revision.
gcloud run services update-traffic agent-tool-server \
    --region="$REGION" \
    --to-revisions="$PREVIOUS_REVISION=100"

Traffic:
  90% agent-tool-server-00001-abc
  10% agent-tool-server-00002-def
Traffic:
  100% agent-tool-server-00001-abc

You should see the split, then every request back on the previous revision. Each revision starts with an empty receipt log, so export the outgoing revision's receipts before it scales to zero.

Operational logs reach Cloud Logging; signed receipts stay in the receipt path

Stdout and stderr from both containers land under cloud_run_revision, and the CLI writes redacted text events. Query signed receipts through the receipt store or its export path.

gcloud logging read \
    'resource.type="cloud_run_revision"
     AND resource.labels.service_name="agent-tool-server"
     AND labels.container_name="chio-sidecar"' \
    --limit=50 --format=json

# Tail the chio sidecar
gcloud beta run services logs tail agent-tool-server \
    --region="$REGION" \
    --log-filter='labels.container_name="chio-sidecar"'

Scrape /metrics from a loopback collector or an authenticated remote scraper. Trace export needs application instrumentation and a configured exporter; collected CLI text logs carry neither receipts nor traces.

Read more in Node Observability.

Internal ingress and private egress are annotations on the service

The default *.run.app URL is public over HTTPS with a Google-managed certificate. For an internal-only service, add run.googleapis.com/ingress: internal to the service annotations and front it with an internal HTTPS load balancer. When the kernel reaches a private capability authority or a VPC-peered receipt store, attach a Serverless VPC connector:

# Two annotations to add to the service metadata. Neither is in the
# reference manifest, which deploys with the default egress path.
metadata:
  annotations:
    run.googleapis.com/vpc-access-connector: projects/PROJECT_ID/locations/REGION/connectors/chio-vpc
    run.googleapis.com/vpc-access-egress: private-ranges-only

To join a shared Trust Control Plane over that connector, pass the global --control-url flag and the CHIO_CONTROL_TOKEN bearer token. Terminate any additional mTLS at the load balancer or an Envoy sidecar.

Each failure shows up at a probe, a header, or the drain

The revision never becomes ready

A spec or receipt store that fails to open exits the sidecar non-zero, and Cloud Run fails the revision after 30 startup attempts. Read gcloud run revisions describe "$NEW_REVISION" --region="$REGION" --format="value(status.conditions)", then check the secret versions and the accessor role.

Responses arrive without an x-chio-receipt-id

The app container declares a port, so ingress reaches it directly. Remove the port; only chio-sidecar declares containerPort: 9090.

Calls end without receipts during scale-in

The hop ceiling is missing or above 5 seconds, so the drain outlasts the grace. Restore --upstream-timeout-secs 4 in the sidecar's args.

Manifest, probes, and scaling reference

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: agent-tool-server
  annotations:
    run.googleapis.com/launch-stage: GA
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "1"
        autoscaling.knative.dev/maxScale: "1"
        run.googleapis.com/container-dependencies: '{"app":["chio-sidecar"]}'
        run.googleapis.com/execution-environment: gen2
        prometheus.io/path: "/metrics"
        prometheus.io/port: "9090"

    spec:
      serviceAccountName: chio-sidecar@PROJECT_ID.iam.gserviceaccount.com
      containerConcurrency: 80
      timeoutSeconds: 300

      containers:
        - name: app
          image: APP_IMAGE_PLACEHOLDER
          env:
            - name: CHIO_SIDECAR_URL
              value: "http://localhost:9090"
          resources:
            limits:
              cpu: "1"
              memory: 512Mi
          startupProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 2
            periodSeconds: 2
            failureThreshold: 30
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 10
            failureThreshold: 3

        - name: chio-sidecar
          image: ghcr.io/backbay-labs/chio-sidecar:latest
          ports:
            - containerPort: 9090

args:
  - "api"
  - "protect"
  - "--upstream"
  - "http://127.0.0.1:8080"
  - "--spec"
  - "/etc/chio/spec/openapi.yaml"
  - "--listen"
  - "0.0.0.0:9090"
  # Cloud Run has no per-instance persistent disk, so the audit log
  # cannot be durable here. Opt into ephemeral in-memory receipts
  # explicitly instead of pointing --receipt-store at scratch storage,
  # which would boot reporting a durable backend yet lose every receipt
  # on revision recycle. For a durable audit trail, front a client-server
  # store or run on a per-instance-disk platform.
  - "--allow-ephemeral-receipts"
  - "--authority-seed-file"
  - "/etc/chio/seed/authority.seed"

          env:
            - name: RUST_LOG
              value: "info"
            - name: CHIO_SIDECAR_CONTROL_TOKEN
              valueFrom:
                secretKeyRef:
                  name: chio-sidecar-control-token
                  key: latest
          volumeMounts:
            - name: chio-openapi-spec
              mountPath: /etc/chio/spec
            - name: chio-authority-seed
              mountPath: /etc/chio/seed

          resources:
            limits:
              cpu: "500m"
              memory: 128Mi
          startupProbe:
            httpGet:
              path: /chio/health
              port: 9090
            initialDelaySeconds: 1
            periodSeconds: 1
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /chio/health
              port: 9090
            periodSeconds: 10
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /chio/live
              port: 9090
            periodSeconds: 10
            failureThreshold: 3

      volumes:
        - name: chio-openapi-spec
          secret:
            secretName: chio-openapi-spec
            items:
              - key: latest
                path: openapi.yaml
        - name: chio-authority-seed
          secret:
            secretName: chio-authority-seed
            items:
              - key: latest
                path: authority.seed

# A template annotation to add. The reference manifest does not set it,
# so it runs on request-based CPU.
spec:
  template:
    metadata:
      annotations:
        run.googleapis.com/cpu-throttling: "false"

Next: keep receipts beyond the instance