Azure Container Apps

The sidecar container runs one subcommand, chio api protect. The reverse proxy behind it is the chio-api-protect crate, and that crate owns the three things this recipe has to fit around: the upstream hop ceiling, the drain window derived from it, and the two health routes the template probes.

Contracts this recipe must satisfy

Four obligations are defined at the Node rung, not here. The Bicep template below is one vendor's way of meeting them. If an edit to it breaks a row in this table, the edit is wrong, whatever Container Apps accepts.

The default grace already fits, and stops fitting when you tune the hop

chio api protect writes each receipt synchronously inside the request handler, so finishing the in-flight requests during the drain is the whole durability guarantee: nothing is queued to flush afterwards. It therefore runs no generic request timeout and instead derives its drain window from the upstream hop ceiling plus a five-second margin. The default --upstream-timeout-secs 20 asks for 25 seconds, inside the 30-second terminationGracePeriodSeconds default. That default is the platform's, not the template's: the template block opens straight onto containers and declares no grace period at all.

template: {
  containers: [
    {
      name: 'app'

So anything above a 25-second hop ceiling needs the grace period stated explicitly, in the same revision, at least five seconds above --upstream-timeout-secs:

// A key to add to the template block. Hold it at least five seconds
// above --upstream-timeout-secs, which is what the sidecar adds to derive
// its own drain window.
template: {
  terminationGracePeriodSeconds: 60
  containers: [ ... ]
}

Container Apps removes the replica from the load-balancing pool as soon as the Readiness probe starts failing, so traffic shed overlaps the drain rather than following it. That is a real advantage over platforms where the two are serial, and it is worth nothing if the grace period is shorter than the hop the kernel is still waiting on.

Two different storage failures, not one

The state contract has two halves, and Container Apps fails a different one with each storage type. AzureFile (SMB) and NfsAzureFile are network filesystems: SQLite keeps the WAL index in shared memory that processes on separate hosts cannot share, so a WAL receipt database is excluded outright. storageType: 'EmptyDir' is genuinely replica-local and would carry a single writer, but it is ephemeral by definition: the files live for the lifetime of the replica, surviving a container restart inside it and nothing beyond that. Its ceiling is set by the replica's vCPU allocation, 1 GiB at 0.25 vCPU or lower through 8 GiB above 1 vCPU.

A receipt store on EmptyDir is therefore a WAL database with no durability guarantee, which reads as durable in /chio/health and is not. This recipe declares the audit log ephemeral instead, which is the honest version of the same property, and pins maxReplicas: 1 to keep one coherent stream. The corollary is that no scale.rules entry belongs on this revision: an http rule with concurrentRequests, or any KEDA trigger with its pollingInterval and cooldownPeriod, fans the audit trail out across replicas that cannot see each other.

Container Apps has no container start ordering

This is the one obligation the platform gives you nothing for. Entries in the containers array start concurrently and there is no dependsOn between them. initContainers do not close the gap either: they run to completion before any app container starts, so an init container cannot poll a sidecar that has not been started yet, and they support no probes.

What the sidecar's Readiness probe on /chio/health does buy is that no external request reaches the replica until the kernel is ready, so the exposure is narrow: it is the app's own startup work, before its first inbound request. Close it in the app. Block on $CHIO_SIDECAR_URL/chio/health returning 200 before issuing the first governed call, and treat a non-200 as fatal rather than proceeding ungoverned.

Architecture

The container app exposes a single ingress on port 9090 (the sidecar); the app listens on 8080 over loopback. One Key Vault-backed secret (the authority signing seed) resolves through the managed identity and is projected into the sidecar as a mounted file. The revision runs a single replica: the audit log is an in-memory stream, so fanning out would split it.

Manifest walkthrough

This reference deploys one Microsoft.App/containerApps@2024-03-01 resource. Each section below maps to one operational concern.

Parameters

The template takes eight parameters. Four are mandatory at deploy time: managedEnvironmentId, userAssignedIdentityId, chioAuthoritySeedSecretUri (the Key Vault URI of the authority signing seed), and specStorageName (the environment storage name backing the read-only OpenAPI spec share). The rest default: location (resourceGroup().location), containerAppName (agent-tool-server), appImage (a placeholder overridden at deploy time), and chioSidecarImage (ghcr.io/backbay-labs/chio-sidecar:latest).

Identity and ingress

resource containerApp 'Microsoft.App/containerApps@2024-03-01' = {
  name: containerAppName
  location: location
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${userAssignedIdentityId}': {}
    }
  }
  properties: {
    managedEnvironmentId: managedEnvironmentId
    configuration: {
      activeRevisionsMode: 'Single'
      ingress: {
        external: true
        targetPort: 9090
        transport: 'auto'
        allowInsecure: false
      }

The app runs as a user-assigned managed identity, which needs Key Vault Secrets User on the referenced secrets. Use a user-assigned identity when it must survive revision-set rotation or serve multiple apps in the same trust boundary; unlike a system-assigned identity, it can be reused across apps. activeRevisionsMode: 'Single' replaces the previous revision on every deploy; switch to 'Multiple' for blue-green (covered below). external: true attaches a public FQDN; targetPort: 9090 binds ingress to the sidecar; allowInsecure: false forces HTTPS at the edge.

Secrets block (Key Vault reference)

      secrets: [
        {
          name: 'chio-authority-seed'
          keyVaultUrl: chioAuthoritySeedSecretUri
          identity: userAssignedIdentityId
        }
      ]
    }

One entry declares the authority signing seed, backed by a Key Vault URI and the managed identity used to read it. It is delivered to the sidecar as a mounted file (see the volumes block below), never injected as an env var. Container Apps caches the resolved value at revision creation. Updating the Key Vault secret does not propagate to a running revision; you have to create a new revision (or set resyncSecrets via az CLI) for the new value to land.

Application container

    template: {
      containers: [
        {
          name: 'app'
          image: appImage
          resources: {
            cpu: json('0.75')
            memory: '1.5Gi'
          }
          env: [
            {
              name: 'CHIO_SIDECAR_URL'
              value: 'http://localhost:9090'
            }
          ]
          probes: [
            {
              type: 'Startup'
              httpGet: {
                path: '/healthz'
                port: 8080
              }
              initialDelaySeconds: 2
              periodSeconds: 2
              failureThreshold: 30
            }
            {
              type: 'Liveness'
              httpGet: {
                path: '/healthz'
                port: 8080
              }
              periodSeconds: 10
              failureThreshold: 3
            }
          ]
        }

CPU is declared as a JSON number (json('0.75') for 0.75 cores); memory uses the Gi suffix. Total container CPU + memory must align with allowed workload profile combinations. The startup probe gives the app 60 seconds (30 attempts × 2s) before liveness takes over.

Sidecar container

        {
          name: 'chio-sidecar'
          image: chioSidecarImage

The argument vector is the whole kernel configuration. There is no config file and no second source:

args: [
  'api'
  'protect'
  '--upstream'
  'http://127.0.0.1:8080'
  '--spec'
  '/etc/chio/spec/openapi.yaml'
  '--listen'
  '0.0.0.0:9090'
  // Container Apps has no per-replica persistent disk, so the audit log
  // cannot be durable here. Opt into ephemeral in-memory receipts
  // explicitly instead of pointing --receipt-store at scratch storage,
  // which would boot reporting a durable backend yet lose every receipt
  // on revision recycle. For a durable audit trail, front a client-server
  // store or run on a per-instance-disk platform.
  '--allow-ephemeral-receipts'
  '--authority-seed-file'
  '/etc/chio/seed/authority.seed'
]

          resources: {
            cpu: json('0.25')
            memory: '0.5Gi'
          }
          volumeMounts: [
            {
              volumeName: 'chio-openapi-spec'
              mountPath: '/etc/chio/spec'
            }
            {
              volumeName: 'chio-authority-seed'
              mountPath: '/etc/chio/seed'
            }
          ]

Only args is set so the image entrypoint is preserved, whichever entrypoint the pulled image carries. The published tag is built from deploy/docker/Dockerfile.sidecar, an Alpine image entered through /sbin/tini. An image built locally from the repository's own instructions is the distroless deploy/sidecar/Dockerfile and enters at chio-sidecar with no tini. Both carry the same tag. The default image CMD is --help, which would exit immediately; the override turns it into a long-running api protect reverse proxy. --spec points at the operator-provided OpenAPI document mounted from the AzureFile share; the kernel derives its route and scope table from it, never from the upstream. --allow-ephemeral-receipts opts into an in-memory audit log because Container Apps has no per-replica persistent disk, and --authority-seed-file loads the signing seed from the mounted Key Vault secret file.

Sidecar environment

          env: [
            {
              name: 'CHIO_LOG_LEVEL'
              value: 'info'
            }
          ]

The sidecar takes a single plain env var, CHIO_LOG_LEVEL. Everything else it needs, the OpenAPI spec and the authority seed, arrives as mounted files declared in the volumes block. There is no signing-key, capability-authority, kernel-config, policy-source, or receipt-sink env var: the kernel is configured entirely from the args above plus the mounted OpenAPI document.

Volumes

      volumes: [
        {
          name: 'chio-openapi-spec'
          storageType: 'AzureFile'
          storageName: specStorageName
        }
        {
          name: 'chio-authority-seed'
          storageType: 'Secret'
          secrets: [
            {
              secretRef: 'chio-authority-seed'
              path: 'authority.seed'
            }
          ]
        }
      ]

Two volumes. chio-openapi-spec is a read-only AzureFile share backed by specStorageName; chio-authority-seed is a Secret-type volume that projects the Key Vault secret onto disk at /etc/chio/seed/authority.seed. There is deliberately no receipt-store volume: Container Apps offers no per-replica persistent disk, and a durable receipt log is a single-writer SQLite database in WAL mode that needs a local filesystem, so it cannot run on an Azure Files share. The template keeps the audit log in memory with --allow-ephemeral-receipts.

Sidecar probes

          probes: [
            {
              type: 'Startup'
              httpGet: {
                path: '/chio/health'
                port: 9090
              }
              initialDelaySeconds: 1
              periodSeconds: 1
              failureThreshold: 30
            }
            {
              // Process-only liveness: a dependency blip must not restart a
              // container that is still serving. Readiness gates on /chio/health.
              type: 'Liveness'
              httpGet: {
                path: '/chio/live'
                port: 9090
              }
              periodSeconds: 10
              failureThreshold: 3
            }
            {
              type: 'Readiness'
              httpGet: {
                path: '/chio/health'
                port: 9090
              }
              periodSeconds: 5
              failureThreshold: 3
            }
          ]

Three probes on two paths. Startup and readiness poll /chio/health, which is dependency-aware: it returns 503 when the receipt store can no longer persist. Liveness polls a different path, /chio/live, which is process-only and deliberately does not flap on a dependency blip. Readiness gates ingress: if the kernel goes unhealthy, Container Apps pulls the replica from the load-balancing pool before liveness would ever recycle it.

Scale block

      scale: {
        minReplicas: 1
        maxReplicas: 1
      }

maxReplicas is pinned to 1 with no scale rules. The audit log is an explicitly ephemeral in-memory stream, so one replica keeps one coherent stream; a second replica would keep its own separate, incoherent log. For a single durable audit trail across scale or restart, front a client-server audit store or move to a per-instance-disk platform. See Scaling for the full rationale.

Outputs

output containerAppFqdn string = containerApp.properties.configuration.ingress.fqdn
output containerAppName string = containerApp.name

The deployment exports the assigned FQDN and the resource name so downstream automation (Front Door routes, DNS records, smoke tests) can pick them up without a second az call.

Secrets

Create the Key Vault secret holding the authority signing seed and grant the managed identity get permission before deploying the Bicep template.

# Create the Key Vault.
$ az keyvault create --resource-group my-rg --name chio-prod-kv \
    --location eastus --enable-rbac-authorization true

# Add the authority signing seed. This is the raw file the sidecar loads via
# --authority-seed-file; the sidecar auto-generates one if absent, but pinning
# it keeps the signing identity stable across revisions.
$ az keyvault secret set --vault-name chio-prod-kv --name chio-authority-seed \
    --file ./authority.seed

# Create the user-assigned managed identity.
$ az identity create --resource-group my-rg --name chio-prod-mi

# Grant the identity Key Vault Secrets User on the vault scope.
$ MI_PRINCIPAL_ID=$(az identity show --resource-group my-rg --name chio-prod-mi \
    --query principalId -o tsv)
$ az role assignment create --role "Key Vault Secrets User" \
    --assignee-object-id "$MI_PRINCIPAL_ID" --assignee-principal-type ServicePrincipal \
    --scope $(az keyvault show --name chio-prod-kv --query id -o tsv)

# Capture the secret URI and identity ID for the deploy parameters.
$ SEED_URI=$(az keyvault secret show --vault-name chio-prod-kv \
    --name chio-authority-seed --query id -o tsv)
$ MI_ID=$(az identity show --resource-group my-rg --name chio-prod-mi --query id -o tsv)

Networking

External and internal ingress

With ingress.external: true, the app gets a public FQDN under *.azurecontainerapps.io with managed TLS. Switch to false for an internal-only environment; the FQDN resolves only inside the VNet the managed environment is attached to. Front internal apps with Application Gateway or Front Door for WAF and custom domains.

Custom domains

# Bind a custom domain with a managed certificate.
$ az containerapp hostname add --resource-group my-rg \
    --name agent-tool-server --hostname tools.example.com
$ az containerapp hostname bind --resource-group my-rg \
    --name agent-tool-server --hostname tools.example.com \
    --environment my-env --validation-method CNAME

VNet integration

For private capability authorities or VNet-peered receipt stores, the managed environment must be created with a delegated subnet; VNet-attached apps reach internal endpoints directly.

Health probes and graceful shutdown

Probe configuration above. On revision rollover or scale-in, Container Apps sends SIGTERM and respects the terminationGracePeriodSeconds on the template (defaults to 30). The sidecar handles SIGTERM by stopping ingress, draining in-flight evaluations, and exiting. Container Apps removes the replica from the load-balancing pool as soon as the readiness probe starts failing, so drain is concurrent with traffic shed.

Scaling

The reference manifest pins minReplicas and maxReplicas to 1. That is not a throughput ceiling chosen for cost; it is a correctness constraint. The audit log runs in memory (--allow-ephemeral-receipts) because Container Apps has no per-replica persistent disk, and a durable receipt log is a single-writer SQLite database in WAL mode that needs a local filesystem. A second replica would keep its own separate in-memory audit stream, so the trail would fragment across replicas.

To scale horizontally you first have to move the audit log off the replica: front a client-server audit store, or run on a platform that attaches a per-instance block volume (an ECS task with a launch-attached EBS volume, or a StatefulSet PVC) and point --receipt-store at it. Only then does adding replicas keep one coherent trail. See ECS Fargate for the per-task-disk durable-receipts shape.

Observability

Stdout / stderr from both containers ships to the Log Analytics workspace bound to the managed environment. Sidecar log lines are structured JSON; query them in Log Analytics:

# Find recent denied receipts on the sidecar container
$ az monitor log-analytics query \
    --workspace $LAW_ID \
    --analytics-query '
ContainerAppConsoleLogs_CL
| where ContainerName_s == "chio-sidecar"
| where Log_s contains "\"event\":\"receipt\""
| where Log_s contains "\"verdict\":\"deny\""
| order by TimeGenerated desc
| take 50
'

For metrics and tracing, attach a third sidecar container running the OTel collector (or use Azure Monitor Agent integration on the environment) and point the kernel at it via OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317. See Observability for collector wiring.

Revisions and blue-green

The reference manifest uses activeRevisionsMode: 'Single', which replaces the previous revision on every deploy. To run blue-green, switch to 'Multiple' and pass a revision suffix per deploy:

// Two edits to the reference template, which ships neither key. The
// template declares activeRevisionsMode: 'Single' and no revisionSuffix.
configuration: {
  activeRevisionsMode: 'Multiple'
  ...
}
template: {
  revisionSuffix: 'v42'   // bumped per deploy
  ...
}

# Deploy a new revision, send 10% of traffic to it.
$ az containerapp ingress traffic set \
    --resource-group my-rg \
    --name agent-tool-server \
    --revision-weight agent-tool-server--v42=10 agent-tool-server--v41=90

# Promote.
$ az containerapp ingress traffic set \
    --resource-group my-rg \
    --name agent-tool-server \
    --revision-weight agent-tool-server--v42=100

# Roll back.
$ az containerapp ingress traffic set \
    --resource-group my-rg \
    --name agent-tool-server \
    --revision-weight agent-tool-server--v41=100

Cost considerations

Container Apps bills on vCPU-seconds and memory-GiB-seconds, with a free per-month tier on the consumption profile. Three knobs dominate: minReplicas (every warm replica bills 24/7 on its full container allocation; set to 0 in dev for scale-to-zero), workload profile (consumption is cheapest; the dedicated profile is required for VNet integration with private endpoints), and Log Analytics retention (cap workspace retention at 30 days unless compliance needs longer). This reference uses an in-memory receipt log. Log Analytics therefore retains operational logs but not durable receipts.

Operations

Deploy is a single az deployment group create. Rollback in 'Single' mode is a redeploy of the previous template; in 'Multiple' mode, shift traffic weights to a prior revision (no redeploy).

# Deploy.
$ az deployment group create --resource-group my-rg \
    --template-file deploy/azure/container-app.bicep \
    --parameters location=eastus managedEnvironmentId=$ENV_ID \
        userAssignedIdentityId=$MI_ID \
        chioAuthoritySeedSecretUri=$SEED_URI \
        specStorageName=$SPEC_STORAGE \
        appImage=ghcr.io/your-org/your-app:1.4.2

# Roll back via traffic split.
$ az containerapp ingress traffic set --resource-group my-rg \
    --name agent-tool-server \
    --revision-weight agent-tool-server--v41=100 agent-tool-server--v42=0

# Tail sidecar logs.
$ az containerapp logs show --resource-group my-rg --name agent-tool-server \
    --container chio-sidecar --follow

# Open a shell on a running replica.
$ az containerapp exec --resource-group my-rg --name agent-tool-server \
    --container chio-sidecar --command "/bin/sh"

Worked example

Deploy from a new resource group:

# Create Log Analytics workspace and managed environment.
$ az monitor log-analytics workspace create --resource-group my-rg \
    --workspace-name chio-law
$ LAW_ID=$(az monitor log-analytics workspace show --resource-group my-rg \
    --workspace-name chio-law --query customerId -o tsv)
$ LAW_KEY=$(az monitor log-analytics workspace get-shared-keys \
    --resource-group my-rg --workspace-name chio-law \
    --query primarySharedKey -o tsv)
$ az containerapp env create --resource-group my-rg --name my-env \
    --location eastus --logs-workspace-id "$LAW_ID" --logs-workspace-key "$LAW_KEY"
$ ENV_ID=$(az containerapp env show --resource-group my-rg --name my-env \
    --query id -o tsv)

# Register the read-only OpenAPI spec share on the environment.
$ az containerapp env storage set --resource-group my-rg --name my-env \
    --storage-name chio-spec --azure-file-account-name mystorageacct \
    --azure-file-account-key "$STORAGE_KEY" --azure-file-share-name chio-spec \
    --access-mode ReadOnly
$ SPEC_STORAGE=chio-spec

# Create Key Vault, the authority-seed secret, and managed identity
# (see Secrets section). Then deploy.
$ az deployment group create --resource-group my-rg \
    --template-file deploy/azure/container-app.bicep \
    --parameters location=eastus managedEnvironmentId="$ENV_ID" \
        userAssignedIdentityId="$MI_ID" \
        chioAuthoritySeedSecretUri="$SEED_URI" \
        specStorageName="$SPEC_STORAGE"

# Capture the FQDN.
$ FQDN=$(az deployment group show --resource-group my-rg --name container-app \
    --query 'properties.outputs.containerAppFqdn.value' -o tsv)
$ echo "$FQDN"
agent-tool-server.calmplant-d3a1b2c3.eastus.azurecontainerapps.io

# Verify the sidecar is the front door. Because this reference runs
# --allow-ephemeral-receipts, the receipt backend reports "ephemeral".
$ curl -fsS "https://$FQDN/chio/health" | jq
{ "status": "healthy", "version": "0.1.0", "receipt_backend": "ephemeral", "revocation_backend": "ephemeral" }

# The app is unreachable except through the kernel.
$ curl -fsS "https://$FQDN/api/search" \
    -H "X-Chio-Capability: $CHIO_CAPABILITY_TOKEN" \
    -H "Content-Type: application/json" -d '{"query":"hello"}'

For other deployment shapes, see Cloud Run and ECS Fargate. For receipt querying and key rotation, see Trust Control Plane.