ECS Fargate

Behind your Application Load Balancer, each Fargate task runs the Chio sidecar as the only target and your application as its upstream on 127.0.0.1:8080. Each admitted call leaves a signed receipt in a SQLite store on the task's own EBS volume.

Both containers share one awsvpc network namespace and one private IP, so the app answers only through the kernel. dependsOn orders startup, an EFS access point mounts the signing seed read-only, and stopTimeout bounds the drain. Each task keeps an independent receipt database, and ECS deletes a service-managed volume when its task terminates.

1. Stage the control token, the spec, and the signing seed

One Secrets Manager entry holds the control token; the spec and the seed live on EFS and mount as files.

# Sidecar control token (bearer token gating admin + /metrics).
aws secretsmanager create-secret --name chio/sidecar-control-token \
    --secret-string "$(openssl rand -hex 32)"

# Stage the OpenAPI spec into the chio-config EFS share and the
# authority seed into the chio-seed EFS access point (DataSync or a
# one-shot helper task), layout:
#   /chio-config/spec/openapi.yaml   -> mounted read-only at /etc/chio
#   <seed access point>/authority.seed -> mounted read-only at /etc/chio/seed

{
    "ARN": "arn:aws:secretsmanager:us-east-1:ACCOUNT_ID:secret:chio/sidecar-control-token-Ab12Cd",
    "Name": "chio/sidecar-control-token",
    "VersionId": "4f0b…9e21"
}

You should see the secret's ARN. The seed is a 64-character hex file that the kernel reads, never key material in a secret environment variable. To keep the token in SSM Parameter Store instead, point valueFrom at the parameter ARN and grant ssm:GetParameters on the execution role.

2. Register the task definition with a launch-attached receipt volume

Replace the account, region, image, EFS, and secret placeholders in the downloaded file with digest-pinned images the task can pull, and rename its logging variable to RUST_LOG, which the sidecar reads. Export EBS_INFRA_ROLE_ARN for the volume configuration.

curl -fsS https://raw.githubusercontent.com/backbay-labs/chio/fe5657020cb669d7f9cce5efcae17ff3bdcde3bb/deploy/ecs/task-definition.json -o task-definition.json
# Fill the account, region, images, EFS, and secret values described below.

jq -n --arg role "$EBS_INFRA_ROLE_ARN" '[{
  name: "chio-receipts",
  managedEBSVolume: {
    roleArn: $role, volumeType: "gp3", sizeInGiB: 20,
    encrypted: true, filesystemType: "ext4"
  }
}]' > volume-configurations.json

TASK_DEFINITION=$(aws ecs register-task-definition \
    --cli-input-json file://task-definition.json \
    --query taskDefinition.taskDefinitionArn --output text)
echo "$TASK_DEFINITION"

arn:aws:ecs:us-east-1:ACCOUNT_ID:task-definition/agent-tool-server:1

You should see the revision ARN. The receipt volume's size, type, and role live in volume-configurations.json, which step 3 passes at service creation.

3. Create the service behind the load balancer and read its health

Export CLUSTER, SUBNET_A, SUBNET_B, TASK_SECURITY_GROUP, TARGET_GROUP_ARN, and the certificate-backed SERVICE_URL.

aws ecs create-service --cluster "$CLUSTER" --service-name agent-tool-server \
    --task-definition "$TASK_DEFINITION" --desired-count 2 --launch-type FARGATE \
    --network-configuration "awsvpcConfiguration={subnets=[$SUBNET_A,$SUBNET_B],securityGroups=[$TASK_SECURITY_GROUP],assignPublicIp=DISABLED}" \
    --volume-configurations file://volume-configurations.json \
    --load-balancers "targetGroupArn=$TARGET_GROUP_ARN,containerName=chio-sidecar,containerPort=9090" \
    --query service.status --output text
aws ecs wait services-stable --cluster "$CLUSTER" --services agent-tool-server
curl -fsS "$SERVICE_URL/chio/health" | jq '{status, receipt_backend}'

ACTIVE
{
  "status": "healthy",
  "receipt_backend": "durable"
}

You should see the service active, then healthy with a durable receipt backend once both tasks are stable. The target group registers chio-sidecar on 9090, never the app on 8080.

4. Send a protected call without a capability, then with one

The task runs the pricing-service image as its app, so POST /v1/quote is the capability-protected route. Export the outside buyer's quote grant from Quickstart part 2 as CHIO_CAPABILITY_TOKEN.

curl -sS -D - -o /dev/null "$SERVICE_URL/v1/quote" \
    -H "Content-Type: application/json" -d '{"sku":"sku-118","percent":10}'
curl -sS -D - -o /dev/null "$SERVICE_URL/v1/quote" \
    -H "X-Chio-Capability: $CHIO_CAPABILITY_TOKEN" \
    -H "Content-Type: application/json" -d '{"sku":"sku-118","percent":10}'

HTTP/1.1 403 Forbidden
content-type: application/json
x-chio-receipt-id: 2d7a…c418

HTTP/1.1 200 OK
content-type: application/json
x-chio-receipt-id: e03f…5b9a

You should see the refusal and the admitted call, each carrying an x-chio-receipt-id. Only the admitted quote reached the pricing handler; look up either receipt in the task's store by its id.

The app waits for HEALTHY, and only /chio/health grants it

The app's dependsOn entry names chio-sidecar with "condition": "HEALTHY", which ECS resolves from the sidecar's container health check:

1. Listen. The sidecar starts and opens its listener on 9090.
2. Check. ECS runs curl /chio/health every 10 seconds and ignores failures during the 15-second startPeriod; a success inside that period marks the container healthy at once.
3. Healthy. The first successful check marks the sidecar HEALTHY.
4. Start. The app becomes eligible to start, and its own probe at the load balancer or runtime takes over.

"condition": "START" waits only for the sidecar process. The check must curl the dependency-aware /chio/health, which returns 503 when the receipt store cannot persist, rather than the process-only /chio/live, or the gate opens while receipts cannot be written. The task runs /usr/bin/curl directly through CMD, so deploy an image that ships that executable and its libraries, since a registry tag may lack a shell, curl, or wget. Keep the entrypoint intact when you change images.

Read more in Sidecar health routes.

Only the load balancer reaches the sidecar, and it checks the same route

Each task gets its own ENI in the subnets you assign. The task security group admits 9090 from the ALB security group only, admits nothing on 8080, and allows egress to the task's dependencies; the EFS security group admits NFS on port 2049 from the task security group.

Point the target group's health check at GET /chio/health on 9090, expecting HTTP 200. A sidecar that can no longer persist receipts then leaves rotation before the container check recycles the task; after 3 consecutive failed container checks, ECS marks the sidecar unhealthy and replaces the task.

stopTimeout moves in the same revision as the hop ceiling

Container dependencies govern shutdown in reverse: ECS stops the app, then sends the sidecar SIGTERM and waits its stopTimeout before SIGKILL. The field is set per container definition, defaults to 30 seconds, and ranges from 2 to 120 on Fargate platform version 1.3.0 or later.

The sidecar runs no generic request timeout; each handler writes its receipt synchronously, and the drain lasts the upstream hop ceiling plus 5 seconds. The default --upstream-timeout-secs 20 asks for 25 seconds, inside the 30-second default with 5 to spare. A higher hop ceiling without a higher stopTimeout severs in-flight calls before they are receipted, and the 120-second cap holds the hop ceiling at 115.

The target group's deregistration_delay.timeout_seconds runs before SIGTERM, so it lengthens deployments without lengthening the drain. Set it a few seconds above the in-flight request budget in place of the 300-second default, so the load balancer stops sending traffic before the task receives the signal.

Read more in Sidecar shutdown.

Each task writes receipts to its own EBS volume

EFS is NFS, and SQLite keeps its WAL index in shared memory that processes on separate hosts cannot share, so --receipt-store stays out of the chio-config and chio-seed mounts. It resolves to /var/lib/chio on the ECS-managed EBS volume, declared "configuredAtLaunch": true in the task definition. roleArn, sizeInGiB, volumeType, and filesystemType go in volumeConfigurations on CreateService or RunTask.

The volume gives each task one local writer, so two tasks hold two databases. ECS deletes service-managed volumes when their tasks terminate, replacement during a deployment included, so export receipts or take a consistent backup first, to a destination that also retains and aggregates them. See the ECS volume lifecycle.

Read more in Receipt store mechanics.

The signing seed arrives as a file that the task role mounts

Every entry in a container's secrets[] becomes an environment variable, so the control token is its only entry. An environment-injected seed is readable from /proc/self/environ by anything sharing the container and is inherited by every process it spawns. The seed comes from the chio-seed access point, read-only at /etc/chio/seed and read by --authority-seed-file, through elasticfilesystem:ClientMount on the task role; the execution role only resolves Secrets Manager.

Read more in Secrets & Signing Keys.

Target tracking adds tasks, and each new task brings its own receipt log

Scale with the service's desiredCount and Application Auto Scaling target tracking; CPU and ALBRequestCountPerTarget are the two predefined metrics to wire first.

# Register the service as a scalable target.
aws application-autoscaling register-scalable-target \
    --service-namespace ecs \
    --resource-id service/$CLUSTER/agent-tool-server \
    --scalable-dimension ecs:service:DesiredCount \
    --min-capacity 2 --max-capacity 50

# Track average CPU at 60% across the service.
aws application-autoscaling put-scaling-policy \
    --service-namespace ecs \
    --resource-id service/$CLUSTER/agent-tool-server \
    --scalable-dimension ecs:service:DesiredCount \
    --policy-name cpu60 --policy-type TargetTrackingScaling \
    --target-tracking-scaling-policy-configuration '{
      "TargetValue": 60.0,
      "PredefinedMetricSpecification": {
        "PredefinedMetricType": "ECSServiceAverageCPUUtilization"
      },
      "ScaleOutCooldown": 60, "ScaleInCooldown": 300
    }'

# Blend Fargate Spot for cost: one on-demand base, the rest on Spot.
aws ecs update-service --cluster "$CLUSTER" --service agent-tool-server \
    --capacity-provider-strategy \
        capacityProvider=FARGATE,weight=1,base=1 \
        capacityProvider=FARGATE_SPOT,weight=4,base=0

Fargate Spot can cut compute cost and can also interrupt a task, so size the drain and the evidence export for that lifecycle. The bill weighs task CPU and memory, EBS capacity, data transfer, image pulls, and log retention; set log and receipt retention independently. See Fargate pricing and Spot capacity behavior.

Both containers log to one CloudWatch group

The task definition routes both containers to /ecs/agent-tool-server with a stream prefix per container.

# Live tail (AWS CLI v2)
aws logs tail /ecs/agent-tool-server --follow --since 5m

# Search recent operational log messages
aws logs start-query \
    --log-group-name /ecs/agent-tool-server \
    --start-time $(( $(date +%s) - 3600 )) \
    --end-time $(date +%s) \
    --query-string 'fields @timestamp, @message
                    | sort @timestamp desc
                    | limit 50'

# CloudWatch log groups never expire by default; set retention.
aws logs put-retention-policy \
    --log-group-name /ecs/agent-tool-server \
    --retention-in-days 30

Scrape the authenticated /metrics endpoint remotely with the sidecar control token, or over loopback from a co-located collector. Trace export needs application instrumentation and a configured exporter.

Read more in Node Observability.

A rollout starts fresh tasks with fresh receipt volumes

Register the prepared task definition and update the service to it, keeping the service's volume configuration in place.

# Deploy a new revision.
aws ecs register-task-definition --cli-input-json file://task-definition.json
aws ecs update-service --cluster "$CLUSTER" --service agent-tool-server \
    --task-definition agent-tool-server --force-new-deployment

# Roll back to revision 42.
aws ecs update-service --cluster "$CLUSTER" --service agent-tool-server \
    --task-definition agent-tool-server:42

Rollback selects a previous task revision and starts replacement tasks; the terminated task's database leaves with its volume. ECS Exec needs a writable root filesystem and a compatible image, which this read-only configuration leaves out.

A stuck task names its cause in the health check or the drain

The app container never starts

The sidecar never reaches HEALTHY. The image lacks /usr/bin/curl, or /chio/health answers 503 because the receipt store failed to open. Check the sidecar's log stream, then the image and the EBS volume configuration.

Calls end without receipts during a deployment

The hop ceiling plus 5 seconds exceeds stopTimeout. Raise stopTimeout in the same revision as --upstream-timeout-secs.

Receipts are missing after tasks were replaced

The replaced tasks' volumes were deleted with them. Export or back up each task's store before termination.

Task definition and IAM reference

"family": "agent-tool-server",
"networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE"],
"cpu": "512",
"memory": "1024",
"executionRoleArn": "arn:aws:iam::ACCOUNT_ID:role/ecsTaskExecutionRole",
"taskRoleArn": "arn:aws:iam::ACCOUNT_ID:role/chio-sidecar-task-role",
"runtimePlatform": {
  "cpuArchitecture": "X86_64",
  "operatingSystemFamily": "LINUX"
},

"name": "app",
"image": "APP_IMAGE_PLACEHOLDER",
"essential": true,
"cpu": 384,
"memory": 896,
"portMappings": [
  { "containerPort": 8080, "hostPort": 8080, "protocol": "tcp" }
],
"environment": [
  { "name": "CHIO_SIDECAR_URL", "value": "http://localhost:9090" }
],
"dependsOn": [
  { "containerName": "chio-sidecar", "condition": "HEALTHY" }
],
"restartPolicy": {
  "enabled": true,
  "ignoredExitCodes": [],
  "restartAttemptPeriod": 60
},
"logConfiguration": {
  "logDriver": "awslogs",
  "options": {
    "awslogs-group": "/ecs/agent-tool-server",
    "awslogs-region": "us-east-1",
    "awslogs-stream-prefix": "app",
    "awslogs-create-group": "true"
  }
}

"name": "chio-sidecar",
"image": "ghcr.io/backbay-labs/chio-sidecar:latest",
"essential": true,
"cpu": 128,
"memory": 128,
"portMappings": [
  { "containerPort": 9090, "hostPort": 9090, "protocol": "tcp" }
],
"dockerLabels": {
  "prometheus.io/path": "/metrics",
  "prometheus.io/port": "9090"
},

"command": [
  "api",
  "protect",
  "--upstream",
  "http://127.0.0.1:8080",
  "--spec",
  "/etc/chio/spec/openapi.yaml",
  "--listen",
  "0.0.0.0:9090",
  "--receipt-store",
  "/var/lib/chio/receipts.db",
  "--authority-seed-file",
  "/etc/chio/seed/authority.seed"
],

"secrets": [
  {
    "name": "CHIO_SIDECAR_CONTROL_TOKEN",
    "valueFrom": "arn:aws:secretsmanager:us-east-1:ACCOUNT_ID:secret:chio/sidecar-control-token"
  }
],

"mountPoints": [
  {
    "sourceVolume": "chio-config",
    "containerPath": "/etc/chio",
    "readOnly": true
  },
  {
    "sourceVolume": "chio-seed",
    "containerPath": "/etc/chio/seed",
    "readOnly": true
  },
  {
    "sourceVolume": "chio-receipts",
    "containerPath": "/var/lib/chio",
    "readOnly": false
  }
],
"healthCheck": {
  "command": ["CMD", "/usr/bin/curl", "-fsS", "http://localhost:9090/chio/health"],
  "interval": 10,
  "timeout": 5,
  "retries": 3,
  "startPeriod": 15
},
"restartPolicy": {
  "enabled": true,
  "ignoredExitCodes": [],
  "restartAttemptPeriod": 60
},
"logConfiguration": {
  "logDriver": "awslogs",
  "options": {
    "awslogs-group": "/ecs/agent-tool-server",
    "awslogs-region": "us-east-1",
    "awslogs-stream-prefix": "chio-sidecar",
    "awslogs-create-group": "true"
  }
},
"readonlyRootFilesystem": true,
"user": "65532:65532"

"volumes": [
  {
    "name": "chio-config",
    "efsVolumeConfiguration": {
      "fileSystemId": "EFS_FILESYSTEM_ID",
      "rootDirectory": "/chio-config",
      "transitEncryption": "ENABLED",
      "authorizationConfig": {
        "iam": "ENABLED"
      }
    }
  },
  {
    "name": "chio-seed",
    "efsVolumeConfiguration": {
      "fileSystemId": "EFS_FILESYSTEM_ID",
      "transitEncryption": "ENABLED",
      "authorizationConfig": {
        "accessPointId": "EFS_SEED_ACCESS_POINT_ID",
        "iam": "ENABLED"
      }
    }
  },
  {
    "name": "chio-receipts",
    "configuredAtLaunch": true
  }
]

Next: export receipts before tasks are replaced