BuildIntegrated Examples
Internet of Agents: Incident Network
A commander delegates through six narrowing capability hops, then a provider runs one bounded operation through Chio.
Where the code lives
examples/internet-of-agents-incident-network/. Set OPENAI_API_KEY or ANTHROPIC_API_KEY for a live agent loop, or leave both unset for the deterministic fallback used in CI.The smoke stops at the second lineage record
./smoke.sh boots trust-control, four MCP edges, the broker, the coordinator and the executor, issues the root capability, and records its lineage. The second call fails: orchestrate.py:201 (trust.record_lineage(triage_cap, root_cap["id"])) gets HTTP 500 from POST /v1/lineage with capability lineage inc-...-triage contains a signed token with an invalid signature, and the run stops there. Delegated capabilities on this path are built and signed client-side by delegate(), and trust-control rejects them. Everything past Phase 3 on this page is therefore read from the code rather than from a completed run, and is marked where it matters.What It Shows
- Six chained capability hops: root commander, triage, change, vendor-liaison, provider coordinator, provider executor.
- Each hop is a delegation that narrows
grants,maxTotalCost,maxInvocations, andttl. - Cross-org dispatch through an ACP broker, with the provider coordinator forwarding into a bounded executor.
- Receipts and lineage chains for each capability, queryable through trust-control after the run.
- Five scenarios in one binary: happy path, attenuation deny, mid-chain revocation, approval required, capability TTL expiry.
Architecture
Service Inventory
smoke.sh assigns free ports to the local processes:
| Process | Command | Role |
|---|---|---|
| trust-control | chio trust serve | Capability authority, revocation, receipt store, budget store. |
| mcp-observability | chio mcp serve-http + tools/observability.py | Read-only triage tools (incident summary, spans, deploy timeline, SLO). |
| mcp-github | chio mcp serve-http + tools/github.py | Search commits, get diff, get file. |
| mcp-pagerduty | chio mcp serve-http + tools/pagerduty.py | On-call state, escalation timeline. |
| mcp-provider-ops | chio mcp serve-http + tools/provider_ops.py | Disable edge rule (write tool, costed). |
| acp-broker | services/acp_broker.py | Cross-org task queue. |
| coordinator | services/coordinator.py | Provider entry point. Receives the ACP task and dispatches the executor. |
| executor | services/executor.py | Bounded operation runner. Validates the executor capability against trust-control. |
| coordinator sidecar | chio api protect | Receipt-signing front for the coordinator API. |
Run It
# From the chio workspace root
cargo build --bin chio
cd examples/internet-of-agents-incident-network
./smoke.sh
# Optional: live agent reasoning
export OPENAI_API_KEY=...
./smoke.sh
# Run a non-default scenario via orchestrate.py --mode
./scenario/01-happy-path.sh
./scenario/02-attenuation-deny.sh
./scenario/03-revoke-midchain.sh
./scenario/04-approval-required.sh
./scenario/05-expiry-async-failure.shPhase 1: Incident Trigger
The orchestrator loads a fixture incident from workspaces/customer-lab/incident/current-incident.json and persists it as the run header. It also generates an Ed25519 identity for each agent in the chain.
incident = json.loads(
(ROOT / "workspaces" / "customer-lab" / "incident" / "current-incident.json").read_text()
)
ids = {n: gen_identity(n) for n in [
"commander-agent", "triage-agent", "change-agent",
"vendor-liaison-agent", "provider-coordinator", "provider-executor",
]}Phase 2: Root Capability
Trust-control issues a root capability bound to the commander agent's public key. Read-only investigation tools are free. Cross-org engagement (acp-broker.create_task) carries a $10 budget at $5 per call.
root_cap = trust.issue_capability(
ids["commander-agent"].pk,
_scope(
_grant("mcp-observability", "get_incident_summary", ["invoke", "delegate"]),
_grant("mcp-observability", "query_spans", ["invoke", "delegate"]),
_grant("mcp-observability", "get_deploy_timeline", ["invoke", "delegate"]),
_grant("mcp-observability", "get_slo_status", ["invoke", "delegate"]),
_grant("mcp-github", "search_commits", ["invoke", "delegate"]),
_grant("mcp-github", "get_diff", ["invoke", "delegate"]),
_grant("mcp-github", "get_file", ["invoke", "delegate"]),
_grant("mcp-pagerduty", "get_oncall_state", ["invoke", "delegate"]),
_grant("mcp-pagerduty", "get_escalation_timeline", ["invoke", "delegate"]),
_grant("acp-broker", "create_task", ["invoke", "delegate"],
max_cost=1000, max_per_call=500),
),
ttl=1800,
)
trust.record_lineage(root_cap, None)Phase 3: Sub-Agent Delegation
The commander mints three narrowed delegations. The triage agent gets read-only investigation grants under ["invoke"] only (no further delegation). The change agent gets an empty scope (it reasons over the triage output, no tool calls). The vendor liaison gets a single grant: open one ACP task, $5 budget.
triage_cap = delegate(
parent=root_cap, delegator=ids["commander-agent"], delegatee=ids["triage-agent"],
scope=_scope(
_grant("mcp-observability", "get_incident_summary", ["invoke"]),
_grant("mcp-observability", "query_spans", ["invoke"]),
# ... investigation grants without "delegate" ...
),
ttl=900, cap_id=f"{incident['incident_id']}-triage",
)
vendor_cap = delegate(
parent=root_cap, delegator=ids["commander-agent"], delegatee=ids["vendor-liaison-agent"],
scope=_scope(_grant("acp-broker", "create_task", ["invoke", "delegate"],
max_cost=500, max_per_call=500)),
ttl=vendor_ttl, cap_id=f"{incident['incident_id']}-vendor",
)The root capability is issued by trust-control with ten grants, one per tool the commander may reach, and a 30-minute TTL. Only the acp-broker/create_task grant asks for money: max_cost=1000 and max_per_call=500 at orchestrate.py:177-178. No grant anywhere in this example sets an invocation cap.
{
"schema": "chio.capability.v1",
"id": "cap-01a07189-5280-7ca1-acea-66b0ce120347",
"issuer": "210611cd27f2e29bd71150dc258b79afbecb11b6d5b987fde31bd5264c99b201",
"subject": "a07f105a5b2c528ffba4362c6f6a5172e7fd3bb8e12c858e5b8fe6cd1ee4f577",
"scope": {
"grants": [
{
"server_id": "acp-broker",
"tool_name": "create_task",
"operations": ["invoke", "delegate"]
}
]
},
"issued_at": 1788611220,
"expires_at": 1788613020,
"signature": "b55460285c721115730b4b9255153b449e41be26c5c8149e88d5eff04fe007e0..."
}The budget envelope does not survive issuance
orchestrate.py builds each grant with camelCase keys, maxInvocations, maxTotalCost and maxCostPerInvocation (orchestrate.py:67-73), and posts them to /v1/capabilities/issue. The wire keys ToolGrant actually deserializes are snake_case: max_invocations, max_total_cost and max_cost_per_invocation (crates/core/chio-core-types/src/capability/scope.rs:94-118), and the type sets neither rename_all nor deny_unknown_fields, so the camelCase keys are dropped without an error. The block above is what actually came back: three keys per grant and no cost fields at all. Read the ceilings on this page as what the example intends, not as what the issued token carries.Capabilities here carry no ttl_secs, delegation_depth or parent_capability_id field. Lifetime is the issued_at / expires_at pair, depth is the length of delegation_chain, and the parent is that chain's last capability_id. Child ids are built from the incident id, so they read inc-meridian-inference-gw-2026-04-15-0317z-triage, -change and -vendor (orchestrate.py:198,205,215; the incident id comes from workspaces/customer-lab/incident/current-incident.json).
Delegated capabilities are built and signed client-side by the delegate() helper, so widening is caught at request time rather than at issue time: when a capability presents a scope broader than its chain allows, the request is refused with the reason attenuation_violation (exercised by scenario/02-attenuation-deny.sh). That refusal is the provider-side executor, not the kernel: attenuation_violation has zero occurrences under crates/ and one in services/executor.py:130.
Phase 4: Investigation
The triage agent runs a tool-use loop against the chio MCP edges (observability, github, pagerduty). The change agent reasons over the triage output. The commander composes both into a decision. The vendor liaison decides whether to escalate to Stratos.
The edges do not log per call
chio mcp serve-http writes one line when it comes up, remote MCP edge listening on http://127.0.0.1:<port>/mcp, and nothing per tools/call. Read the outcome of a call from the receipt store or from the bundle under agents/, not from the edge logs. Receipt ids are 64 lowercase hex characters with no prefix (crates/platform/chio-http-core/src/receipt.rs:343-353).If the live agent path is enabled, the model returns structured output naming the suspected rule. The deterministic fallback wires the same shape from the fixture data.
Phase 5: Cross-Org Dispatch
The orchestrator posts a task to the ACP broker. The broker hands it to the provider coordinator. The coordinator delegates a narrowed two-call capability to the executor and forwards the operation request:
coord_cap = delegate(
parent=vendor_cap, delegator=ids["vendor-liaison-agent"],
delegatee=PublicKey(name="provider-coordinator", pk=ids["provider-coordinator"].pk),
scope=_scope(_grant("provider-ops", "disable_edge_rule", ["invoke", "delegate"],
max_cost=500, max_per_call=500, max_invocations=2)),
ttl=600, cap_id=f"{task['task_id']}-provider-coordinator",
)At this point three capabilities sit on the chain in trust-control: commander <- vendor-liaison <- coordinator <- executor. Each child capability has a narrower scope. trust.record_lineage(child, parent_id) makes the chain queryable.
The vendor-liaison agent sends an ACP task to the broker. The broker forwards a coordinator-shaped request into the provider coordinator endpoint:
{
"task_id": "acp-task-4f2c1e90b7a3",
"incident_id": "inc-meridian-inference-gw-2026-04-15-0317z",
"target_service": "inference-gateway",
"target_rule": "geo-restrict-v42",
"bounded_action": "disable_rule",
"provider_instructions": "Disable the geo-restrict rule for the affected region.",
"vendor_liaison_capability": { "id": "inc-meridian-inference-gw-2026-04-15-0317z-vendor", "...": "..." },
"execution_deadline": 1776273675
}{
"task": {
"task_id": "acp-task-4f2c1e90b7a3",
"target_service": "inference-gateway",
"target_rule": "geo-restrict-v42",
"bounded_action": "disable_rule",
"provider_coordinator_capability": {
"id": "acp-task-4f2c1e90b7a3-provider-coordinator",
"delegation_depth": 2,
"parent_capability_id": "inc-meridian-inference-gw-2026-04-15-0317z-vendor"
}
},
"provider_coordinator_seed_hex": "...",
"provider_executor_public_key": "executor-pk-...",
"control_url": "http://127.0.0.1:<TRUST_PORT>",
"service_token": "demo-token",
"provider_executor_url": "http://127.0.0.1:<EXEC_PORT>",
"executor_ttl_seconds": 600
}Phase 6: Bounded Execution
The executor calls disable_edge_rule on mcp-provider-ops, presents its capability to chio, and the edge admits the call. The provider workspace state at workspaces/provider-lab/tenants/MeridianLabs/services/inference-gateway.json flips to reflect the disabled rule. An audit-log entry lands at workspaces/provider-lab/operations/audit-log.json.
The orchestrator marks the ACP task complete and pulls back the finalized record.
Phase 7: Audit
After execution, the orchestrator retrieves capability lineage and writes financial reports:
for label, cid in cap_ids.items():
_write(out / "lineage" / f"{label}-chain.json", trust.lineage_chain(cid))
budget_state = trust.query_budgets()
_write(out / "financial" / "budget-usage.json", budget_state)
exposure = trust.exposure_ledger(agent_subject=executor_pk)
_write(out / "financial" / "exposure-ledger.json", exposure)
scorecard = trust.credit_scorecard(agent_subject=executor_pk)
_write(out / "financial" / "credit-scorecard.json", scorecard)
settlements = trust.settlement_report()
_write(out / "financial" / "settlement-report.json", settlements)Scenarios
The orchestrator supports five scenarios through --mode:
| Mode | What it tests | Driver |
|---|---|---|
happy-path | Full six-hop delegation lands the fix. | scenario/01-happy-path.sh |
attenuation-deny | Executor asks for a broader rule than the chain allows. Edge denies. | scenario/02-attenuation-deny.sh |
revoke-midchain | Upstream cap is revoked while the executor is mid-call. Revocation propagates. | scenario/03-revoke-midchain.sh |
approval-required | Broader rollback needs a signed approval before dispatch. | scenario/04-approval-required.sh |
expiry-async-failure | Vendor cap TTL is set to 2s. The chain expires before the executor runs. | scenario/05-expiry-async-failure.sh |
Capability Lineage Query
After the run, trust-control stores a chain for each capability ID. The example retrieves each chain over the trust-control HTTP API ( GET /v1/lineage/{capability_id}/chain, wrapped as trust.lineage_chain(cid) in the orchestrator). The response is a bare JSON array, not an object with a chain key, and the id field is capability_id (crates/platform/chio-control-plane/src/trust_control/service_types/responses.rs:104-122), so filter it like this:
curl -sS -H "Authorization: Bearer $SERVICE_TOKEN" \
"$CONTROL_URL/v1/lineage/$CAP_ID/chain" \
| jq 'map({capability_id, delegation_depth, parent_capability_id})'The chain walks upward from the id you asked about, so it is linear: get_combined_delegation_chain (crates/platform/chio-store-sqlite/src/receipt_store/bootstrap/federated.rs:738-775) returns the queried capability and its ancestors, and delegation_depth is that entry's position in the returned list. Asking about the root returns one entry. Asking about the executor returns the four hops above it. There is no query that returns siblings, so a chain never holds two entries at the same depth.
[
{ "capability_id": "cap-01a07189-5280-7ca1-acea-66b0ce120347",
"delegation_depth": 0, "parent_capability_id": null },
{ "capability_id": "inc-meridian-inference-gw-2026-04-15-0317z-vendor",
"delegation_depth": 1,
"parent_capability_id": "cap-01a07189-5280-7ca1-acea-66b0ce120347" },
{ "capability_id": "acp-task-4f2c1e90b7a3-provider-coordinator",
"delegation_depth": 2,
"parent_capability_id": "inc-meridian-inference-gw-2026-04-15-0317z-vendor" },
{ "capability_id": "acp-task-4f2c1e90b7a3-provider-executor",
"delegation_depth": 3,
"parent_capability_id": "acp-task-4f2c1e90b7a3-provider-coordinator" }
]Each entry carries more than the three keys above: subject_key, issuer_key, issued_at, expires_at, grants_json, provenance, snapshot_delegation_depth, snapshot_parent_capability_id, and optionally federated_parent_capability_id and the full signed_capability.
Mid-Chain Revocation
The revoke-midchain scenario revokes the vendor-liaison cap after the executor cap has been issued but before the executor calls disable_edge_rule. The executor presents its own valid cap, and the check that catches this is the provider-side executor, not the kernel: services/executor.py:99-106 loops over delegation_chain plus the capability's own id and asks trust-control is_revoked for each. The response carries camelCase keys and no timestamp:
{
"capabilityId": "inc-meridian-inference-gw-2026-04-15-0317z-vendor",
"revoked": true,
"newlyRevoked": true
}{
"execution": {
"verdict": "deny",
"reason": "revoked_ancestor",
"executor_capability_id": "acp-task-4f2c1e90b7a3-provider-executor",
"revoked_capability_id": "inc-meridian-inference-gw-2026-04-15-0317z-vendor",
"requested_service": "inference-gateway",
"requested_rule": "geo-restrict-v42"
}
}The downstream call is denied, and the deny body names which ancestor was revoked. It carries no delegation depth: the executor walks the chain by id and returns the first revoked one it finds (services/executor.py:99-106). No receipt is written on this path either; the denial is the HTTP response.
Smoke Assertions
The smoke runs the in-tree verifier after the orchestrator. The verifier checks required output files, cross-checks each file against SHA-256 hashes in bundle-manifest.json, verifies each capability's Ed25519 signature and its delegation-chain lineage (parent linkage, delegator/delegatee consistency, and expiry contained within the parent), and asserts the scenario-specific operational verdicts:
uv run --project "${EXAMPLE_ROOT}" python -c "
import sys; sys.path.insert(0, '${EXAMPLE_ROOT}')
from incident_network.verify import verify_bundle
import json
r = verify_bundle('${ARTIFACT_ROOT}')
json.dump(r, open('${ARTIFACT_ROOT}/review-result.json', 'w'), indent=2)
assert r['ok'], r['errors']
"
printf 'internet-of-agents-incident-network smoke passed\n'
printf 'artifacts: %s\n' "${ARTIFACT_ROOT}"Inspect After
cd artifacts/live/<timestamp>
export TRUST_DB="$(pwd)/state/trust-receipts.sqlite3"
export REVOKE_DB="$(pwd)/state/trust-revocations.sqlite3"
export BUDGET_DB="$(pwd)/state/trust-budgets.sqlite3"
# Receipts grouped by tool and verdict
sqlite3 "$TRUST_DB" \
'select tool_server, tool_name, decision_kind, count(*) from chio_tool_receipts group by tool_server, tool_name, decision_kind;'
# Revocation table after a revoke-midchain run
sqlite3 "$REVOKE_DB" \
'select capability_id, revoked_at from revoked_capabilities;'
# Budget consumption per grant
sqlite3 "$BUDGET_DB" \
'select capability_id, grant_index, invocation_count, total_cost_exposed, total_cost_realized_spend from capability_grant_budgets;'
# Lineage projection (one file per capability)
ls lineage/
jq 'length' lineage/provider-executor-chain.json
# 4 -- root, vendor, coordinator, executor (a bare array, not {chain: [...]})
# Final ACP task state
cat acp/task-final.json | jq '{status, completed_at}'Federation Note
The default smoke runs the Chio components in one process tree and one trust-control. The cross-org boundary is enforced by the ACP broker and the chained capability scope, not by separate trust domains. To run this example with two separate trust-control instances, see Bootstrap Federated Trust for the handshake and Bilateral Federation for the receipt-pair semantics across federated kernels.
Inspect the Bundle
cd artifacts/live/<timestamp>
cat incident.json
cat identities/public-identities.json
# One file per capability on the chain
ls capabilities/
cat capabilities/root-commander.json
cat capabilities/triage-agent.json
cat capabilities/provider-executor.json
# One lineage chain per capability
ls lineage/
cat lineage/provider-executor-chain.json
# Agent decisions
cat agents/triage-output.json
cat agents/commander-output.json
cat agents/vendor-liaison-output.json
# ACP task lifecycle
cat acp/task-created.json
cat acp/task-final.json
cat provider/process-task-response.json
# Financial reports from trust-control
cat financial/budget-usage.json
cat financial/exposure-ledger.json
cat financial/credit-scorecard.json
cat financial/settlement-report.json
# Manifest hash list for offline verification
cat bundle-manifest.json
cat summary.jsonThe smoke runs incident_network.verify.verify_bundle at the end. It rehashes each file listed in bundle-manifest.json, verifies each capability signature and delegation-chain lineage, and checks the scenario-specific verdicts (for the happy path: the commander engaged the external provider, the ACP task completed, and the execution was allowed). Output lands in review-result.json.
Use this example when...
Where to read more