Build federated incident response
Investigate a service failure across three independently governed organizations, approve one bounded repair, and confirm health after the change.
On this page
An incident crosses organizations when the customer sees a failure that only the provider can repair. This application runs customer, provider, and specialist hosts in separate processes against an HTTP service that returns a failure; the customer shares selected telemetry, the specialist proposes a bounded change, and the provider decides whether to authorize that exact repair.
Run the incidentA 403 becomes a 200 only after the provider’s own approval
Three organizations resolve an incident
Customer telemetry, provider configuration, and specialist diagnosis converge on one exact repair. The provider approves it; a separate health check decides whether the incident closes.
Start with the failing regional edge rule: the first health check returns 403. Approve the repair to remove that rule, then watch a separate customer health check observe 200. Disable approval and the failure stays. Enable the backend failure to see an applied change leave the incident open.
Each organization admits work at its own kernel
The customer, the specialist, and the provider each run a kernel. Requests cross between them, and the organization that owns an effect admits it.
The provider authorizes its own repair
The provider prepares a governed intent from the current service, rule, and configuration digest, and approves only that request. A customer investigation grant never widens into provider remediation authority, so a missing approval, revoked authority, or wider repair leaves the rule in place. Approve the exact repair.
The customer shares only the investigation payload
Each organization registers its own tools and owns their effects. The customer’s telemetry tool returns the service, region, and observed status, and raw customer material stays in the customer process. The specialist diagnoses the disclosed condition and returns a proposal, holding no provider remediation grant.
async fn invoke(&self, tool: &str, args: Value) -> Result<Value> {
match (self.config.role.as_str(), tool) {
("customer", "telemetry") => {
let peer = self
.peers
.lock()
.map_err(|_| anyhow::anyhow!("Peer lock failed"))?
.get("provider")
.cloned()
.context("Provider is not enrolled")?;
let response = reqwest::Client::new()
.get(format!("{}/service", peer.url))
.timeout(std::time::Duration::from_secs(5))
.send()
.await?;
let status = response.status().as_u16();
Ok(json!({
"service":"inference-gateway",
"region":"eu-west",
"http_status":status,
"observed_at_ms":now_ms(),
"customer_records":self.config.customer_raw.lines().count(),
"raw_customer_material_shared":false
}))
}
("provider", "inspect") => {
let service = self
.service
.lock()
.map_err(|_| anyhow::anyhow!("Service lock failed"))?;
Ok(json!({"configuration":*service,"configuration_sha256":digest(&*service)?}))
}
("specialist", "diagnose") => {
anyhow::ensure!(
args["telemetry"]["http_status"] == 403
&& args["configuration"]["rule_enabled"] == true,
"This evidence does not support the bounded rule repair"
);
Ok(json!({
"explanation":"The observed EU request is refused while the regional restriction is active.",
"proposal":{
"service":"inference-gateway",
"rule_id":"geo-restrict-v42",
"action":"disable_rule",
"configuration_sha256":args["configuration_sha256"]
},
"shared_input_sha256":digest(&args)?
}))
}
("provider", "repair") => {
let mut service = self
.service
.lock()
.map_err(|_| anyhow::anyhow!("Service lock failed"))?;
let expected = json!({
"service":"inference-gateway",
"rule_id":"geo-restrict-v42",
"action":"disable_rule",
"configuration_sha256":digest(&*service)?
});
anyhow::ensure!(args==expected,"Repair is outside the provider's exact service, rule, action, or configuration version");
anyhow::ensure!(
service["rule_enabled"] == true,
"The rule is already disabled; reconcile the existing repair"
);
let mut next = service.clone();
next["rule_enabled"] = json!(false);
next["version"] = json!(service["version"].as_u64().unwrap_or(0) + 1);
write_json(&self.config.root.join("service.json"), &next)?;
*service = next.clone();
self.repairs.fetch_add(1, Ordering::SeqCst);
Ok(
json!({"changed":true,"configuration":next,"configuration_sha256":digest(&next)?}),
)
}
_ => anyhow::bail!("This organization does not expose that tool"),
}
}
}
struct Node {
config: Config,
passport: Keypair,
host: Host,
tools: Arc<Tools>,
run: Run,
}In the verified run, event 02 discloses service inference-gateway, region eu-west, and status 403.
The provider approves the exact repair it owns
The provider prepares a governed intent from the current service, rule, and configuration digest. Its approval endpoint checks that intent and issues an approval bound to the exact request. A customer investigation grant cannot widen into provider remediation authority; the effect owner decides.
async fn approve(
State(node): State<Arc<Node>>,
headers: HeaderMap,
Json(call): Json<ToolCallRequest>,
) -> Api {
admin(&node, &headers).map_err(error)?;
if node.config.role != "provider" || call.tool_name != "repair" {
return Err(error("Only the provider may approve this remediation"));
}
let service = node
.tools
.service
.lock()
.map_err(|_| error("Service lock failed"))?;
let expected = json!({
"service":"inference-gateway",
"rule_id":"geo-restrict-v42",
"action":"disable_rule",
"configuration_sha256":digest(&*service).map_err(error)?
});
if call.arguments != expected {
return Err(error(
"Approval requires the exact current remediation proposal",
));
}
Ok(Json(
json!({"approval_token":node.host.approve(&call,true,call.capability.expires_at).map_err(error)?}),
))
}Event 04 approves only rule geo-restrict-v42 at configuration version 1.
The provider’s kernel executes the change
The provider authenticates the remote request and submits it to its own Chio kernel. A missing approval, revoked authority, or excessive repair parameters each stop at their own check. The receipt comes from the provider that owns the change, and the caller verifies that identity.
async fn invoke(State(node): State<Arc<Node>>, Json(call): Json<SignedCall>) -> Api {
validate(&node, &call).map_err(error)?;
let result = node
.host
.call_controlled(
&node.run,
&call.sender.to_hex(),
call.request,
None,
Arc::new(AtomicBool::new(false)),
)
.await
.map_err(error)?;
let snapshot = node.run.snapshot().map_err(error)?;
let event = snapshot["events"]
.as_array()
.and_then(|events| {
events
.iter()
.rev()
.find(|event| event["data"]["receipt_id"] == result.receipt_id)
})
.map(|e| e["data"].clone())
.unwrap_or(Value::Null);
Ok(Json(
json!({"allowed":result.allowed,"output":result.output,"receipt_id":result.receipt_id,"evidence":event}),
))
}Event 03 records the refused attempts: missing approval, expanded scope, and revoked authority produced no repair.
Resolution waits for an independent health check
The service endpoint reads the provider’s state. Removing the blocked edge rule changes its response, while a separate backend failure can keep the service unhealthy. The application runs another governed customer health check and marks the incident resolved only when that check succeeds.
async fn service(State(node): State<Arc<Node>>) -> (StatusCode, Json<Value>) {
if node.config.role != "provider" {
return (
StatusCode::NOT_FOUND,
Json(json!({"error":"No service on this host"})),
);
}
let Ok(config) = node.tools.service.lock() else {
return (
StatusCode::INTERNAL_SERVER_ERROR,
Json(json!({"error":"Service state unavailable"})),
);
};
if config["rule_enabled"] == true {
(
StatusCode::FORBIDDEN,
Json(json!({"service":"inference-gateway","error":"region_denied","region":"eu-west"})),
)
} else if config["backend_failed"] == true {
(
StatusCode::SERVICE_UNAVAILABLE,
Json(json!({"service":"inference-gateway","error":"backend_unavailable"})),
)
} else {
(
StatusCode::OK,
Json(json!({
"service":"inference-gateway",
"region":"eu-west",
"answer":"The supplied service accepted and completed the request"
})),
)
}
}Event 05 observes status 200 and closes the incident.
Missing approval, a wider repair, and revoked authority leave the rule in place
The sample input tries the repair three ways before it is approved. The retained run holds the refusals, the approval, and the health check:
./run --run input.json03 incident.refused provider
Missing approval, expanded scope, and revoked authority produced no repair
04 incident.approved provider
Approve only this rule and configuration version
05 incident.postcheck customer
Service accepted the request; close the incident- Withhold approval: the edge rule stays active and the incident stays open.
- Request a wider repair or use revoked authority: no remediation effect occurs.
- Leave the backend broken: the change can apply while the incident stays unresolved.
- Inspect the specialist’s retained record: excluded customer material is absent.
The downloadable tests assert each effect or its absence: run cargo test --locked -p chio-incident-response from the extracted project.
Run the incident on your machine
Keep the run directory to connect the disclosed telemetry, the remediation proposal, the approval, and the observed service health.
cd chio-incident-response
./runThe archive holds the application, the shared host code, tests, sample input, pinned dependencies, and a README. The launcher builds and runs the Rust source included in this archive. Install Rust 1.94.1 and the README’s native prerequisites; filesystem isolation and operator workloads require the listed Linux host profile. Cargo resolves immutable public dependencies, so a sibling checkout is unnecessary.
For model workers, set OPENROUTER_API_KEY on the local host and select model mode; credentials stay on that host and never enter the docs form. The qualified model is openai/gpt-4.1-mini. Analysis, retrieval, and deterministic worker modes run without a model key. Ctrl-C stops the local web process and its active child work and keeps the run directory.
Download identity and source revision
Download complete project · View public source
1bb98580ee193a6bacc2f14fb04ecb19061d6bdc220c3e48603364cea94c65d9Swap in one operational control
Replace the provider’s edge-rule adapter with one operational control and keep the same prepare, approve, execute, and post-check sequence. Credentials stay with the organization that owns the effect. Add another specialist by defining its permitted disclosure and investigation tools before you connect it.