Troubleshooting

A call was refused

A refusal names the check that stopped it. The kernel runs its checks in a fixed order and stops at the first failure, so the reason locates the stage, and later checks did not run.

// Step 2: subject binding.
if verified.subject_hex != input.request.agent_id {
    // … SubjectMismatch { expected, actual }
    return deny(core_err, None, Some(verified));
}

// Step 3: scope match.
let matched_grant_index = match resolve_matched_grant_index(&verified.scope, input.request) {
    Ok(index) => index,
    Err(error) => return deny(error, None, Some(verified)),
};

// Step 4: guard pipeline.
// … each guard in order; a deny returns GuardDenied, an error GuardError

// Step 5: mutate delegated sibling budget only after every deny-capable
// local check has passed.
if let Err(error) = admit_delegated_budget(input.capability, budgets) {
    let core_err = KernelCoreError::InvalidCapability(error);
    return deny(core_err, Some(matched_grant_index), Some(verified));
}

Outside the grant

00:09  refused    read /secrets/.env: outside the grant

The grant names no resource that covers the request, so the scope check refused it before any rule ran. No policy change can fix a scope refusal. If the agent should never read the file, the refusal is the intended outcome. If the task needs it, reissue the grant on a new fork and name the narrowest path that serves the task.

const next = await project.fork();
const widened = authority.reissue({
  on: next,
  resources: {
    "/workspace": ["read", "write"],
    "/models/default": ["invoke"],
    "/secrets/ci.env": ["read"],
  },
});

next_fork = project.fork()
widened = authority.reissue(
    on=next_fork,
    resources={
        "/workspace": ["read", "write"],
        "/models/default": ["invoke"],
        "/secrets/ci.env": ["read"],
    },
)

let next = project.fork().await?;
let widened = authority.reissue_with(&next, |grant| grant.resource("/secrets/ci.env", ["read"]))?;

Refused by a rule

00:31  refused    write /workspace/tests/test_pricing.py: rule tests-are-evidence

The request was inside the grant and a policy rule refused it. The receipt's evidence line names the rule and the pattern it matched. Change the request, or change the rule; describe the policy to see every rule in force.

console.log(await project.policy.describe());

print(project.policy.describe())

println!("{}", project.policy().describe().await?);

Grant expired

20:01  refused    invoke /models/default: grant expired

The token's expires_at passed before the kernel evaluated the call. Work done before expiry stands. Reissue the same terms on a new fork with a longer lifetime, and the agent continues from the fork's changes.

const next = await project.fork();
const fresh = authority.reissue({ on: next, expiresIn: "40m" });

next_fork = project.fork()
fresh = authority.reissue(on=next_fork, expires_in="40m")

let next = project.fork().await?;
let fresh = authority.reissue_with(&next, |grant| grant.expires_in("40m"))?;

Budget exhausted

11:42  refused    invoke /models/default: budget exhausted, spent USD 2.00 of 2.00

The kernel holds the cost of each priced call before dispatch, and the hold that would overdraw the grant was refused. Nothing was charged for the refused call. Reissue with a larger budget, or split the task across forks with a budget each.

const next = await project.fork();
const larger = authority.reissue({ on: next, budget: USD(5) });

next_fork = project.fork()
larger = authority.reissue(on=next_fork, budget=USD(5))

let next = project.fork().await?;
let larger = authority.reissue_with(&next, |grant| grant.budget(USD(5)))?;

Call limit reached

58:12  z6MkbY4  quote   sku-118 at 10%        refused   100 calls per hour reached

The caller used every call its grant allows in the window. The limit resets on its own; to raise it, issue the caller a new grant with a higher limits value. The refused call was not charged.

Waiting for approval

00:09  z6MkbY4  quote   sku-301 at 80%        pending   approval required

This call has not been refused. An approval rule matched it, and the kernel parked it, undispatched, until a person signs for that exact request. List what is waiting and approve or deny it.

chio approvals pending
chio approve apr-01ac…5e2

pending    apr-01ac…5e2   z6MkbY4…e1R   quote sku-301 at 80%   waiting 12s
approved   apr-01ac…5e2, signed by did:chio:z6Mkf3…Qp9, binds request 9e1f…c2a0

Delegation chain revoked

00:02  refused    run pytest tests/test_pricing.py::test_refund_window: delegation chain revoked at cap-01d2…2a90

An ancestor of the presented token was revoked, which ends every token delegated from it. Do not retry with the same token. Issue a new grant from a live parent, as in Delegate Between Agents.

HTTP 403 with chio_access_denied

HTTP/1.1 403 Forbidden

{"error":"chio_access_denied","message":"side-effect route requires a capability token","receipt_id":"rcpt-01ad…3c14","suggestion":"provide a valid capability token in the X-Chio-Capability header or chio_capability query parameter"}

The request reached a route that changes state without a capability token. Send the caller's grant in the X-Chio-Capability header, as in Connect an Application. The receipt_id names the signed refusal.

HTTP 401 from the MCP edge

error[CHIO-CLI-CREDENTIAL]: transport authentication was missing, expired, or invalid
  help: send a grant from the serving kernel as the bearer token
  code: urn:chio:error:transport:auth-missing-or-invalid

The edge rejected the session before any tool call. Open the session with the caller's grant as the bearer, ChioClient.withStaticBearer(url, grant), and check that the grant has not expired.

A command failed

A receipt read without a boundary

error[CHIO-KERNEL-RECEIPT-READ-CONTEXT-MISSING]: a local receipt read needs a read boundary
  help: pass --tenant <id> to read one tenant's receipts, or --admin-all to read every tenant as the operator
  code: urn:chio:error:kernel:receipt-read-context-missing

A receipt store can hold several tenants' records, so every local read names whose records it reads. Rerun with the boundary you mean.

chio receipt list --tenant harbor-goods

State directory not writable

error[CHIO-CLI-ADMISSION-STORE-REQUIRED]: durable admission needs a session database, and ~/.chio/state is not writable
  help: make ~/.chio/state writable, or pass --session-db <path>
  code: urn:chio:error:cli:admission-store-required

Serving processes record each admission before dispatch so that a restart cannot repeat it. Fix the directory's permissions, or point the process at a database path it can write.

chmod u+w ~/.chio/state

Evidence fails verification

File hash mismatch

error[CHIO-ATTEST-PROVENANCE-MISSING]: evidence package file hash mismatch for receipts.ndjson
  help: receipts.ndjson changed after the package was exported; verify the package as exported, or export it again
  code: urn:chio:error:attest:provenance-missing

A file in the package no longer matches the manifest, so the verifier stopped before reading a signature. Treat the copy as untrusted and export again from the project.

chio evidence verify evidence/ --kernel 4f1c…a2e9

Signed by an unpinned key

error[CHIO-ATTEST-RECEIPT-VERIFICATION-FAILED]: receipt rcpt-01a8…9c04 is signed by 7e21…0b4f, not the pinned kernel 4f1c…a2e9
  help: reject the evidence, or pin the key the operator published for this kernel
  code: urn:chio:error:attest:receipt-verification-failed

Every record verifies under the key it names, and that key is not the one you pinned. A package re-signed under another key passes every internal check, which is why the pin decides. Confirm the operator's kernel key out of band before you pin a different one.

When a record is missing rather than failing, check that the query names the store and tenant the original call used, with the receipt ID from its response. The query cookbook covers each filter.

A partner will not connect

Partner key does not match its pin

error[CHIO-FEDERATION-PEER-KEY-MISMATCH]: partner presented kernel key 7e21…0b4f; its card pins 9b07…d31c
  help: verify the partner's new card out of band, then add it again
  code: urn:chio:error:federation:peer-key-mismatch

The handshake authenticated a key other than the one you pinned, so the kernel refused the connection before any work crossed. A partner that rotated its kernel key sends a new card. Confirm the new key with the partner directly, then replace the pin.

chio partner add halden-card.json --name partner --transport iroh

partner    did:chio:z6MkpA7…c2T, named partner
kernel     4b6e…19a0, pinned from its card, replaces 9b07…d31c

A connection that succeeds and a request that is still refused can both be correct: the treaty and the receiving kernel's policy decide each request. Federation health checks separate transport, identity, and admission.

Work stopped without an error

A worker that dies does not erase its process. The durable process keeps its identity, last checkpoint, and open calls, and a replacement worker resumes from them. Inspect the process in the incident runbook and recover open calls with their existing identity through Call Recovery. An external effect whose outcome is unknown is resolved through Recover an Operation, never by sending it again.

When the text is not listed

Look up the code from the code: line, then place the failure in the order the kernel checks a call.