Mutation Testing

What mutation testing does

For each expression in the examined source, cargo-mutants generates a mutant: one mechanical edit such as flipping a comparison operator, swapping a boolean operator, deleting a negation, or replacing a function body with a type-appropriate stub. It rebuilds the crate with that one edit applied and reruns cargo test --workspace --exclude chio-cpp-kernel-ffi against it, the additional_cargo_test_args set in .cargo/mutants.toml (the C++ FFI crate ships through a separate vcpkg / conan lane the default workspace test run does not exercise).

Each mutant has one of four outcomes, recorded verbatim in outcomes.json and read by downstream script:

• CaughtMutant · a test failed. The mutant is killed; the suite noticed.
• MissedMutant · the test suite still passed. The mutant survived: a test gap.
• Timeout · the mutated build exceeded timeout_multiplier (3x the baseline, floored at minimum_test_timeout = 60 seconds). Scored as a survivor, the same as a miss.
• Unviable · the mutated code does not even compile. Excluded from scoring: scoreable = total - unviable, and the kill rate is caught / scoreable.

scripts/mutants-comment.sh buckets survivors by shape for its pull-request summary: deleted code, a swapped comparison or boolean operator, a deleted negation, a boolean or Result return stubbed to a fixed value, a constructed value replaced with Default::default(), or other. The script calls the mapping intentionally heuristic, since cargo-mutants exposes readable replacement text rather than a stable class enum across versions, but the shapes above are the actual edits a mutant can be.

How it runs

The lane lives in .github/workflows/mutants.yml. Its trigger block decides when mutation testing runs, and it fires on three events:

on:
  # PR runs are scoped to trust-boundary source and mutation-control changes.
  # Untouched matrix packages stop before installing Rust or cargo-mutants.
  pull_request:
    paths:
      - ".cargo/mutants.toml"
      - ".github/workflows/mutants.yml"
      - "crates/kernel/chio-kernel-core/mutants.toml"
      - "crates/kernel/chio-kernel-core/src/**"
      - "crates/guards/chio-guards/mutants.toml"
      - "crates/guards/chio-guards/src/**"
      - "crates/guards/chio-policy/mutants.toml"
      - "crates/guards/chio-policy/src/**"
      - "crates/trust/chio-attest-verify/mutants.toml"
      - "crates/trust/chio-attest-verify/src/**"
      - "crates/trust/chio-credentials/mutants.toml"
      - "crates/trust/chio-credentials/src/**"
      - "crates/economy/chio-anchor/mutants.toml"
      - "crates/economy/chio-anchor/src/**"
      - "releases.toml"
      - "scripts/check-fuzz-budget.sh"
      - "scripts/check-mutants-rationale.sh"
      - "scripts/mutants-*.sh"
  schedule:
    # 05:00 UTC nightly. Sits after fuzz.yml (03:23) and the existing
    # nightly proptest lane (04:23) so the GHA runner pool is not
    # over-subscribed.
    - cron: "0 5 * * *"
  workflow_dispatch:
    inputs:
      package:
        description: "Force a single crate (default: full matrix)"
        required: false
        type: string

The pull_request trigger is path-scoped. A pull request that touches none of those paths never starts the lane, and inside a run that does start, a matrix package whose own crate path and mutation-control files are untouched exits before installing Rust or cargo-mutants. The job it gates is mutants-pr, which runs cargo mutants --config <crate>/mutants.toml --package <crate> --test-workspace false --in-diff <diff> --no-shuffle --timeout 300 --jobs 2 --json against a diff taken from the merge base with the pull request's base ref. It carries no continue-on-error, so the exit code of scripts/mutants-gate.sh propagates and decides whether the lane is advisory or blocking.

mutants-nightly fires at 05:00 UTC, after the fuzz nightly (03:23) and the proptest nightly (04:23), sweeping six crates in parallel with cargo mutants --package <crate> --no-shuffle --timeout 600 --jobs 2 --json under a 240-minute per-crate job timeout. No --in-diff and no --config: this is a complete sweep of the modules the workspace-root config examines. The job carries continue-on-error: true (it never blocks any check) and uploads mutants-out/<package> with a 30-day retention. Both jobs share the 1,800 GHA-runner-minute, 30-day cap with the fuzz lanes: the pull-request job treats the cap as a hard halt, the nightly job runs the budget check in warn mode so measurement keeps flowing.

A companion workflow, mutants-fuzz-cocoverage.yml, runs at 05:45 UTC on a four-crate subset, re-deriving a fresh surviving-mutant set and replaying the accumulated libFuzzer corpus against each survivor: a different, adversarially generated oracle that sometimes catches mutants that the unit-test suite missed. The source documentation estimates a 5 to 15 percent cross-oracle reduction in missed mutants. Its findings are advisory, and its setup and budget gates fail closed.

What it measures

examine_globs in .cargo/mutants.toml scopes mutation to the trust-boundary modules of six crates. Generated mutants therefore affect modules that participate in decisions, rather than pure data or generated code:

• chio-kernel-core · the pure-compute verdict path (evaluate.rs, capability_verify.rs, scope.rs, receipts.rs, passport_verify.rs, guard.rs, normalized.rs).
• chio-policy · the HushSpec evaluator, compiler tree, conditions, detection, merge, resolve, validate, regex safety, and receipts.
• chio-guards · the fail-closed pipeline plus the boundary-enforcing guard set: shell command, forbidden and allowlisted paths, egress and internal-network filters, secret leak, patch integrity, prompt injection, jailbreak detection, data flow, behavioral sequence, and the code-execution, browser-automation, computer-use and remote-desktop guards. The advisory guards, the pure types, and the external adapter tree are excluded.
• chio-credentials · lib.rs and trust_tier.rs: JWT VC, SD-JWT, OID4VCI, and OID4VP.
• chio-attest-verify · the Sigstore bundle parser and certificate-chain verifier.
• chio-anchor · anchor publication, discovery, and verification, including the Bitcoin, EVM, and Solana adapters.

A discovery quirk shapes two of those entries: cargo-mutants walks mod declarations, not include! macros, so chio-credentials/src/lib.rs (which pulls in thirteen trust-boundary files this way) and chio-policy/src/evaluate.rs (five more) are scoped at the umbrella file, which cargo-mutants can inspect; mutating it reaches its included files. Code outside examine_globs (platform adapters, the Kani harness files, tests, benches, fuzz entry points) is either covered by another lane or verification scaffolding, and each exclude_globs entry carries a rationale: comment. scripts/check-mutants-rationale.sh fails closed without one, and both jobs run it before cargo-mutants.

The committed baseline

The baseline at docs/fuzzing/trust-boundary-mutants-baseline.toml is the source for the kill-rate figure the rest of the documentation quotes. It is the whole file:

generated_at = "2026-04-29"
tool = "cargo-mutants"
tool_version = "25.3.1"

[aggregate]
scope = "six-crate trust-boundary mutation baseline"
crate_entries = 6
listed_mutants_total = 2369
evaluated_mutants_total = 442
caught_total = 115
missed_total = 259
unviable_total = 67
timeout_total = 1
measured_kill_rate_excluding_unviable = 30.7
baseline_status = "aggregate complete from existing per-crate baselines; mixed full sweeps and bounded shards"

Read the arithmetic straight through: scoreable = 442 - 67 = 375 evaluated minus unviable, and 115 / 375 = 30.7% caught. listed_mutants_total is 2,369: the six-crate examined code generates more mutants than have been scored, which is what baseline_status means by a mixed aggregate of complete sweeps and bounded shards rather than one full-workspace run. Each mutant reruns the entire test suite inside a four-hour per-crate budget, so the sweep is bounded by wall clock rather than by the mutant count.

This is the figure Formal Assurance cites: the mutation-testing kill rate is about 30 percent, below the 80 percent target. The rate measures how sensitive the test suite is to mechanical edits in the examined modules. It is empirical evidence about test strength, not a proof of anything, and a killed mutant says a test failed rather than that the edit was a real defect. The separate Rust proof-mutation and specification-mutation campaigns quoted in the source README.md score a different corpus and do not restate this number.

Whether this can ever fail a build is governed by the [mutants] section of releases.toml, which carries both the thresholds and the evidence recorded against them:

[mutants]
# The release floor is 80 percent per crate. Proof and specification mutation
# lanes use rotating samples and remain advisory until a full-inventory
# coverage ratchet is promoted.
initial_merge_tag = "v0.0.0-mutation-gate"
target_catch_ratio_percent = 80
activation_threshold_percent_per_crate = 80
required_consecutive_nightly_successes = 2
observed_consecutive_nightly_successes = 0
activation_evidence = """
status: advisory
evidence_pending: true
gate_mode: advisory
observed_consecutive_nightly_successes: 0
required_consecutive_nightly_successes: 2
nightly_runs: []
per_crate_kill_rate_percent:
  chio-policy: pending_full_sweep
  chio-credentials: pending_full_sweep
  chio-attest-verify: pending_full_sweep
  chio-kernel-core: pending_full_sweep
  chio-guards: pending_full_sweep
  chio-anchor: pending_full_sweep
note: "Blocking activation requires confirmed green mutants-nightly full sweeps and measured per-crate kill rates. Until then cycle_end_tag stays empty."
"""
cycle_end_tag = ""
pr_survivor_issue_budget = 5
nightly_wall_budget_hours_per_crate = 4
trust_boundary_crates = [
    "chio-policy",
    "chio-credentials",
    "chio-attest-verify",
    "chio-kernel-core",
    "chio-guards",
    "chio-anchor",
]

scripts/mutants-gate.sh reads that section. With cycle_end_tag empty it prints posture=advisory verdict=pass and exits 0, and it does the same when the tag is set but observed_consecutive_nightly_successes is below required_consecutive_nightly_successes. It exits 1 only when the tag is non-empty, the evidence streak is met, and the caught ratio is below activation_threshold_percent_per_crate, which is 80 for every crate. The activation_evidence block above records the streak at 0 and every per-crate kill rate as pending_full_sweep, so no per-crate score has been measured against that threshold.

Reproduce

Every command below runs from the root of the Chio source checkout, which is where cargo-mutants finds the workspace-root config.

cargo install cargo-mutants --version '~25' --locked

# Scoped to the lines your branch actually changed. This is the same
# scoping the mutants-pr job applies against the pull request's base ref.
git diff origin/main...HEAD > /tmp/diff.patch
cargo mutants --in-diff /tmp/diff.patch

# Full sweep on one crate. Slow: cargo-mutants rebuilds and reruns the
# whole test suite once per generated mutant.
cargo mutants --package chio-kernel-core

--list stops after mutant generation, so it costs one parse rather than one test run per mutant. It is the cheapest way to see exactly which edits the config produces for a crate, and every line names a file, a position, and the replacement:

$ cargo mutants --list --package chio-kernel-core | head -6

crates/kernel/chio-kernel-core/src/capability_verify.rs:73:9: replace VerifiedCapability::normalized -> Result<NormalizedVerifiedCapability, NormalizationError> with Ok(Default::default())
crates/kernel/chio-kernel-core/src/capability_verify.rs:103:9: replace <impl From<BudgetSplitError> for CapabilityError>::from -> Self with Default::default()
crates/kernel/chio-kernel-core/src/capability_verify.rs:121:5: replace verify_capability -> Result<VerifiedCapability, CapabilityError> with Ok(Default::default())
crates/kernel/chio-kernel-core/src/capability_verify.rs:154:5: replace verify_capability_with_floor -> Result<VerifiedCapability, CapabilityError> with Ok(Default::default())
crates/kernel/chio-kernel-core/src/capability_verify.rs:171:5: replace verify_capability_base -> Result<VerifiedCapability, CapabilityError> with Ok(Default::default())
crates/kernel/chio-kernel-core/src/capability_verify.rs:171:8: delete ! in verify_capability_base

Those six lines are the head of a longer list. The whole set for that one crate is:

$ cargo mutants --list --package chio-kernel-core | wc -l

367

That count is the generated mutants for the seven chio-kernel-core modules the config examines, not for the crate as written. It is also the number of full test-suite runs a complete sweep of that one crate costs, which is why the nightly budget is four hours per crate and why the committed baseline mixes complete sweeps with bounded shards.

Survivor triage

A surviving mutant resolves one of three ways, in order of preference:

• Add or strengthen a test. The default path. The mutant found a real gap; write the test that would have caught it.
• Refactor toward equivalence. If the mutant changes nothing observable, restructure the code so cargo-mutants prunes it, rather than skipping it.
• Skip with a rationale. Add the path to exclude_globs with a rationale: comment naming why the mutant is not a useful signal (equivalent code, generated code, a remote-process bridge better served by integration tests). Two of the six per-crate configs sit under a CODEOWNERS path (crates/kernel/chio-kernel-core/** and crates/trust/chio-attest-verify/**); the workspace-root config is not under one.

On a pull request that reaches the lane, scripts/mutants-comment.sh posts one per-crate comment (mutants, scoreable, caught, survivors, catch ratio, plus the top survivors inline) and edits that same comment on every later push, keyed by a hidden chio-mutants-sticky marker. scripts/mutants-autofile-issue.sh then opens one issue per survivor beyond pr_survivor_issue_budget, titled with the first 16 hex characters of a SHA-256 over package, source file, source line, verdict, and replacement text, and searches open issues for that prefix before filing. Both steps run only for a pull request from the same repository, so a fork's pull request produces the report artifact without the comment or the issues. Survivor triage follows the same reproduce-classify-fix shape as the other formal gates; see Failure Modes for the Lean, Kani, and fuzz entries.

See also

• Formal Assurance · the scope page that reports this lane's kill rate and target.
• Formal Assurance Overview · mutation testing's scope and limits among the assurance methods.
• Kani Harnesses · the bounded-model-checking lane sharing chio-kernel-core.
• Fuzz Infrastructure · cross-replayed against survivors nightly.
• Failure Modes · triage conventions for the other formal gates.
• Arena · a co-evolving adversary harness against the same guard pipeline.