Constant-Time Tests

A harness times one operation on two input classes

A harness labels each input Left or Right, times the chosen operation, and compares the two observed distributions with Welch’s t-statistic. The classes define the experiment: a mismatch at the start of a signature against one at the end, or a covering grant at the start of a parent scope against one at the end.

A large statistic is a signal to investigate. A small one means these classes did not separate with the samples collected. The dudect-bencher documentation describes the method and its boundary.

Three harnesses measure three operations near the authority path

Each harness is a separate test binary behind the dudect Cargo feature. Inputs are generated before the timed closure, so input construction stays outside the measured operation.

• Signature bytes. mac_eq builds two random 64-byte Signature values and flips the first or the last byte of one, so both comparisons return false. The closure returns that Boolean, and the runner’s black_box consumes it. The operation is byte equality through Signature’s PartialEq, separate from HMAC or Ed25519 verification, and Rust’s == leaves the machine-code loop to the compiler, so read the optimized code before attributing a difference to an early exit.
• Covering grant. The scope harness gives the child one tool grant and each parent 16 grants, with the covering entry at one end of the vector; both subset results are true. Ordinary normalized evaluation searches for a covering parent grant, which makes its position the distinction to time. This closure discards the result, so black_box receives unit, and an unused pure computation can be optimized away: check the compiled operation before reading the statistic.
• Credential rejection. The credential harness fixes the issuer key and the clock. Left holds 256 ASCII 0 characters with no separators; Right holds random printable ASCII of the same length, and some Right inputs contain the separators that reach later parsing. The closure discards the Result. A difference here can reflect parsing paths for attacker-controlled input, so establish what is secret and what an attacker observes before reading it as key leakage. The complete credential harness shows the input construction and the exact call.

fn mac_eq_bench(runner: &mut CtRunner, rng: &mut BenchRng) {
    let mut inputs: Vec<(Class, Signature, Signature)> = Vec::with_capacity(SAMPLES_PER_RUN);
    for _ in 0..SAMPLES_PER_RUN {
        if rng.random::<bool>() {
            let (a, b) = signature_pair_differing_at(rng, 0);
            inputs.push((Class::Left, a, b));
        } else {
            let (a, b) = signature_pair_differing_at(rng, 63);
            inputs.push((Class::Right, a, b));
        }
    }

    for (class, a, b) in inputs {
        runner.run_one(class, || {
            // The verdict is always `false` because the inputs differ by
            // construction; we only care about the time the compare takes.
            // Return the comparison result so `run_one`'s `black_box` keeps
            // the compare in the optimized binary; assigning to `_` would
            // let `--release` LLVM eliminate the comparison and leave the
            // harness measuring runner overhead instead of byte equality.
            a == b
        });
    }
}

fn scope_subset_bench(runner: &mut CtRunner, rng: &mut BenchRng) {
    let child = child_scope_matching();

    let mut inputs: Vec<(Class, NormalizedScope)> = Vec::with_capacity(SAMPLES_PER_RUN);
    for _ in 0..SAMPLES_PER_RUN {
        if rng.random::<bool>() {
            inputs.push((Class::Left, parent_scope_with_match_at(0)));
        } else {
            inputs.push((Class::Right, parent_scope_with_match_at(PARENT_FANOUT - 1)));
        }
    }

    for (class, parent) in inputs {
        runner.run_one(class, || {
            // Verdict is always `true` by construction; we are measuring
            // the time the check takes to return that verdict.
            let _ = child.is_subset_of(&parent);
        });
    }
}

The release binary times the three operations on your machine

A timing statistic belongs to one machine, so the release ships the three harnesses compiled in its own build and runs them on the host that will serve calls. The timing suite of chio security selftest runs each harness once with a fresh seed, classifies the largest absolute max t against the 4.5 threshold, and keeps the raw output when you name an evidence directory.

chio security selftest timing --evidence ./timing-evidence

suite    bench          samples  max|t|  threshold  result
timing   mac_eq         0.98M    1.21    4.5        pass
timing   scope_subset   0.97M    2.04    4.5        pass
timing   jwt_verify     0.99M    5.12    4.5        FAIL   repeat on an idle core before investigating
1 suite: 0 passed, 1 failed; evidence in ./timing-evidence

The run above shows a signal on jwt_verify and exits 1. One FAIL on a loaded laptop is a reason to rerun on an idle core with the same binary, then compare the two raw outputs in ./timing-evidence; a repeated FAIL is the signal the boundary section below describes. The evidence directory holds each bench's raw output, its seed, the binary's digest, and a SHA-256 manifest over all of them.

Every release also passes the timing gate on the release builder before it is tagged; Formal Quickstart shows how to check that result with chio release verify --evidence formal. Maintainers run the harnesses and the threshold script from source as described in Qualification: formal gates.

The verdict classifies the largest statistic in the output

Each bench line reports three values. Keep the seed, the raw output, the revision, the compiler, the CPU, and the load conditions with them.

• n. The samples behind the reported statistic, in millions. Cropping can leave fewer than the inputs generated.
• max t. The signed t-value with the largest magnitude among the tested percentile cutoffs.
• max tau. That t-value scaled by the square root of its sample count.

The threshold check, the same in the self-test and in the release gate, takes the largest absolute max t it parses. Its default threshold is 4.5: below it, the check passes; at or above it, the check fails. Output with no recognized measurement is an error rather than a verdict. These are parser verdicts about one output, and four synthetic cases fix their edges:

• max t = +4.49996 passes, because the check keeps the precision that would otherwise round it up.
• max t = -4.5 fails: the absolute value reaches the threshold.
• -7.1 followed by +1.0 fails: the largest absolute value stays 7.1.
• Output with no max t line is an error: there is no measurement to classify.

# dudect-bencher 0.7 stdout shape (relevant lines):
#
#     running 1 bench
#     bench jwt_verify_bench seeded with 0xdeadbeef_cafef00d
#     bench jwt_verify_bench ... : n == +0.040M, max t = +2.13, max tau = ...
#
# The parser extracts every "max t = <signed-float>" occurrence, takes
# the absolute value, tracks the maximum, and compares against THRESHOLD.

verdict() {
    local input="$1"
    local threshold="$2"

    if [[ ! -r "${input}" ]]; then
        err "input file not readable: ${input}"
        exit 2
    fi

    local max_t
    max_t="$(extract_max_abs_t "${input}")"

    if [[ -z "${max_t}" ]]; then
        err "no 'max t = <float>' lines found in ${input}; nothing to verdict"
        exit 2
    fi

    # Numeric compare via awk (avoids bc dep). Returns 1 (FAIL) when
    # max_t >= threshold per the dudect paper's "highly significant" rule.
    local result
    result="$(awk -v a="${max_t}" -v b="${threshold}" 'BEGIN { print (a + 0.0 >= b + 0.0) ? "FAIL" : "PASS" }')"

    printf 'dudect verdict: max|t|=%s threshold=%s result=%s\n' \
        "${max_t}" "${threshold}" "${result}"

    if [[ "${result}" == "FAIL" ]]; then
        exit 1
    fi
    exit 0
}

The scheduled timing workflow runs on a schedule or by manual dispatch. It classifies each harness output, compares FAIL verdicts across two runs by taking the newest completed matching run other than the current one, and opens or updates an issue for a repeated signal. Its measurement and correlation jobs are advisory, and an absent verdict or a missing comparison run stays distinct from a measurement below the threshold.

A timing result covers its classes, its binary, and its machine

Each measured operation sits near an authorization or evidence mechanism and stays distinct from it: signature-byte equality from signature verification, the normalized subset helper from a complete tool call, and chosen invalid JWT strings from secret-dependent signing.

A repeated FAIL warrants investigation: compare equivalent inputs and environments, inspect the generated code, and test one specific explanation. Power and electromagnetic leakage and speculative-execution attacks are outside these harnesses; report a run with no detected difference together with these limits.

Next: ask whether the tests notice a changed decision

Mutation Testing asks a different question of the same trust-boundary code: whether the test suite fails when a decision rule changes.