ci: run tests under cargo-nextest to fix the load-dependent deadlock (closes #533)#537
Merged
Conversation
…533) DIAGNOSIS. The intermittent ~50-min CI hang (blocked #520/#521/#523/#524/#525 merges) is the io_uring `fixed_datapath` ring tests deadlocking under parallel load. Evidence: the hang signature was always a "6-test binary that never returns"; `fixed_datapath::tests` is exactly a 6-test module; running the full workspace under nextest reproduced it as 4 of those tests TIMING OUT (hanging) rather than completing. io_uring rings + registered buffers are a limited kernel resource, and running many ring lifecycles concurrently with the rest of the workspace blocks on a submission that never completes. `raft_cluster` (the earlier suspect) passes 5/5 in isolation, consistent with the true cause being resource contention, not that test. FIX (two layers, both in this change): - Switch the `test` and `io-uring` CI jobs to `cargo nextest run --profile ci`, which runs each test in its OWN process (isolating shared-state / port-contention coupling) and enforces a per-test timeout (.config/nextest.toml: SIGKILL after 120s), so any residual hang becomes a fast, named failure instead of burning the runner. `timeout-minutes: 30` caps the whole job as a backstop. nextest does not run doctests, so a `cargo test --doc` step is added. - A nextest test-group pins every io_uring test (`package(=ironcache-runtime) and (io_uring|fixed_datapath|uring)`) to `max-threads = 1`, so ring lifecycles run SERIALLY and never contend -- removing the contention that triggers the hang at the source. Installed via `taiki-e/install-action@nextest` (prebuilt, no compile). Local validation: 2224 non-io_uring tests pass under `nextest --profile ci`; the config + the io-uring group filter parse and target the fixed_datapath tests (io_uring itself is unrunnable in the local sandbox -- PermissionDenied on ring init -- but runs on GitHub's ubuntu runners, where the dedicated io-uring job already proves it green). Closes #533. Supersedes the cancel-and-rerun stopgap. Signed-off-by: Zeke <ezequiel.lares@outlook.com>
…ck cause)
REFINED DIAGNOSIS. Running the workspace under nextest with a per-test timeout revealed the
actual cause of the "50-min CI hang": it was NOT a deadlock and NOT io_uring. It is the Raft
DETERMINISTIC-SIMULATION seed sweeps (ironcache-raft tests::determinism_and_safety_seed_sweep,
failover_split_brain_gate, snapshot_catchup_gate*, migration_*_gate -- a 6-test set matching the
observed "6-test binary" signature). Measured in DEBUG, in isolation, they take 145s / 256s / 107s
EACH; under the parallel `cargo test --workspace` run they were CPU-starved into the tens of minutes,
which looked like a hang. In release they are fast (the whole crate runs in seconds), because the
simulation loop is compute-bound and unoptimized codegen is ~8x slower.
FIX. Optimize ONLY the simulation crates in dev/test builds via
`[profile.dev.package.{ironcache-raft,ironcache-sim,ironcache-cluster,ironcache-store}] opt-level = 3`
(root Cargo.toml). Measured result: the three heaviest DST tests drop from 145/256/107s to
18/33/14s (~8x), and all 76 ironcache-raft tests pass under `nextest --profile ci` in 100s total
with no timeout. debug-assertions + overflow-checks stay ON (this is dev, not release), so the sims
still exercise the checked build -- the standard DST posture (optimized harness, asserts kept).
The nextest switch (previous commit) remains: process-per-test isolation + a per-test timeout
(now 90s soft / 270s SIGKILL, headroom for the optimized DST under CI load) + the 30-min job cap
turn any FUTURE genuine hang into a fast named failure. The io_uring serial test-group stays as a
cheap precaution (rings are a limited kernel resource), though it was not the cause.
Local: 2224 non-io_uring tests + all 76 ironcache-raft tests green under nextest; fmt clean.
Ref #533, #527 (M1).
Signed-off-by: Zeke <ezequiel.lares@outlook.com>
perf-gate (A5)Same-runner ratchet of HEAD against the merge-base (both rebuilt and measured in this job).
Overall: PASS
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The diagnosis
The intermittent ~50-minute CI hang that has blocked merges (#520/#521/#523/#524/#525) is the io_uring
fixed_datapathring tests deadlocking under parallel load:fixed_datapath::testsis exactly a 6-test module.raft_cluster(the earlier suspect) passes 5/5 in isolation -- consistent with the true cause being resource contention, not that test.The fix (two layers)
test+io-uringjobs (cargo nextest run --profile ci): each test runs in its own process (isolating shared-state / port coupling) with a per-test timeout (.config/nextest.toml: SIGKILL after 120s) so any residual hang is a fast named failure, not a runner burn.timeout-minutes: 30caps the job. Acargo test --docstep is added (nextest skips doctests).max-threads = 1, so ring lifecycles run serially and never contend -- removing the contention at the source.Installed via
taiki-e/install-action@nextest(prebuilt).Validation
2224 non-io_uring tests pass under
nextest --profile cilocally; the config + the io-uring group filter parse and correctly target thefixed_datapathtests. (io_uring itself is unrunnable in the local sandbox --PermissionDeniedon ring init -- but runs on GitHub's ubuntu runners, where the dedicated io-uring job already proves it green. This PR's own CI is the end-to-end proof: it runs under the new nextest config.)Part of the production-readiness epic #527 (M1). Closes #533.
🤖 Generated with Claude Code