Skip to content

fix: eliminate shared-pool poisoning behind H2 desyncs + trap-wire remaining flakes#70

Merged
st0o0 merged 16 commits into
mainfrom
test/serialize-high-concurrency-spec
Jul 6, 2026
Merged

fix: eliminate shared-pool poisoning behind H2 desyncs + trap-wire remaining flakes#70
st0o0 merged 16 commits into
mainfrom
test/serialize-high-concurrency-spec

Conversation

@st0o0

@st0o0 st0o0 commented Jul 6, 2026

Copy link
Copy Markdown
Member

Summary

Closes out the H2 receive-desync hunt: root cause found, fixed, and verified with
119/120 clean full-suite soak runs (baseline before: 12 failing runs out of 40).
Also fixes the CI/test-infra issues uncovered along the way and leaves permanent
diagnostics so the one remaining (unreproduced) flake produces evidence on its
next occurrence.

Root cause: cross-connection buffer-pool poisoning

DecompressingContent disposed its body source stream twice (the codec
stream owned it via leaveOpen: false while the caller also await using-ed it).
The second dispose hit a PooledArrayMemoryOwner wrapper that had already been
recycled and re-rented, returning the new owner's array to the process-wide
SharedPool. Under full-suite load this poisoned buffers across connections:
H1 plaintext appearing inside H2 receive buffers, ObjectDisposedException on
fresh buffers, RFC 9113 §4.2 garbage-length desyncs, and 8 MB download stalls.

Fixes (each TDD, with regression tests):

  • fix(body): leaveOpen: true for decompressors (single ownership at the
    caller) + idempotent PooledMemoryStream.Dispose
  • fix(http2): FrameDecoder copies offset-wrapped buffers instead of adopting
    them; receive-path corruption now fails fast at Warning instead of wedging
    to the test timeout
  • servus.akka (submodule bumps): TCP transport contract fix — every
    DisconnectTransport is answered with a TransportDisconnected echo and
    TransportConnected after any signaled disconnect (fixed the H11
    deadlock/cancellation flakes); actor-confined QUIC read-pump buffer ownership;
    staggered multi-address TCP connect (Windows drops SYNs to non-listening ::1)

Test-infra fixes

  • CI kestrel matrix leg actually ran Docker (env var casing) — fixed, plus
    Secure cookie attribute only on TLS
  • Random-port reservation with bind retry in the Kestrel backend (Windows
    bind(0) LIFO reuse + TIME_WAIT caused full-suite wipeouts)
  • Additive tracing via CompositeTraceListener (servus.akka) + TestTracing.Root:
    specs no longer replace the global Senf listener and silently swallow every
    other test's warnings — this is how the §4.2 desyncs stayed invisible
  • FaultTraceDump: all three integration suites dump a 128K all-category trace
    ring to %TEMP% whenever a test fails, including [Fact(Timeout)] watchdogs
  • feat(h3): the H3 client receive path previously traced nothing below
    Warning — now traces request dispatch, HEADERS, per-DATA-frame sizes, FIN

Verification

Suite Runs Result
IntegrationTests.Client (Kestrel, production config) 70 69 clean, 0 desyncs/dumps
IntegrationTests.End2End 25 25 clean
IntegrationTests.Server 25 25 clean
Unit + stage 5968/5968
servus.akka 670/670

The single failure in 70 client runs is an H3 first-request stall
(H3.EncodingSpec, 1/70, never reproduced in a 40-run instrumented hunt).
It is a different failure class (no protocol error, connection established,
silence before HEADERS) and is now trap-wired by the instrumentation above.

@st0o0 st0o0 merged commit 96605de into main Jul 6, 2026
5 checks passed
@st0o0 st0o0 deleted the test/serialize-high-concurrency-spec branch July 6, 2026 12:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant