fix: eliminate shared-pool poisoning behind H2 desyncs + trap-wire remaining flakes#70
Merged
Merged
Conversation
…ce desyncs at Warning
… on every 8MB-download failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes out the H2 receive-desync hunt: root cause found, fixed, and verified with
119/120 clean full-suite soak runs (baseline before: 12 failing runs out of 40).
Also fixes the CI/test-infra issues uncovered along the way and leaves permanent
diagnostics so the one remaining (unreproduced) flake produces evidence on its
next occurrence.
Root cause: cross-connection buffer-pool poisoning
DecompressingContentdisposed its body source stream twice (the codecstream owned it via
leaveOpen: falsewhile the caller alsoawait using-ed it).The second dispose hit a
PooledArrayMemoryOwnerwrapper that had already beenrecycled and re-rented, returning the new owner's array to the process-wide
SharedPool. Under full-suite load this poisoned buffers across connections:H1 plaintext appearing inside H2 receive buffers,
ObjectDisposedExceptiononfresh buffers, RFC 9113 §4.2 garbage-length desyncs, and 8 MB download stalls.
Fixes (each TDD, with regression tests):
fix(body):leaveOpen: truefor decompressors (single ownership at thecaller) + idempotent
PooledMemoryStream.Disposefix(http2):FrameDecodercopies offset-wrapped buffers instead of adoptingthem; receive-path corruption now fails fast at Warning instead of wedging
to the test timeout
DisconnectTransportis answered with aTransportDisconnectedecho andTransportConnectedafter any signaled disconnect (fixed the H11deadlock/cancellation flakes); actor-confined QUIC read-pump buffer ownership;
staggered multi-address TCP connect (Windows drops SYNs to non-listening ::1)
Test-infra fixes
Securecookie attribute only on TLSbind(0)LIFO reuse + TIME_WAIT caused full-suite wipeouts)CompositeTraceListener(servus.akka) +TestTracing.Root:specs no longer replace the global Senf listener and silently swallow every
other test's warnings — this is how the §4.2 desyncs stayed invisible
FaultTraceDump: all three integration suites dump a 128K all-category tracering to
%TEMP%whenever a test fails, including[Fact(Timeout)]watchdogsfeat(h3): the H3 client receive path previously traced nothing belowWarning — now traces request dispatch, HEADERS, per-DATA-frame sizes, FIN
Verification
The single failure in 70 client runs is an H3 first-request stall
(
H3.EncodingSpec, 1/70, never reproduced in a 40-run instrumented hunt).It is a different failure class (no protocol error, connection established,
silence before HEADERS) and is now trap-wired by the instrumentation above.