Skip to content

feat(client): connection resilience - jittered reconnect backoff, idempotency-safe replay, and fault visibility#77

Merged
st0o0 merged 7 commits into
mainfrom
feat/client-resilience-fixes
Jul 8, 2026
Merged

feat(client): connection resilience - jittered reconnect backoff, idempotency-safe replay, and fault visibility#77
st0o0 merged 7 commits into
mainfrom
feat/client-resilience-fixes

Conversation

@st0o0

@st0o0 st0o0 commented Jul 8, 2026

Copy link
Copy Markdown
Member

Summary

Hardens client (and one server) connection resilience so transient failures are handled predictably instead of turning into busy-loops, silent data loss, duplicate side effects, or hung callers. Five related changes:

  1. Shared exponential reconnect backoff with jitter across all protocols
  2. Idempotency-safe replay + deterministic teardown for H1.1 / H2 / H3
  3. Surface content-decode failures instead of silently truncating
  4. Surface application-handler crashes in the server bridge
  5. Test/dependency housekeeping

Details

Reconnect backoff (feat)

New ReconnectBackoff.Compute centralizes exponential-backoff-with-jitter so HTTP/1.x (ReconnectPolicy), HTTP/2 & HTTP/3 state machines, and ClientStreamOwner all space their retries identically. The first attempt after a real disconnect still fires immediately; only subsequent retries are deferred behind a reconnect-backoff timer. Spacing avoids hammering a connection-refused peer in a tight loop; jitter avoids a thundering herd of correlated reconnects.

New, fully-defaulted knobs on the client options:

  • ReconnectInitialBackoff (default 100 ms), ReconnectMaxBackoff (5 s), ReconnectBackoffMultiplier (2.0), ReconnectBackoffJitter (±20%)
  • StreamRetryBackoffJitter (±20%) for stream re-materialization

Behavior note: reconnect retries are now spaced by default (100 ms initial). Set ReconnectInitialBackoff = TimeSpan.Zero to restore the previous zero-delay immediate-reconnect behavior.

Idempotency-safe replay & deterministic teardown (fix)

  • After connection loss, only idempotent in-flight requests are replayed. Non-idempotent requests (e.g. POST) are **failed rather than replayed*ve processed them, so a replay risks aduplicate side effect (RFC 9110 §9.2.2). Applied consistently to H1.1, H2, and H3.
  • On stage teardown (KillSwitch abort → PostStop → Cleanup), requests still in flight or buffered for replay arenow failed, not silently dropped, so calf hanging until their own client-sidetimeout. request.Fail is idempotent, so streams that already delivered a response are unaffected.
    ### Content-decode failures (fix)
    A corrupt or truncated compressed response body (Content-Encoding: gzip/…) now throws a clear HttpRequestException ("compressed data is corrupt or truncated") instead of silently returning a truncated body. Typed as HttpRequestException so HttpContent's stream-copy wrapping passes it through unwrapped.

Server handler crashes (fix) ApplicationBridgeStage no longer lets ann vanish: it reaches context disposal(OnDispose(context, exception)), emits a 500, and is traced — so operators can observe handler crashes. Crashes during context creation (before an `AppConed via tracing.

Housekeeping (chore)

Removed the obsolete WireBuffer wrapper-pool test hook and bumped servus.akka.

Testing

New/expanded specs cover each change: recoolicy scheduling,idempotent-vs-non-idempotent replay, teardown-fails-buffered/in-flight requests across H1.1/H2/H3, corrupt-gzip decode surfacing, and handler-crash disposeen.

@st0o0 st0o0 force-pushed the feat/client-resilience-fixes branch from 4bfc3af to 1d3bcc1 Compare July 8, 2026 10:36
@st0o0 st0o0 merged commit cc4fcb4 into main Jul 8, 2026
5 checks passed
@st0o0 st0o0 deleted the feat/client-resilience-fixes branch July 8, 2026 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant