tcp: strict RFC 5961 RST sequence validation (contributes to #1132) by ibondarenko1 · Pull Request #13165 · google/gvisor

ibondarenko1 · 2026-05-13T19:01:22Z

Contribution to issue #1132 (open since 2019-08, "Validate segment handling in Netstack as per RFC 793 page 69"). That issue documents the gap explicitly: "We also need to verify which parts have been updated by RFC5961 to make TCP robust against attacks." This PR adds RFC 5961 section 3.2 strict-match RST acceptance.

Problem

pkg/tcpip/transport/tcp/connect.go handleReset accepts any RST whose sequence number is within the advertised receive window (pkg/tcpip/transport/tcp/rcv.go:70-81 acceptable). The source comment cites RFC 793 page 37.

With a 4 MB advertised window the per-packet probability of a blind off-path RST hitting an in-window sequence is roughly window / 2^32 (~0.1 percent). An off-path attacker that knows the 4-tuple can terminate an established TCP connection in a small number of packets.

CVE-2024-10603 (commit cbdb2c61b1 "Randomize TCP source port selection", 2023-11-10) already added source-port randomization, which raised the practical cost by a factor of ~2^16. With both source-port randomization AND the existing window-based RST check, the joint per-packet blind probability is ~window / 2^48. The remaining gap is the RST acceptance rule itself.

RFC 5961 section 3.2

RFC 5961 section 3.2 tightens RFC 793 page 37:

If the RST bit is set and the sequence number exactly matches the next expected sequence number (RCV.NXT), then TCP MUST reset the connection.

If the RST bit is set and the sequence number does not exactly match the next expected sequence value, yet is within the current receive window, TCP MUST send an acknowledgment (challenge ACK)

Linux net/ipv4/tcp_input.c tcp_validate_incoming has implemented this since version 3.6 (2012). gVisor partially implements RFC 5961: section 4.1 (SYN-in-window challenge ACK) is present at connect.go:1290-1312, with an inline RFC 5961 link in the comment. Section 3.2 (RST strict-match) was the remaining gap that #1132 tracks.

Change

handleReset is rewritten to RFC 5961 section 3.2:

Out of window: silent drop. (Unchanged.)
In window but not exactly RCV.NXT: send a challenge ACK via the existing rate-limited helper e.snd.maybeSendOutOfWindowAck(s), then drop. (NEW.)
Exactly RCV.NXT: accept and reset the connection. (Unchanged for legitimate peers.)

The challenge ACK path reuses the same helper used by the existing RFC 5961 section 4.1 implementation in this same file. The rate-limit inside maybeSendOutOfWindowAck also defends against the CVE-2016-5696 class of challenge-ACK side channels.

Tests

New file pkg/tcpip/transport/tcp/test/e2e/handle_reset_rfc5961_test.go adds three focused regression tests:

TestRFC5961_RSTOutOfWindowIsDropped (silent drop, unchanged path)
TestRFC5961_RSTInWindowNotExactSendsChallengeAck (NEW behavior: no abort, challenge ACK observed via the test sniffer)
TestRFC5961_RSTExactMatchAbortsConnection (legitimate RST still aborts)

Local run:

$ bazel test //pkg/tcpip/transport/tcp/test/e2e:handle_reset_rfc5961_test \
    --test_output=all --test_arg=-test.v --cache_test_results=no

--- PASS: TestRFC5961_RSTOutOfWindowIsDropped (0.05s)
--- PASS: TestRFC5961_RSTInWindowNotExactSendsChallengeAck (0.00s)
--- PASS: TestRFC5961_RSTExactMatchAbortsConnection (0.00s)
PASS

//pkg/tcpip/transport/tcp/test/e2e:handle_reset_rfc5961_test  PASSED in 0.1s
Executed 1 out of 1 test: 1 test passes.

Challenge-ACK behavior verified via the test sniffer:

recv tcp 10.0.0.2:4096 -> 10.0.0.1:65106 flags:  R    seqnum:1814 ack:0 win:30000
send tcp 10.0.0.1:65106 -> 10.0.0.2:4096 flags:    A  seqnum:854776498 ack:790 win:65535

RST at SEQ=1814 (in window, not equal to RCV.NXT=790) is answered with a challenge ACK at SND.NXT=854776498 / ACK=790, instead of a connection reset.

//pkg/tcpip/transport/tcp/test/e2e:rcv_test also passes on this branch (independent verification that the receiver path is unchanged for non-RST segments).

bazel build //pkg/tcpip/transport/tcp:tcp builds clean against master.

Framing

Hardening contribution. Class is denial of service (off-path blind RST termination of an established connection). Google VRP does not bounty DoS, and gVisor's published threat model does not claim RFC 5961 compliance. This PR is a Linux-parity uplift contributing to the existing open issue, not a security advisory.

References

Issue Validate segment handling in Netstack as per https://tools.ietf.org/html/rfc793 page 69 #1132 (open) "Validate segment handling in Netstack as per RFC 793 page 69"
RFC 5961 section 3.2 (RST acceptance) and section 4.1 (SYN-in-window challenge ACK)
Linux net/ipv4/tcp_input.c tcp_validate_incoming (RFC 5961 section 3 since Linux 3.6, 2012)
CVE-2016-5696 (Off-Path TCP Exploits, Cao et al., USENIX Security 2016)
CVE-2024-10603 (gVisor TCP source-port randomization, commit cbdb2c61b1, 2023-11)

nybidari

Minor nits! Otherwise LGTM

nybidari · 2026-05-19T21:53:43Z

+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package handle_reset_rfc5961_test


Instead of creating a new test file, we can add these tests in either pkg/tcpip/transport/tcp/test/e2e/e2e.go or pkg/tcpip/transport/tcp/test/e2e/tcp_test.go

nybidari · 2026-05-19T21:53:46Z

+// number falls outside the advertised receive window is silently dropped.
+// Pre-RFC-5961 behavior (RFC 793 page 37) already silently dropped these,
+// so this asserts the unchanged path.
+func TestRFC5961_RSTOutOfWindowIsDropped(t *testing.T) {


Remove the RFC5961 from the test names, the comment about the behavior change and RFC can stay.

nybidari · 2026-05-19T21:53:55Z

+// The strict-match rule defends against off-path blind RST injection.
+// Linux has implemented it since version 3.6 (2012); see
+// net/ipv4/tcp_input.c tcp_validate_incoming(). Contribution to
+// gvisor.dev/issues/1132.


Remove this line from the comments: "Contribution to gvisor.dev/issues/1132"

handleReset in pkg/tcpip/transport/tcp/connect.go accepted any RST whose sequence number was within the advertised receive window (rcv.go:70-81 acceptable). The source comment cited RFC 793 page 37. With a 4 MB advertised window the per-packet probability of a blind off-path RST hitting an in-window sequence is roughly window/2^32 (about 0.1 percent). An off-path attacker that knows the 4-tuple could terminate an established TCP connection in a small number of packets. CVE-2024-10603 (commit cbdb2c6, 2023-11-10) added source-port randomization, which raised the practical cost by a factor of about 2^16. With both source-port randomization AND the existing window- based RST check, the joint per-packet blind probability is around window/2^48. The remaining gap is the RST acceptance rule itself. Rewrite handleReset to implement RFC 5961 section 3.2: * Segment sequence number out of window: silent drop. (Unchanged.) * Segment sequence number in window but not exactly equal to RCV.NXT: send a challenge ACK via the existing rate-limited helper e.snd.maybeSendOutOfWindowAck(s) and drop the segment. (New. Previously such a RST aborted the connection.) * Segment sequence number exactly equals RCV.NXT: accept and reset the connection. (Unchanged for legitimate peers.) The challenge ACK path reuses the same helper used by the existing RFC 5961 section 4.1 implementation at connect.go:1290-1312 (SYN-in-window). The rate-limit is provided by maybeSendOutOfWindowAck which already defends against the CVE-2016-5696 class of challenge-ACK side channels on the Linux side. Linux net/ipv4/tcp_input.c tcp_validate_incoming has implemented RFC 5961 section 3.2 since version 3.6 (2012). gVisor partially implements RFC 5961: section 4.1 is present, this commit adds section 3.2. Tests added to pkg/tcpip/transport/tcp/test/e2e/tcp_test.go: * TestRSTOutOfWindowIsDropped (silent drop, unchanged path) * TestRSTInWindowNotExactSendsChallengeAck (new behavior, challenge ACK observed via sniffer) * TestRSTExactMatchAbortsConnection (legitimate RST still aborts) Tested: bazel build //pkg/tcpip/transport/tcp:tcp bazel test //pkg/tcpip/transport/tcp/test/e2e:tcp_test bazel test //pkg/tcpip/transport/tcp/test/e2e:rcv_test Hardening, not an advisory: class is denial of service (off-path blind RST termination); Google VRP does not bounty DoS, and gVisor's published threat model does not claim RFC 5961 compliance. References: RFC 5961 section 3.2 (RST acceptance) RFC 5961 section 4.1 (SYN-in-window challenge ACK, connect.go:1290) Linux net/ipv4/tcp_input.c tcp_validate_incoming CVE-2016-5696 (Off-Path TCP Exploits, Cao et al., USENIX Sec 2016) CVE-2024-10603 (TCP source-port randomization, gVisor cbdb2c6)

ibondarenko1 · 2026-05-19T23:16:06Z

@nybidari thanks for the review. All three nits addressed in commit 484ef120.

Tests moved from pkg/tcpip/transport/tcp/test/e2e/handle_reset_rfc5961_test.go into pkg/tcpip/transport/tcp/test/e2e/tcp_test.go (inserted before TestMain). The standalone file is deleted and its go_test rule removed from BUILD.
Test names dropped the RFC5961_ prefix:
- TestRFC5961_RSTOutOfWindowIsDropped -> TestRSTOutOfWindowIsDropped
- TestRFC5961_RSTInWindowNotExactSendsChallengeAck -> TestRSTInWindowNotExactSendsChallengeAck
- TestRFC5961_RSTExactMatchAbortsConnection -> TestRSTExactMatchAbortsConnection
  The RFC 5961 reference and behavior-change note stay in each test's docstring.
Contribution to gvisor.dev/issues/1132. removed from the handleReset comment in connect.go.

Net diff vs upstream: +177/-37 across two files (connect.go and tcp_test.go). Force-pushed as 484ef120.

nybidari reviewed May 19, 2026

View reviewed changes

ibondarenko1 force-pushed the hardening/rfc5961-rst-strict-match branch from 3bc57ba to 484ef12 Compare May 19, 2026 23:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tcp: strict RFC 5961 RST sequence validation (contributes to #1132)#13165

tcp: strict RFC 5961 RST sequence validation (contributes to #1132)#13165
ibondarenko1 wants to merge 1 commit into
google:masterfrom
ibondarenko1:hardening/rfc5961-rst-strict-match

ibondarenko1 commented May 13, 2026

Uh oh!

nybidari left a comment

Uh oh!

nybidari May 19, 2026

Uh oh!

nybidari May 19, 2026

Uh oh!

nybidari May 19, 2026

Uh oh!

ibondarenko1 commented May 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ibondarenko1 commented May 13, 2026

Problem

RFC 5961 section 3.2

Change

Tests

Framing

References

Uh oh!

nybidari left a comment

Choose a reason for hiding this comment

Uh oh!

nybidari May 19, 2026

Choose a reason for hiding this comment

Uh oh!

nybidari May 19, 2026

Choose a reason for hiding this comment

Uh oh!

nybidari May 19, 2026

Choose a reason for hiding this comment

Uh oh!

ibondarenko1 commented May 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants