Backend server instances order can be inconsistent across controller instances

[davoustp]

Symptom

The haproxy configuration generated on controller instances are not consistent from one to another: the order of backends can be different, which becomes a problem with a non randomized / round-robin load balancing algorithm.

As a result, when using load balancing algorithm such as first, the request will be routed differently depending on which haproxy controller instance received the request in the first place.
In other words, the routing will be inconsistent across controller instances.

Expected behaviour

The backend orders must be the same on all controller instances across a ingress controller deployment.

Environment details

kubernetes-ingress is a custom build (bringing FIPS compliance) using tag v3.2.11 as the baseline (FIPS-compliant Ubuntu 22.04.5 base image), haproxy tag 3.2.20 and go version 1.26.2.

Deployed using the official Helm chart inDeployment mode with 2 replicas.

Use case details

The haproxy ingress controller is deployed with 2 instances, sitting behind an edge load balancer.

An external (non Kubernetes) HTTP service is running on two distinct IP addresses, same port.
This service is declared using an Ingress pointing at a Service with a corresponding EndpointSlice as shown below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ext-svc
  namespace: test-ns
  annotations:
    haproxy.org/check: "true"
    haproxy.org/server-ca: test-ns/server-ca
    haproxy.org/server-ssl: "true"
spec:
  ingressClassName: ingress-classname
  rules:
  - host: a.fully.qualified.domain
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: ext-svc
            port:
              name: https
  tls:
  - hosts:
    - a.fully.qualified.domain
    secretName: ingress-tls

apiVersion: v1
kind: Service
metadata:
  name: ext-svc
  namespace: test-ns
  annotations:
    haproxy.org/check-http: GET /
    haproxy.org/check-interval: 5s
    haproxy.org/load-balance: first
    haproxy.org/pod-maxconn: "65535"
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443

apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: ext-svc
  namespace: test-ns
  labels:
    kubernetes.io/service-name: ext-svc
addressType: IPv4
ports:
- name: https
  port: 443
  protocol: TCP
endpoints:
- addresses:
  - 10.164.37.182
  - 10.164.37.183
  conditions:
    ready: true
    serving: true
    terminating: false

The ingress controller is deployed as a Deployment(using the official Helm chart) with 2 replicas.

Important to reproduce:

• there are two nodes in the cluster, node A (IP 10.164.37.182) and node B (IP 10.164.37.183)
• the two haproxy controllers are running on these nodes A and B
• the two external service instances run on the same node A and B (as a systemd unit, not as Kubernetes workload)
• restart node A, then wait for it to become ready again before restarting node B.

The backend generated for the resources above (located in file /etc/haproxy/haproxy.cfg) are:

• Ingress Controller instance 1 (node A)

backend test-ns_svc_ext-svc_https from haproxytech
  mode http
  balance first
  option forwardfor
  no option abortonclose
  option httpchk GET /
  default-server check ssl alpn h2,http/1.1 ca-file /etc/haproxy/certs/ca/test-ns_server-ca.pem inter 5000 maxconn 32767 verify required
  server SRV_1 10.164.37.182:443 enabled
  server SRV_2 10.164.37.183:443 enabled
  server SRV_3 127.0.0.1:1 disabled

=> SRV_1 is 10.164.37.182

• Ingress Controller instance 2 (node B)

backend test-ns_svc_ext-svc_https from haproxytech
  mode http
  balance first
  option forwardfor
  no option abortonclose
  option httpchk GET /
  default-server check ssl alpn h2,http/1.1 ca-file /etc/haproxy/certs/ca/test-ns_server-ca.pem inter 5000 maxconn 32767 verify required
  server SRV_1 10.164.37.183:443 enabled
  server SRV_2 10.164.37.182:443 enabled
  server SRV_3 127.0.0.1:1 disabled

=> SRV_1 is 10.164.37.183

Hitting the front load balancer routes to either haproxy instances, and shows that both backend server instances are hit instead of only one of them, defeating the firstalgorithm entirely.

[ivanmatmati]

Hi @davoustp, thanks for reporting. A fix is on its way and will be released soon.

[unasd]

We are experiencing the same issue with balance source on v3.2.6 (DaemonSet, 3 replicas, AKS).

Our Setup

A Deployment with 2 replicas behind a Service, using haproxy.org/load-balance: "source" with hash-type consistent in backend-config-snippet.

Observed Behavior

Comparing haproxy.cfg across 3 controller pods:

Pod 1:

server SRV_1 10.244.1.50:8080 enabled
server SRV_2 10.244.2.51:8080 enabled
server SRV_3 127.0.0.1:1 disabled

Pod 2:

server SRV_1 10.244.1.50:8080 enabled
server SRV_2 10.244.2.51:8080 enabled
server SRV_3 127.0.0.1:1 disabled

Pod 3:

server SRV_1 10.244.2.51:8080 enabled
server SRV_2 10.244.1.50:8080 enabled
server SRV_3 127.0.0.1:1 disabled

Pod 3 has the addresses swapped.

Access Log Evidence

Two requests from the same client session, seconds apart. Both hashed to SRV_1, but the actual backend differs — meaning a different controller pod handled each request and its SRV_1 pointed to a different address:

192.168.1.100 - - [27/Jul/2026:01:38:26.836]
  "GET https://app.example.com/MainPage/view.do HTTP/2.0" 200
  [backend_svc_app-svc_http] [SRV_1] [10.244.2.51:8080]
  "JSESSIONID=BA07BD583DBCA6546483124168F1BB35"

192.168.1.100 - - [27/Jul/2026:01:40:46.941]
  "POST https://app.example.com/MainPage/selectInfo.do HTTP/2.0" 200
  [backend_svc_app-svc_http] [SRV_1] [10.244.1.50:8080]
  "JSESSIONID=BA07BD583DBCA6546483124168F1BB35"

Same client IP, same JSESSIONID, same slot name (SRV_1), different backend pod. Session broke at the second request.

Additional Impact Beyond first

This issue affects all hash-based algorithms — source, uri, hdr, url_param, rdp-cookie — since they all select a server by slot position. For balance source specifically, clients that cannot use cookies (e.g., .NET WinForms thick clients, RDP gateways, native apps) have no alternative stickiness mechanism, making this a complete loss of session affinity.

Note that cookie-persistence is not affected because it uses dynamic-cookie-key (which derives the value from the actual IP:PORT, not the slot name). This confirms the slot ordering is known to be inconsistent across replicas, but the same handling was not extended to hash-based algorithms.