Read requests are being sent to few nodes

[abonacin]

Hi. We have a cluster with 50+ nodes. When we redirected app connections to the ZDM Proxy, it started reading from only 8 nodes, and all others dropped to 0 reads.
I didn't find anything helpful to understand this behavior. Is there anything I have to change?

We also had an issue with writes, but after a period, it was sending writes to all nodes (apparently)

At 14:30, we rolled back and moved app connections directly to cassandra.

[joao-r-reis]

ZDM proxy logs and zdm grafana dashboard screenshots would be helpful

[abonacin]

Hi,
Logs are something similar to:

time="2025-12-10T16:23:51Z" level=info msg="Starting ZDM proxy version 2.3.4"
time="2025-12-10T16:23:51Z" level=info msg="Parsed configuration: {\"PrimaryCluster\":\"ORIGIN\",\"ReadMode\":\"PRIMARY_ONLY\",\"ReplaceCqlFunctions\":false,\"AsyncHandshakeTimeoutMs\":4000,\"LogLevel\":\"INFO\",\"ControlConnMaxProtocolVersion\":\"DseV2\",\"TracingEnabled\":false,\"TracingRateLimit\":0,\"TracingRequestIdKey\":\"request-id\",\"ProxyTopologyIndex\":0,\"ProxyTopologyAddresses\":\"\",\"ProxyTopologyNumTokens\":8,\"OriginContactPoints\":\"172.x.x.4,172.x.x.3,172.x.x.2,172.x.x.1\",\"OriginPort\":9042,\"OriginSecureConnectBundlePath\":\"\",\"OriginLocalDatacenter\":\"DC2\",\"OriginUsername\":\"cassandra\",\"OriginConnectionTimeoutMs\":30000,\"OriginTlsServerCaPath\":\"\",\"OriginTlsClientCertPath\":\"\",\"OriginTlsClientKeyPath\":\"\",\"TargetContactPoints\":\"172.x.2x.1,172.x.2x.2\",\"TargetPort\":9042,\"TargetSecureConnectBundlePath\":\"\",\"TargetLocalDatacenter\":\"\",\"TargetUsername\":\"cassandra\",\"TargetConnectionTimeoutMs\":30000,\"TargetTlsServerCaPath\":\"\",\"TargetTlsClientCertPath\":\"\",\"TargetTlsClientKeyPath\":\"\",\"ProxyListenAddress\":\"172.y.y.y\",\"ProxyListenPort\":9042,\"ProxyRequestTimeoutMs\":10000,\"ProxyMaxClientConnections\":1000,\"ProxyMaxStreamIds\":2048,\"ProxyTlsCaPath\":\"\",\"ProxyTlsCertPath\":\"\",\"ProxyTlsKeyPath\":\"\",\"ProxyTlsRequireClientAuth\":false,\"MetricsEnabled\":true,\"MetricsAddress\":\"172.y.y.y\",\"MetricsPort\":14001,\"MetricsPrefix\":\"zdm\",\"MetricsOriginLatencyBucketsMs\":\"1, 4, 7, 10, 25, 40, 60, 80, 100, 150, 250, 500, 1000, 2500, 5000, 10000, 15000\",\"MetricsTargetLatencyBucketsMs\":\"1, 4, 7, 10, 25, 40, 60, 80, 100, 150, 250, 500, 1000, 2500, 5000, 10000, 15000\",\"MetricsAsyncReadLatencyBucketsMs\":\"1, 4, 7, 10, 25, 40, 60, 80, 100, 150, 250, 500, 1000, 2500, 5000, 10000, 15000\",\"HeartbeatIntervalMs\":30000,\"HeartbeatRetryIntervalMinMs\":250,\"HeartbeatRetryIntervalMaxMs\":30000,\"HeartbeatRetryBackoffFactor\":2,\"HeartbeatFailureThreshold\":1,\"SystemQueriesMode\":\"ORIGIN\",\"ForwardClientCredentialsToOrigin\":false,\"OriginEnableHostAssignment\":true,\"TargetEnableHostAssignment\":true,\"RequestWriteQueueSizeFrames\":128,\"RequestWriteBufferSizeBytes\":4096,\"RequestReadBufferSizeBytes\":32768,\"ResponseWriteQueueSizeFrames\":128,\"ResponseWriteBufferSizeBytes\":8192,\"ResponseReadBufferSizeBytes\":32768,\"RequestResponseMaxWorkers\":-1,\"WriteMaxWorkers\":-1,\"ReadMaxWorkers\":-1,\"ListenerMaxWorkers\":-1,\"EventQueueSizeFrames\":12,\"AsyncConnectorWriteQueueSizeFrames\":2048,\"AsyncConnectorWriteBufferSizeBytes\":4096}"
time="2025-12-10T16:23:51Z" level=info msg="SIGINT/SIGTERM listener started."
time="2025-12-10T16:23:51Z" level=info msg="Starting http server (metrics and health checks) on 172.y.y.y:14001"
time="2025-12-10T16:23:51Z" level=info msg="Attempting to start the proxy..."
time="2025-12-10T16:23:51Z" level=info msg="Using 16 request / response workers."
time="2025-12-10T16:23:51Z" level=info msg="Using 16 write workers."
time="2025-12-10T16:23:51Z" level=info msg="Using 32 read workers."
time="2025-12-10T16:23:51Z" level=info msg="Using 4 listener workers."
time="2025-12-10T16:23:51Z" level=info msg="Parsed Origin latency buckets: [0.001 0.004 0.007 0.01 0.025 0.04 0.06 0.08 0.1 0.15 0.25 0.5 1 2.5 5 10 15]"
time="2025-12-10T16:23:51Z" level=info msg="Parsed Target latency buckets: [0.001 0.004 0.007 0.01 0.025 0.04 0.06 0.08 0.1 0.15 0.25 0.5 1 2.5 5 10 15]"
time="2025-12-10T16:23:51Z" level=info msg="Parsed Async latency buckets: [0.001 0.004 0.007 0.01 0.025 0.04 0.06 0.08 0.1 0.15 0.25 0.5 1 2.5 5 10 15]"
time="2025-12-10T16:23:51Z" level=info msg="Validating config..."
time="2025-12-10T16:23:51Z" level=info msg="Proxy TLS was not configured."
time="2025-12-10T16:23:51Z" level=info msg="Starting proxy..."
time="2025-12-10T16:23:51Z" level=info msg="Parsed Topology Config: TopologyConfig{VirtualizationEnabled=true, Addresses=[172.y.y.y], Count=1, Index=0, NumTokens=8}"
time="2025-12-10T16:23:51Z" level=info msg="Parsed Origin contact points: [172.x.x.4 172.x.x.3 172.x.x.2 172.x.x.1]"
time="2025-12-10T16:23:51Z" level=info msg="Parsed Target contact points: [172.x.2x.1 172.x.2x.2]"
time="2025-12-10T16:23:51Z" level=info msg="TLS was not configured for Origin"
time="2025-12-10T16:23:51Z" level=info msg="TLS was not configured for Target"
time="2025-12-10T16:23:51Z" level=info msg="[openTCPConnection] Opening connection to 172.x.x.1:9042"
time="2025-12-10T16:23:51Z" level=info msg="[openTCPConnection] Successfully established connection with 172.x.x.1:9042"
time="2025-12-10T16:23:51Z" level=error msg="cqlConn{conn: 172.x.x.1:9042}: handshake failed: ERROR PROTOCOL ERROR (code=ErrorCode ProtocolError [0x0000000A], msg=Invalid or unsupported protocol version (66); the lowest supported version is 3 and the greatest is 4)"
time="2025-12-10T16:23:51Z" level=info msg="[ORIGIN] Control connection to 172.x.x.1:9042 disconnected"
time="2025-12-10T16:23:51Z" level=info msg="[openTCPConnection] Opening connection to 172.x.x.1:9042"
time="2025-12-10T16:23:51Z" level=info msg="[openTCPConnection] Successfully established connection with 172.x.x.1:9042"
time="2025-12-10T16:23:51Z" level=error msg="cqlConn{conn: 172.x.x.1:9042}: handshake failed: ERROR PROTOCOL ERROR (code=ErrorCode ProtocolError [0x0000000A], msg=Invalid or unsupported protocol version (65); the lowest supported version is 3 and the greatest is 4)"
time="2025-12-10T16:23:51Z" level=info msg="[ORIGIN] Control connection to 172.x.x.1:9042 disconnected"
time="2025-12-10T16:23:51Z" level=info msg="[openTCPConnection] Opening connection to 172.x.x.1:9042"
time="2025-12-10T16:23:51Z" level=info msg="[openTCPConnection] Successfully established connection with 172.x.x.1:9042"
time="2025-12-10T16:23:51Z" level=warning msg="cqlConn{conn: 172.x.x.1:9042}: expected AUTHENTICATE, got READY – is authentication required?"
time="2025-12-10T16:23:51Z" level=info msg="Refreshed ORIGIN orderedHostsInLocalDc. Assigned Hosts: [Host{addr: 172.x.x.3, port: 9042, host_id: 64a6b137f7d34493a245f6589f0143a9} Host{addr: 172.x.x.1, port: 9042, host_id: 60ab4a9c8d6a40fc9dfcc047467c676a} Host{addr: 172.x.x.2, port: 9042, host_id: 45a71f9fdb324d42b063b22c93584bfc} Host{addr: 172.x.x.4, port: 9042, host_id: aed8382b8caa4c398fc673573c848f47}], VirtualHosts: [VirtualHost{addr: 172.y.y.y, host_id: 383ca50a-d11c-5cc4-b104-af653652e29d, rack: rack0, tokens: [-9223372036854775808 -6917529027641081856 -4611686018427387904 -2305843009213693952 0 2305843009213693952 4611686018427387904 6917529027641081856], host: Host{addr: 172.x.x.3, port: 9042, host_id: 64a6b137f7d34493a245f6589f0143a9}, partitioner: org.apache.cassandra.dht.Murmur3Partitioner}], ProxyTopologyIndex: 0"
time="2025-12-10T16:23:51Z" level=info msg="Successfully opened control connection to ORIGIN using endpoint 172.x.x.1:9042 with ProtocolVersion OSS 4."
time="2025-12-10T16:23:51Z" level=info msg="[openTCPConnection] Opening connection to 172.x.2x.1:9042"
time="2025-12-10T16:23:51Z" level=info msg="[openTCPConnection] Successfully established connection with 172.x.2x.1:9042"
time="2025-12-10T16:23:51Z" level=error msg="cqlConn{conn: 172.x.2x.1:9042}: handshake failed: ERROR PROTOCOL ERROR (code=ErrorCode ProtocolError [0x0000000A], msg=Invalid or unsupported protocol version (66); the lowest supported version is 3 and the greatest is 4)"
time="2025-12-10T16:23:51Z" level=info msg="[TARGET] Control connection to 172.x.2x.1:9042 disconnected"
time="2025-12-10T16:23:51Z" level=info msg="[openTCPConnection] Opening connection to 172.x.2x.1:9042"
time="2025-12-10T16:23:51Z" level=info msg="[openTCPConnection] Successfully established connection with 172.x.2x.1:9042"
time="2025-12-10T16:23:51Z" level=error msg="cqlConn{conn: 172.x.2x.1:9042}: handshake failed: ERROR PROTOCOL ERROR (code=ErrorCode ProtocolError [0x0000000A], msg=Invalid or unsupported protocol version (65); the lowest supported version is 3 and the greatest is 4)"
time="2025-12-10T16:23:51Z" level=info msg="[TARGET] Control connection to 172.x.2x.1:9042 disconnected"
time="2025-12-10T16:23:51Z" level=info msg="[openTCPConnection] Opening connection to 172.x.2x.1:9042"
time="2025-12-10T16:23:51Z" level=info msg="[openTCPConnection] Successfully established connection with 172.x.2x.1:9042"
time="2025-12-10T16:23:51Z" level=warning msg="cqlConn{conn: 172.x.2x.1:9042}: expected AUTHENTICATE, got READY – is authentication required?"
time="2025-12-10T16:23:51Z" level=info msg="Refreshed TARGET orderedHostsInLocalDc. Assigned Hosts: [Host{addr: 172.x.2x.2, port: 9042, host_id: 266baf7bf90d407a8681831ef9bdfd8c} Host{addr: 172.x.2x.1, port: 9042, host_id: 64fbdbed7b4a4f65988407947bf21937}], VirtualHosts: [VirtualHost{addr: 172.y.y.y, host_id: 383ca50a-d11c-5cc4-b104-af653652e29d, rack: rack0, tokens: [-9223372036854775808 -6917529027641081856 -4611686018427387904 -2305843009213693952 0 2305843009213693952 4611686018427387904 6917529027641081856], host: Host{addr: 172.x.2x.1, port: 9042, host_id: 64fbdbed7b4a4f65988407947bf21937}, partitioner: org.apache.cassandra.dht.Murmur3Partitioner}], ProxyTopologyIndex: 0"
time="2025-12-10T16:23:51Z" level=info msg="Successfully opened control connection to TARGET using endpoint 172.x.2x.1:9042 with ProtocolVersion OSS 4."
time="2025-12-10T16:23:51Z" level=info msg="Initialized origin control connection. Cluster Name: TestCluster, Hosts: map[45a71f9f-db32-4d42-b063-b22c93584bfc:Host{addr: 172.x.x.2, port: 9042, host_id: 45a71f9fdb324d42b063b22c93584bfc} 60ab4a9c-8d6a-40fc-9dfc-c047467c676a:Host{addr: 172.x.x.1, port: 9042, host_id: 60ab4a9c8d6a40fc9dfcc047467c676a} 64a6b137-f7d3-4493-a245-f6589f0143a9:Host{addr: 172.x.x.3, port: 9042, host_id: 64a6b137f7d34493a245f6589f0143a9} aed8382b-8caa-4c39-8fc6-73573c848f47:Host{addr: 172.x.x.4, port: 9042, host_id: aed8382b8caa4c398fc673573c848f47}], Assigned Hosts: [Host{addr: 172.x.x.3, port: 9042, host_id: 64a6b137f7d34493a245f6589f0143a9} Host{addr: 172.x.x.1, port: 9042, host_id: 60ab4a9c8d6a40fc9dfcc047467c676a} Host{addr: 172.x.x.2, port: 9042, host_id: 45a71f9fdb324d42b063b22c93584bfc} Host{addr: 172.x.x.4, port: 9042, host_id: aed8382b8caa4c398fc673573c848f47}]."
time="2025-12-10T16:23:51Z" level=info msg="Initialized target control connection. Cluster Name: EDC-SBX, Hosts: map[266baf7b-f90d-407a-8681-831ef9bdfd8c:Host{addr: 172.x.2x.2, port: 9042, host_id: 266baf7bf90d407a8681831ef9bdfd8c} 64fbdbed-7b4a-4f65-9884-07947bf21937:Host{addr: 172.x.2x.1, port: 9042, host_id: 64fbdbed7b4a4f65988407947bf21937}], Assigned Hosts: [Host{addr: 172.x.2x.2, port: 9042, host_id: 266baf7bf90d407a8681831ef9bdfd8c} Host{addr: 172.x.2x.1, port: 9042, host_id: 64fbdbed7b4a4f65988407947bf21937}]."
time="2025-12-10T16:23:51Z" level=info msg="Proxy connected and ready to accept queries on 172.y.y.y:9042"
time="2025-12-10T16:23:51Z" level=info msg="Proxy started. Waiting for SIGINT/SIGTERM to shutdown."
time="2025-12-10T16:23:52Z" level=info msg="Accepted connection from 172.c.c.1:48458"
time="2025-12-10T16:23:52Z" level=info msg="Accepted connection from 172.c.c.7:48214"
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Opening request connection to ORIGIN (172.x.x.1:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Opening request connection to ORIGIN (172.x.x.2:9042)."
time="2025-12-10T16:23:52Z" level=info msg="Accepted connection from 172.c.c.3:40354"
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Opening request connection to ORIGIN (172.x.x.4:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Request connection to ORIGIN (172.x.x.2:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Opening request connection to TARGET (172.x.2x.2:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Request connection to ORIGIN (172.x.x.1:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Opening request connection to TARGET (172.x.2x.1:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Request connection to ORIGIN (172.x.x.4:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Opening request connection to TARGET (172.x.2x.1:9042)."
time="2025-12-10T16:23:52Z" level=info msg="Accepted connection from 172.c.c.6:45198"
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Request connection to TARGET (172.x.2x.1:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Request connection to TARGET (172.x.2x.1:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Opening request connection to ORIGIN (172.x.x.3:9042)."
time="2025-12-10T16:23:52Z" level=info msg="Accepted connection from 172.c.c.8:34828"
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Request connection to TARGET (172.x.2x.2:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Opening request connection to ORIGIN (172.x.x.1:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Request connection to ORIGIN (172.x.x.3:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Opening request connection to TARGET (172.x.2x.2:9042)."
time="2025-12-10T16:23:52Z" level=info msg="Accepted connection from 172.c.c.5:60898"
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Opening request connection to ORIGIN (172.x.x.2:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Request connection to ORIGIN (172.x.x.1:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Opening request connection to TARGET (172.x.2x.1:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Request connection to TARGET (172.x.2x.2:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Request connection to ORIGIN (172.x.x.2:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Opening request connection to TARGET (172.x.2x.2:9042)."
time="2025-12-10T16:23:52Z" level=info msg="Initiating startup between 172.c.c.7:48214 and 172.x.2x.2:9042 (TARGET)"
time="2025-12-10T16:23:52Z" level=info msg="Handshake successful with client 172.c.c.7:48214"
time="2025-12-10T16:23:52Z" level=info msg="Accepted connection from 172.c.c.2:39090"
time="2025-12-10T16:23:52Z" level=info msg="Accepted connection from 172.c.c.4:57482"
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Opening request connection to ORIGIN (172.x.x.4:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Request connection to TARGET (172.x.2x.1:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Opening request connection to ORIGIN (172.x.x.3:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Request connection to ORIGIN (172.x.x.3:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Opening request connection to TARGET (172.x.2x.2:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Request connection to TARGET (172.x.2x.2:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Request connection to ORIGIN (172.x.x.4:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Request connection to TARGET (172.x.2x.2:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Opening request connection to TARGET (172.x.2x.1:9042)."
time="2025-12-10T16:23:52Z" level=info msg="Initiating startup between 172.c.c.6:45198 and 172.x.2x.2:9042 (TARGET)"
time="2025-12-10T16:23:52Z" level=info msg="Handshake successful with client 172.c.c.6:45198"
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Request connection to TARGET (172.x.2x.1:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="Initiating startup between 172.c.c.4:57482 and 172.x.2x.2:9042 (TARGET)"
time="2025-12-10T16:23:52Z" level=info msg="Handshake successful with client 172.c.c.4:57482"
time="2025-12-10T16:23:52Z" level=info msg="Initiating startup between 172.c.c.5:60898 and 172.x.2x.2:9042 (TARGET)"
time="2025-12-10T16:23:52Z" level=info msg="Handshake successful with client 172.c.c.5:60898"
time="2025-12-10T16:23:52Z" level=info msg="Initiating startup between 172.c.c.2:39090 and 172.x.2x.1:9042 (TARGET)"
time="2025-12-10T16:23:52Z" level=info msg="Handshake successful with client 172.c.c.2:39090"
time="2025-12-10T16:23:52Z" level=info msg="Initiating startup between 172.c.c.3:40354 and 172.x.2x.1:9042 (TARGET)"
time="2025-12-10T16:23:52Z" level=info msg="Handshake successful with client 172.c.c.3:40354"
time="2025-12-10T16:23:52Z" level=info msg="Initiating startup between 172.c.c.8:34828 and 172.x.2x.1:9042 (TARGET)"
time="2025-12-10T16:23:52Z" level=info msg="Handshake successful with client 172.c.c.8:34828"
time="2025-12-10T16:23:52Z" level=info msg="Initiating startup between 172.c.c.1:48458 and 172.x.2x.1:9042 (TARGET)"
time="2025-12-10T16:23:52Z" level=info msg="Handshake successful with client 172.c.c.1:48458"
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '5ba35c4c9e66e265e86433aecd1e6e40' for OpCode BATCH [0x0D]."
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '5ba35c4c9e66e265e86433aecd1e6e40' for OpCode BATCH [0x0D]."
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '178280958907f73c13ff20495c725635' for OpCode BATCH [0x0D]."
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '178280958907f73c13ff20495c725635' for OpCode BATCH [0x0D]."
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '093dcbeeb3fefb75f0ffb397d55c5bd7' for OpCode BATCH [0x0D]."
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '093dcbeeb3fefb75f0ffb397d55c5bd7' for OpCode BATCH [0x0D]."
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '7d36085ccd5725aff8154e710192a7a9' for OpCode EXECUTE [0x0A]."
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '02710454a3fa7d1276ab36701cd768bb' for OpCode BATCH [0x0D]."
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '6130f9ebf9c3c68e3facd2d83f55f07e' for OpCode BATCH [0x0D]."
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '02710454a3fa7d1276ab36701cd768bb' for OpCode BATCH [0x0D]."
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '02710454a3fa7d1276ab36701cd768bb' for OpCode BATCH [0x0D]."
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '02710454a3fa7d1276ab36701cd768bb' for OpCode BATCH [0x0D]."
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '7d29c10e27cd89c2f206d3f93ee7e266' for OpCode BATCH [0x0D]."
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '7d29c10e27cd89c2f206d3f93ee7e266' for OpCode BATCH [0x0D]."
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '7d29c10e27cd89c2f206d3f93ee7e266' for OpCode BATCH [0x0D]."
time="2025-12-10T16:23:52Z" level=warning msg="No cached entry for prepared-id = '7d29c10e27cd89c2f206d3f93ee7e266' for OpCode BATCH [0x0D]."
time="2025-12-10T16:23:52Z" level=info msg="Accepted connection from 172.c.c.6:45200"
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Opening request connection to ORIGIN (172.x.x.1:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Request connection to ORIGIN (172.x.x.1:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Opening request connection to TARGET (172.x.2x.1:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Request connection to TARGET (172.x.2x.1:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="Initiating startup between 172.c.c.6:45200 and 172.x.2x.1:9042 (TARGET)"
time="2025-12-10T16:23:52Z" level=info msg="Handshake successful with client 172.c.c.6:45200"
time="2025-12-10T16:23:52Z" level=info msg="Accepted connection from 172.c.c.2:39092"
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Opening request connection to ORIGIN (172.x.x.2:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Request connection to ORIGIN (172.x.x.2:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Opening request connection to TARGET (172.x.2x.2:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Request connection to TARGET (172.x.2x.2:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="Initiating startup between 172.c.c.2:39092 and 172.x.2x.2:9042 (TARGET)"
time="2025-12-10T16:23:52Z" level=info msg="Handshake successful with client 172.c.c.2:39092"
time="2025-12-10T16:23:52Z" level=info msg="Accepted connection from 172.c.c.4:57484"
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Opening request connection to ORIGIN (172.x.x.4:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Request connection to ORIGIN (172.x.x.4:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Opening request connection to TARGET (172.x.2x.1:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Request connection to TARGET (172.x.2x.1:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="Initiating startup between 172.c.c.4:57484 and 172.x.2x.1:9042 (TARGET)"
time="2025-12-10T16:23:52Z" level=info msg="Handshake successful with client 172.c.c.4:57484"
time="2025-12-10T16:23:52Z" level=info msg="Accepted connection from 172.c.c.9:55624"
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Opening request connection to ORIGIN (172.x.x.3:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Request connection to ORIGIN (172.x.x.3:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Opening request connection to TARGET (172.x.2x.2:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Request connection to TARGET (172.x.2x.2:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="Initiating startup between 172.c.c.9:55624 and 172.x.2x.2:9042 (TARGET)"
time="2025-12-10T16:23:52Z" level=info msg="Handshake successful with client 172.c.c.9:55624"
time="2025-12-10T16:23:52Z" level=info msg="Accepted connection from 172.c.c.8:34830"
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Opening request connection to ORIGIN (172.x.x.1:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Request connection to ORIGIN (172.x.x.1:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Opening request connection to TARGET (172.x.2x.1:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Request connection to TARGET (172.x.2x.1:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="Initiating startup between 172.c.c.8:34830 and 172.x.2x.1:9042 (TARGET)"
time="2025-12-10T16:23:52Z" level=info msg="Handshake successful with client 172.c.c.8:34830"
time="2025-12-10T16:23:52Z" level=info msg="Accepted connection from 172.c.c.1:48460"
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Opening request connection to ORIGIN (172.x.x.2:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[ORIGIN-CONNECTOR] Request connection to ORIGIN (172.x.x.2:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Opening request connection to TARGET (172.x.2x.2:9042)."
time="2025-12-10T16:23:52Z" level=info msg="[TARGET-CONNECTOR] Request connection to TARGET (172.x.2x.2:9042) has been opened."
time="2025-12-10T16:23:52Z" level=info msg="Initiating startup between 172.c.c.1:48460 and 172.x.2x.2:9042 (TARGET)"
time="2025-12-10T16:23:52Z" level=info msg="Handshake successful with client 172.c.c.1:48460"
time="2025-12-10T16:23:53Z" level=warning msg="error sending request with opcode 4f70436f6465204241544348205b307830445d and streamid 5: could not decode raw frame: cannot decode body: cannot decompress body: cannot decompress message: lz4: invalid source or destination buffer too short"
time="2025-12-10T16:23:53Z" level=warning msg="error sending request with opcode 4f70436f6465204241544348205b307830445d and streamid 1: could not decode raw frame: cannot decode body: cannot decompress body: cannot decompress message: lz4: invalid source or destination buffer too short"
time="2025-12-10T16:23:53Z" level=warning msg="error sending request with opcode 4f70436f6465204241544348205b307830445d and streamid 0: could not decode raw frame: cannot decode body: cannot decompress body: cannot decompress message: lz4: invalid source or destination buffer too short"
time="2025-12-10T16:23:53Z" level=warning msg="error sending request with opcode 4f70436f6465204241544348205b307830445d and streamid 2: could not decode raw frame: cannot decode body: cannot decompress body: cannot decompress message: lz4: invalid source or destination buffer too short"

And I attached a screenshot show this behavior. Is there anything specific you want to see?
Starting at 07:30, requested are sent to Proxy, and only few nodes received the read traffic.

Thank you in advance.

[joao-r-reis]

You only have 1 zdm proxy instance running? ZDM tries to "assign" server nodes to each zdm instance in the zdm cluster and each zdm instance will then round robin the server endpoints when a new connection is opened by the driver. If the application driver only opens one connection per node and some zdm proxy instances have more than 1 origin or target node assigned then some nodes will not be used. Try increasing the number of connections per node on your driver configuration.

[joao-r-reis]

Any updates?