Kafka scaler counts retention-deleted messages as lag when offsetResetPolicy: earliest and a partition has no committed offset #7936

Report

When a consumer group has no committed offset for a partition (Kafka returns offset -1) and the trigger sets offsetResetPolicy: earliest with scaleToZeroOnInvalidOffset: false (the default), the scaler reports lag equal to the partition's log end offset, i.e. every offset ever assigned in that partition. Messages before the log start offset have been deleted by retention and can never be consumed, so for long-lived topics this overstates lag by the entire expired history and scales idle workloads out to the topic's partition count.

In our production environment this held idle consumer deployments at elevated replica counts even when no messages were flowing.

Expected Behavior

Lag reflects what a consumer starting from earliest could actually fetch: logEndOffset - logStartOffset. For a partition with an empty retained window, that is 0.

Actual Behavior

Lag is logEndOffset, counting retention-deleted messages as backlog.

Steps to Reproduce the Problem

1. Topic where retention has deleted old messages, so log start offset > 0
2. Kafka trigger with topic set explicitly, offsetResetPolicy: earliest, and scaleToZeroOnInvalidOffset at its default of false. Setting topic is required: when omitted, topics are enumerated from the group's committed offsets, so a group with no commits yields none.
3. Consumer group that has never committed an offset for that partition
4. KEDA reports lag equal to the log end offset, and the workload scales out despite having nothing to consume

KEDA Version

2.20.1

Scaler Details

Kafka (sarama-based kafka trigger)

Would you be open to contributing a fix?

Yes

Anything else?

The docs describe the current behavior and its intent: "In the case of a new topic the last offset will be 0, so it will scale the deployment to 0 replicas." That reasoning only holds for a genuinely new topic, whose log starts at offset 0. Once retention has advanced the log start offset, the same code reports the full log end offset as lag for a partition that has nothing left to consume. Unlike the latest policy path, this branch logs nothing, so there is no operator log entry pointing at the invalid offset. Using logEndOffset - logStartOffset preserves the documented intent for every topic, not just new ones.

This is the root cause of #5274, where the reporter confirmed offsetResetPolicy: earliest and measured lag equal to the sum of the uncommitted partitions' log end offsets. It was closed by the stale bot without a fix.

I have a PR ready, plus a keda-docs PR updating the earliest description.