ScaledObject/ScaledJob finalizers can get stuck in Terminating during bulk namespace teardown #7950

Report

When many ScaledObjects (and their scale targets) are deleted concurrently — e.g. during a bulk environment/namespace teardown — some ScaledObjects remain stuck in Terminating indefinitely, blocking namespace deletion. The finalizer.keda.sh finalizer is never removed.

Root cause: finalizer removal (finalizeScaledObject/finalizeScaledJob/FinalizeAuthenticationResource) uses a plain read-modify-write (client.Update()) to drop the finalizer. During bulk teardown, the object's background scale-loop goroutine can still be writing status.conditions concurrently (until stopScaleLoop cancels it), which can race the finalizer-removal Update() and fail it with a resourceVersion conflict. On conflict, the entire reconcile is discarded and requeued on controller-runtime's exponential backoff (capped ~16 min) instead of being retried immediately. Under load (dozens of objects deleted at once), this can recur repeatedly, leaving the finalizer unremoved for 20–30+ minutes in observed cases, requiring a manual kubectl patch to strip the finalizer and unblock namespace deletion.

Note: the accompanying "target deployment doesn't exist" error (from the scale loop's periodic polling, once the scale target Deployment is deleted before its ScaledObject) is a coincidental/cosmetic symptom visible during this window — that error path is structurally decoupled from finalizer removal and does not itself block it.

Expected Behavior

Finalizer removal shouldn't fail an entire reconcile (and fall into long exponential backoff) due to an expected, transient resourceVersion conflict with the object's own status writes. Deletion should complete promptly during bulk teardown without manual finalizer removal.

Actual Behavior

ScaledObjects stuck with finalizer.keda.sh present and deletionTimestamp set for 20+ minutes. Example (sanitized):

metadata:
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2026-07-31T15:12:18Z"
  finalizers:
  - finalizer.keda.sh

Steps to Reproduce the Problem

1. Create a ScaledObject targeting a Deployment, where the Deployment and ScaledObject are both owned/managed by tooling that can delete many resources concurrently (e.g. a GitOps controller cascading a namespace teardown).
2. Trigger deletion of a large batch of such Deployments + ScaledObjects at once, increasing the chance of a concurrent status write racing the finalizer-removal call.
3. Observe: some ScaledObjects remain Terminating with finalizer.keda.sh still present well past their deletionTimestamp, with the scale loop still polling/logging errors about the deleted target.

Logs from KEDA operator

2026-07-31T15:45:39Z ERROR scale_handler error getting state of scaledObject {"scaledObject.Namespace": "<ns>", "scaledObject.Name": "<name>", "error": "error getting scalers cache Deployment.apps \"<name>\" not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
    github.com/kedacore/keda/v2/pkg/scaling/scale_handler.go:269
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
    github.com/kedacore/keda/v2/pkg/scaling/scale_handler.go:197

(repeats every ~pollingInterval, well past deletionTimestamp)

KEDA Version

2.18.3 (root cause confirmed unchanged through 2.20.1/main)

Kubernetes Version

Other

Platform

Other

Scaler Details

Redis (not relevant to root cause — applies regardless of scaler type)

Would you be open to contributing a fix?

Yes