Describe the bug
In our production environment, we have observed a state consistency issue where tasks are marked as ACTIVE but their corresponding lease records are missing from the queue.
These tasks appear to be "orphaned"—they hold an active status indefinitely but are never picked up by workers or the scheduler for re-execution. This leads to a permanent deadlock for these specific tasks.
Expected behavior
If a task is ACTIVEbut its lease expires or is lost, the system should:
Detect the inconsistency.
Automatically transition the task back to a PENDINGstate.
Allow it to be rescheduled and retried by available workers.
Actual Behavior
The scheduler seems unable to detect these "orphaned" tasks. Since they are technically still ACTIVE, they are not picked up for retry logic, effectively hanging forever.
Additional
During the service restart, it will be removed from the whitelist and lose access to Redis (existing connections may persist briefly while new connections will fail).
Describe the bug
In our production environment, we have observed a state consistency issue where tasks are marked as ACTIVE but their corresponding lease records are missing from the queue.
These tasks appear to be "orphaned"—they hold an active status indefinitely but are never picked up by workers or the scheduler for re-execution. This leads to a permanent deadlock for these specific tasks.
Expected behavior
If a task is ACTIVEbut its lease expires or is lost, the system should:
Detect the inconsistency.
Automatically transition the task back to a PENDINGstate.
Allow it to be rescheduled and retried by available workers.
Actual Behavior
The scheduler seems unable to detect these "orphaned" tasks. Since they are technically still ACTIVE, they are not picked up for retry logic, effectively hanging forever.
Additional
During the service restart, it will be removed from the whitelist and lose access to Redis (existing connections may persist briefly while new connections will fail).