We recently observed what looks like premature policy run end using db_result_size_max, when entries from the candidate set are deleted mid-run.
When db_result_size_max splits a policy run into batches, fill_workers_queue() decides whether to query again by comparing db_current_list_count with the batch size. That counter is incremented only when iter_next() returns an entry.
ListMgr_GetNext(), however, silently skips an ID if its ENTRIES row disappears before the follow-up attribute lookup. If a full batch contains k such rows, the policy loop sees db_result_size_max - k entries and treats the batch as the end of the list, even when more candidates remain.
This race can occur during normal operation when users or another policy remove files. It becomes more likely as batch duration and deletion activity increase.
We saw a pattern consistent with this bug on a Lustre filesystem with about 1.4 billion ENTRIES rows, MariaDB 10.11, and db_result_size_max = 1000000. Users and another policy were unlinking files while changelog processing removed the corresponding database rows.
One possible way to reproduce this would be to:
- configure a policy with at least 25 matching entries,
db_result_size_max = 10, queue_size = 1, one worker, and a slow action.
- start the policy and identify an ID near the end of the initial ten-row candidate result.
- while the feeder is blocked behind the worker queue, delete that row from
ENTRIES in the test database before the feeder consumes it.
The iterator will only deliver 9 entries, and the run report end of list instead of issuing the next candidate query.
Affected candidates are not lost permanently; a later policy trigger can pick them up. But the false ending still reduces throughput, adds an idle gap until the next trigger, and resets per-run continuation state.
We recently observed what looks like premature policy run end using
db_result_size_max, when entries from the candidate set are deleted mid-run.When
db_result_size_maxsplits a policy run into batches,fill_workers_queue()decides whether to query again by comparingdb_current_list_countwith the batch size. That counter is incremented only wheniter_next()returns an entry.ListMgr_GetNext(), however, silently skips an ID if itsENTRIESrow disappears before the follow-up attribute lookup. If a full batch containsksuch rows, the policy loop seesdb_result_size_max - kentries and treats the batch as the end of the list, even when more candidates remain.This race can occur during normal operation when users or another policy remove files. It becomes more likely as batch duration and deletion activity increase.
We saw a pattern consistent with this bug on a Lustre filesystem with about 1.4 billion
ENTRIESrows, MariaDB 10.11, anddb_result_size_max = 1000000. Users and another policy were unlinking files while changelog processing removed the corresponding database rows.One possible way to reproduce this would be to:
db_result_size_max = 10,queue_size = 1, one worker, and a slow action.ENTRIESin the test database before the feeder consumes it.The iterator will only deliver 9 entries, and the run report end of list instead of issuing the next candidate query.
Affected candidates are not lost permanently; a later policy trigger can pick them up. But the false ending still reduces throughput, adds an idle gap until the next trigger, and resets per-run continuation state.