fix(db): don't stop iteration early when a page has fewer rows than size - #14960
Open
Adityaj0 wants to merge 1 commit into
Open
fix(db): don't stop iteration early when a page has fewer rows than size#14960Adityaj0 wants to merge 1 commit into
Adityaj0 wants to merge 1 commit into
Conversation
kong.db.iteration's page_iterator assumed that whenever a strategy returned a non-nil offset, the current page necessarily contained exactly `size` rows, and only fetched the next page once its internal counter passed `size`. This assumption doesn't hold for the `off` (DB-less/Hybrid mode) strategy, which drops expired TTL'd entities from a page while still reporting an offset when more raw entries remain. When a page came back short, the iterator stopped before reaching that offset, silently dropping the remaining entities from DAO:each() / DAO:each_for_export() -- affecting Admin API config export, `kong config db_export`, and db_cache_warmup_entities. Fetch the next page whenever the current page is exhausted and an offset is present, instead of relying on the row count matching size.
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #14959.
kong.db.iteration.page_iteratorassumed that a strategy's page always contains exactlysizerows whenever it also returns a non-niloffset. Theoff(DB-less/Hybrid mode) strategy breaks that assumption: it drops rows from a page when the corresponding entity has expired via TTL, while still returning an offset if more raw entries remain. When a page came back short for that reason, the iterator's continuation check (i > size) never triggered, so it silently stopped instead of fetching the next page -- dropping the remainder of the data set fromDAO:each()/DAO:each_for_export()(used by Admin API config export,kong config db_export, and cache warmup).Changes
kong/db/iteration.lua: fetch the next page whenever the current page is exhausted (rows[i]isnil) and anoffsetis present, instead of additionally requiringi > size. This removes the flawed invariant entirely rather than patching around it, and is behavior-preserving for strategies (e.g. postgres) that do always return exactlysizerows per non-final page: for those,istill only exceeds#rowsoncei > size, exactly matching the old check.spec/01-unit/01-db/14-iteration_spec.lua, a focused unit test that drivesiteration.by_row()with a fake pager returning a short first page (1 row where size=2) with a non-nil offset, and asserts all 3 rows across both pages are yielded.Test plan
ireachesnilati=2whilesize=2, soi > sizeis false and iteration stops after 1 row instead of continuing to fetch the 2nd page).luac -psyntax-checked the changed files.