feat: opt-in time-based closing for tumbling windows on idle streams (#790)#1133
Draft
saigautam9 wants to merge 1 commit into
Draft
feat: opt-in time-based closing for tumbling windows on idle streams (#790)#1133saigautam9 wants to merge 1 commit into
saigautam9 wants to merge 1 commit into
Conversation
Time-based windows only expire when a new message advances event time, so a stream that goes quiet never emits its final window. Add an opt-in close_on_idle flag on tumbling_window() that closes windows on wall-clock time once window_end + grace_ms passes, without waiting for new messages. Scoped to final(closing_strategy="partition"), where windows can be expired for the whole partition by a single time threshold. Default is unchanged (event-time only). Adds TimeWindow.expire_idle() reusing the existing partition expiry, plus validation and unit tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #790 (draft — opening early to get design feedback before finishing the app-loop wiring).
Problem
Time-based windows only expire when a new message advances event time, so a stream that goes quiet never emits its final window. The current workaround is publishing dummy "heartbeat" messages on a timer to force a tumble.
Approach
Opt-in
close_on_idleflag ontumbling_window(). When enabled, windows close on wall-clock time oncewindow_end + grace_mspasses — no new message required. The close-time math is unchanged; only the clock that drives it changes (event time → real time).Scoped to
final(closing_strategy="partition"), the one strategy where a whole partition can be expired by a single time threshold (expire_by_partition) instead of scanning every key. A newTimeWindow.expire_idle(transaction, now_ms)reuses that existing partition expiry:Default is
False, so existing pipelines are unaffected (event-time only).Open questions for maintainers
compose_allonly exposes per-topic root executors — there's no handle to the pipeline stage right after the window. What's the preferred way to route idle-closed windows downstream? This is the main thing gating a non-draft PR.Still to do
expire_idlefrom the empty-poll (rows is None) branchTests
Added under
TestTumblingWindowCloseOnIdle: closes on wall clock pastwindow_end, respects grace, no-op when disabled, and validation rejects the key strategy andcurrent(). Full windows suite passes locally (205 passed).Happy to change direction — if the dummy-heartbeat approach built into the framework is preferred instead, I can do that.