[Entity Store] Keep per-process extraction state - #290481
Open
kubasobon wants to merge 8 commits into
Open
Conversation
|
🤖 Jobs for this PR can be triggered through checkboxes. 🚧
ℹ️ To trigger the CI, please tick the checkbox below 👇
|
kubasobon
added
release_note:skip
kubasobon
changed the title
kubasobon
changed the title
kubasobon
added this pull request to stack #290482
September 11, 2026 09:30
kubasobon
force-pushed
the
entity-store/extraction-state
branch
from
September 11, 2026 09:43
3e68d5c to
b81043b
Compare
elastic-vault-github-plugin-prod
Bot
requested a review
from a team
as a code owner
September 11, 2026 10:52
Contributor
Saved Objects CI check failed1 issue(s) across 1 type(s).
|
hop-dev
reviewed
Sep 11, 2026
| /** Returns the SO update patch for this extraction mode. Each mode writes to its own cursor field | ||
| * so the two processes do not overwrite each other's position. */ | ||
| private cursorPatch(state: EngineLogExtractionState): Partial<EngineDescriptor> { | ||
| return this.extractionMode === 'nonPriority' |
Contributor
There was a problem hiding this comment.
Not a blocker, but might it be more explicit to map each mode to a state to show that single and priority share?
I am coming to this quite fresh but it took me a moment to figure out that this ternary means single and priority share a cursor field. Would this be overkill?
const CURSOR_FIELD = {
single: 'logExtractionState',
priority: 'logExtractionState',
nonPriority: 'nonPriorityLogExtractionState',
} as const;
private cursorPatch(state: EngineLogExtractionState): Partial<EngineDescriptor> {
return { [CURSOR_FIELD[this.extractionMode]]: state };
}
hop-dev
reviewed
Sep 11, 2026
| @@ -2026,3 +2033,122 @@ describe('LogsExtractionClient mid-slice resume', () => { | |||
| } | |||
| ); | |||
| }); | |||
Contributor
There was a problem hiding this comment.
Could be worth adding a test for when priority is the same bookmark as single. It should resume from logExtractionState and keep writing that field.
kubasobon
force-pushed
the
entity-store/extraction-state
branch
from
September 12, 2026 08:08
30b8068 to
b408d77
Compare
Contributor
💔 Build Failed
Failed CI StepsMetrics [docs]
History
cc @kubasobon |
chennn1990
approved these changes
Sep 14, 2026
chennn1990
left a comment
chennn1990
left a comment
Contributor
There was a problem hiding this comment.
Thank you, great one
Please see my comments
| }); | ||
| expect(result).not.toHaveProperty('nonPriorityLogExtractionState'); | ||
| }); | ||
| }); |
Contributor
There was a problem hiding this comment.
I would remove that file, no need to test an already tested SO management
|
|
||
| /** Maps each extraction mode to its cursor field. single and priority share logExtractionState; | ||
| * nonPriority has its own field so the two processes do not overwrite each other's position. */ | ||
| private static readonly CURSOR_FIELD: Record<ExtractionMode, keyof EngineDescriptor> = { |
Contributor
There was a problem hiding this comment.
Maybe CURSOR_FIELD is a bit specific, I would rename it to any constant indicate we aim to achieve the log extraction state.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #288602. Stacked on
entity-store/layered-config(#289855).The entity store keeps one extraction cursor per engine under
logExtractionState. When twoprocesses run against the same engine (#288601), they share that field and one process overwrites
the other's cursor — logs get skipped or reprocessed silently.
This adds a second, independent cursor field
nonPriorityLogExtractionState(model version 9),and routes each process to its own field based on
extractionMode.