Entity store/force run maintainer - #3
Closed
chennn1990 wants to merge 9 commits into
Closed
chennn1990 wants to merge 9 commits into
chennn1990 wants to merge 9 commits into
Conversation
1 task
…rce-run-maintainer
…rce-run-maintainer
chennn1990
pushed a commit
that referenced
this pull request
Jun 30, 2026
…mp (elastic#273513) ## Summary Addresses elastic#273086 (follow-ups #2 and #3 to elastic#273418). Several Synthetics queries that target a specific `monitor.check_group` carried **no `@timestamp` bound**, so Elasticsearch could not prune shards during the `can_match` pre-filter and fanned the query out across **all** `synthetics-*` backing indices — including the deliberately-throttled **frozen** tier (searchable snapshots). On clusters with long frozen retention (e.g. `hot: 3d / warm: 4d / frozen: 365d`) this caused slow screenshot/history loading and contributed to cluster-wide **search thread pool rejections**. All documents of a single Synthetics check run (summary, steps, screenshots, network events) are written at essentially the same instant, so any query scoped to one `check_group` can safely be bounded to a **wide window around that run's `@timestamp`** without risking dropping a run's documents — while still letting `can_match` skip unrelated (frozen) shards. > This is complementary to elastic#273418: that PR added opt-in tier exclusion (`observability:searchExcludedDataTiers`) at the `SyntheticsEsClient` choke point; this PR makes the per-run queries naturally prunable even when an operator has not opted in. ### What changed - New shared helper `getCheckGroupTimeRangeFilter(timestamp)` in `common/constants/client_defaults.ts` — builds a `@timestamp` range of `[ts - 1h, ts + 1h]` (the buffer is far larger than any realistic journey duration, so it can never drop a run's docs). - **Screenshots** (`get_journey_screenshot`): accepts an optional `timestamp` and bounds the query. Threaded end-to-end from the UI: `getScreenshotUrl` → `JOURNEY_SCREENSHOT` route → handler. The run timestamp already reaches `JourneyStepScreenshotContainer` / `JourneyLastScreenshot`, so the overview "last screenshot" (the customer's main symptom) is now bounded. - **Steps** (`get_journey_steps`): accepts an optional `timestamp` and bounds the query. Threaded via `useJourneySteps` → `fetchBrowserJourney` → `JOURNEY` route. Wired from `JourneyLastScreenshot`, `LastTestRun`, and the expanded successful-result row. - **Last successful check** (`last_successful_check` route): bounds its follow-up screenshot lookup to the resolved check's own `@timestamp` (fully self-contained — the route already has the timestamp). - **#3 — Recent pings** (`fetchMonitorRecentPings`): reduced the unused fallback window from **30 days → 24h** for when callers don't pass an explicit range (the only caller, `TestRunsTable`, always passes the UI date range, so this is hygiene on the fallback path). ### Behavior / risk - All new `timestamp` parameters are **optional**; when absent, queries behave exactly as before, so this is backward-compatible for any caller that doesn't pass one. - Bounding is provably non-lossy: a check group's documents share the run instant, and the `± 1h` window dwarfs any journey duration. ### Remaining follow-ups (still tracked in elastic#273086, not in this PR) - Network events (`get_network_events`) and screenshot blocks (`get_journey_screenshot_blocks`) — same optional-`timestamp` pattern; deferred to keep this PR focused. - Latest-ping hooks (`useRemoteMonitorLatestPing`, `use_status_by_location`, `use_monitor_detail`) — these need a `24h→7d→30d` fallback (a hard bound could hide infrequently-run monitors), better done alongside `get_latest_test_run`. ## Testing - `node scripts/jest` for the changed areas — added unit tests: - `get_journey_screenshot.test.ts` / `get_journey_steps.test.ts`: query is bounded when `timestamp` is provided and unchanged when absent. - `journey_screenshot_dialog.test.tsx`: `getScreenshotUrl` appends a url-encoded `timestamp` (and both `remoteName` + `timestamp`). - `browser_journey/api.test.ts`: `fetchBrowserJourney` forwards `timestamp`. - `node scripts/type_check --project x-pack/solutions/observability/plugins/synthetics/tsconfig.json` — passes. - `node scripts/eslint --fix $(git diff --name-only)` — clean. ### Checklist - [x] Unit tests updated/added for the changed query paths. - [ ] [Release Notes](https://www.elastic.co/guide/en/kibana/master/release-notes.html) — performance improvement. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
chennn1990
pushed a commit
that referenced
this pull request
Jul 23, 2026
…tic#273764) (elastic#275378) # Backport This will backport the following commits from `main` to `9.4`: - [[Lens as code] Fix partial cumulative_sum transformations (elastic#273764)](elastic#273764) <!--- Backport version: 11.0.2 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Marco Vettorello","email":"marco.vettorello@elastic.co"},"sourceCommit":{"committedDate":"2026-06-19T12:26:22Z","message":"[Lens as code] Fix partial cumulative_sum transformations (elastic#273764)\n\n## Summary\n\nWith the API flag on, opening saved Lens visualizations failed when a\n`cumulative_sum` referenced a hidden count of records column.\n`toAPIFormat()` only accepted a `sum` reference and threw, causing 500s\non GET /internal/lens/visualizations/{id}\nLike in the case `#3` in https://github.com/elastic/kibana/issues/268821\nwhere the Lens stress-test was using this configuration.\n\nThe fix changed the cumulative_sum API transforms by fixing the\nreference resolution following this:\n\nif the field is `___records___` → underlying ref is `count`\nany other field → underlying ref is `sum`\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"7a9b595ea613c440a96a98a7009ecb6e486c536c","branchLabelMapping":{"^v9.5.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","Team:Visualizations","release_note:skip","backport missing","backport:version","v9.5.0","v9.4.3"],"title":"[Lens as code] Fix partial cumulative_sum transformations","number":273764,"url":"https://github.com/elastic/kibana/pull/273764","mergeCommit":{"message":"[Lens as code] Fix partial cumulative_sum transformations (elastic#273764)\n\n## Summary\n\nWith the API flag on, opening saved Lens visualizations failed when a\n`cumulative_sum` referenced a hidden count of records column.\n`toAPIFormat()` only accepted a `sum` reference and threw, causing 500s\non GET /internal/lens/visualizations/{id}\nLike in the case `#3` in https://github.com/elastic/kibana/issues/268821\nwhere the Lens stress-test was using this configuration.\n\nThe fix changed the cumulative_sum API transforms by fixing the\nreference resolution following this:\n\nif the field is `___records___` → underlying ref is `count`\nany other field → underlying ref is `sum`\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"7a9b595ea613c440a96a98a7009ecb6e486c536c"}},"sourceBranch":"main","suggestedTargetBranches":["9.4"],"targetPullRequestStates":[{"branch":"main","label":"v9.5.0","branchLabelMappingKey":"^v9.5.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/273764","number":273764,"mergeCommit":{"message":"[Lens as code] Fix partial cumulative_sum transformations (elastic#273764)\n\n## Summary\n\nWith the API flag on, opening saved Lens visualizations failed when a\n`cumulative_sum` referenced a hidden count of records column.\n`toAPIFormat()` only accepted a `sum` reference and threw, causing 500s\non GET /internal/lens/visualizations/{id}\nLike in the case `#3` in https://github.com/elastic/kibana/issues/268821\nwhere the Lens stress-test was using this configuration.\n\nThe fix changed the cumulative_sum API transforms by fixing the\nreference resolution following this:\n\nif the field is `___records___` → underlying ref is `count`\nany other field → underlying ref is `sum`\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"7a9b595ea613c440a96a98a7009ecb6e486c536c"}},{"branch":"9.4","label":"v9.4.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run entity maintainer (internal API)
Adds an internal API to run an entity maintainer task immediately by id (without waiting for its schedule).
Behaviour
POST/internal/security/entity_store/entity_maintainers/run/{id}Entity maintainer not found.{ "ok": true }. The task is run as soon as possible via Task Manager'srunSoon. The maintainer must already be initialized (task document must exist); otherwise the run may fail.How to run it (Kibana Console)
In Kibana Console (Dev Tools), run: