[Threat Intel 11/11] Promote and scrub tasks, and plugin wiring behind the flag - #287207
Merged
jonwalstedt merged 52 commits intoSep 11, 2026
Merged
Conversation
stephmilovic
added
release_note:skip
This was referenced Aug 25, 2026
[Threat Intel 1/11] Gate enrichment completion on extract_iocs, drop detection_actionability
#287197
Merged
stephmilovic
force-pushed
the
threat-intel-9-tasks-and-wiring
branch
from
August 26, 2026 20:46
6de32b5 to
4df28f1
Compare
jonwalstedt
force-pushed
the
threat-intel-9-tasks-and-wiring
branch
from
August 28, 2026 20:00
7a67591 to
d939706
Compare
stephmilovic
force-pushed
the
threat-intel-9-tasks-and-wiring
branch
5 times, most recently
from
September 4, 2026 14:47
0bb3dd3 to
3a0600c
Compare
stephmilovic
changed the base branch from
main
to
threat-intel-8-adapters-rest
September 4, 2026 14:49
stephmilovic
force-pushed
the
threat-intel-8-adapters-rest
branch
from
September 4, 2026 15:15
3c3f8bc to
3ad1b0b
Compare
stephmilovic
force-pushed
the
threat-intel-9-tasks-and-wiring
branch
from
September 4, 2026 15:15
3a0600c to
b074bec
Compare
stephmilovic
force-pushed
the
threat-intel-8-adapters-rest
branch
from
September 4, 2026 15:25
3ad1b0b to
b53abe0
Compare
stephmilovic
force-pushed
the
threat-intel-9-tasks-and-wiring
branch
from
September 4, 2026 15:25
b074bec to
5dcf41a
Compare
stephmilovic
force-pushed
the
threat-intel-8-adapters-rest
branch
from
September 4, 2026 15:41
b53abe0 to
3dc08d0
Compare
stephmilovic
force-pushed
the
threat-intel-9-tasks-and-wiring
branch
from
September 4, 2026 15:41
5dcf41a to
c710fc4
Compare
botelastic
Bot
added
the
Team:One Workflow
… param, step handler tests - Guard notes/knownRansomwareCampaignUse against non-strings so a malformed KEV row can't throw in buildFingerprint and fail the whole feed - Carry source._source.name on KEV reports instead of a hardcoded string - Add tierBasis param to parseIndicatorList (default maltrail_indicator_list) - Export IocEntry from fetch_source_common instead of deriving it - Drop the tautological kev enrich isolation test - Add fetch_source_step handler tests (string source, UnknownAdapterError, generic wrap, happy path) - Drop the silent globe icon fallback on the fetch_source step
…ters-rest # Conflicts: # x-pack/solutions/security/plugins/security_solution/server/threat_intel/adapters/fingerprint.test.ts # x-pack/solutions/security/plugins/security_solution/server/threat_intel/adapters/rss/rss_adapter.test.ts # x-pack/solutions/security/plugins/security_solution/server/threat_intel/adapters/types.ts
stephmilovic
added a commit
to stephmilovic/kibana
that referenced
this pull request
Sep 8, 2026
Moving to elastic#287207, the tasks-and-wiring PR, since that one already requests security-threat-hunting review and this bug sits on the bootstrap path that PR is wiring up. Keeping it here would add a reviewer this PR doesn't otherwise need. This reverts commit e884427 and b461149.
Sorting the legacy-source disable scan on _id requires fielddata on _id, which Elasticsearch disallows by default, so this threw illegal_argument_exception and failed the whole bootstrap on a stock cluster. _doc gives a stable total order for an exhaustive scan without needing fielddata; order only matters here as a pagination tie-breaker, not for correctness. The ES client types want the string form (['_doc']), not the object form. Found while getting elastic#289343's Scout suite green; folding it in here since this PR already needs security-threat-hunting review and elastic#289343 doesn't otherwise need it.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Reviewed the promote/scrub tasks, plugin wiring, adapters, and the seed bootstrap fix. The task and wiring code is well-defensed (idempotent writes, retryable-vs-permanent bulk classification, PIT-based stable scan, fail-closed bootstrap gate). One concrete concern on the legacy-source disable scan's pagination stability — left inline.
Generated by Claude Reviewer for #287207 · claude · opus · 275.8 AIC · ⌖ 22.6 AIC · ⊞ 5.5K
Replaces the paged search/update loop (which sorted on _doc and risked skipping documents across refreshes/merges) with a single updateByQuery call using conflicts: 'proceed', matching the existing scrub_report_content pattern. Removes LEGACY_SOURCE_DISABLE_PAGE_SIZE and the unstable _doc sort. Rewrites the unit tests to mock updateByQuery instead of search/ update pagination.
stephmilovic
force-pushed
the
threat-intel-9-tasks-and-wiring
branch
from
September 8, 2026 22:03
dcc0b24 to
30191e2
Compare
Contributor
Author
|
@elasticmachine merge upstream |
This was referenced Sep 9, 2026
stephmilovic
force-pushed
the
threat-intel-9-tasks-and-wiring
branch
from
September 9, 2026 21:37
1fdc1d2 to
a8d8670
Compare
attribute_alerts_to_reports installs per space (workflowIdSuffix) because it queries .alerts-security.alerts-* and writes per-space hit totals; a global install would land every space's totals on one shared document. ingest_threat_feeds and enrich_threat_report install once globally, since the source catalog and the LLM enrichment are identical for every space. Alert analysis and threat intel now install through one installSecurityManagedWorkflowsAndMarkReady that calls ready() once, so neither install set is silently dropped. New spaces are covered by a reconcile pass on the promote_threat_indicators task. All three workflows ship enabled: false, and enrich routes its HTTP calls through a fixed space (default) instead of the install-time workflow.spaceId, which is '*' globally and produced invalid /s/*/... routes. Co-authored-by: Cursor <cursoragent@cursor.com>
stephmilovic
force-pushed
the
threat-intel-9-tasks-and-wiring
branch
from
September 9, 2026 21:44
a8d8670 to
eb25c98
Compare
Contributor
There was a problem hiding this comment.
Reviewed the threat-intel wiring, promote/scrub tasks, and managed-workflow install. The change is well-structured and thoroughly tested; the flag-off/flag-on wiring guarantees and the retryable-vs-permanent bulk classification in the promote task are solid. One concrete finding on per-space enumeration is left inline. The cross-space isolation caveat is already acknowledged in the PR and gated behind the default-off flag.
Generated by Claude Reviewer for #287207 · claude · opus · 183.6 AIC · ⌖ 22.6 AIC · ⊞ 5.5K
Shortens several multi-paragraph rationale comments in promote_threat_indicators.ts (MAX_SOURCE_CITATIONS, PROMOTABLE_TIERS, canonicalIndicatorValue, indicatorId, isRetryableBulkFailure) to their load-bearing conclusions, keeping the why but cutting the derivation. Adds a shared isTransientEsStatus helper (threat_intel/lib/es_retry.ts) so the promote and scrub tasks classify retryable ES statuses the same way instead of each inlining its own 503/429 (or 503/429/500/408) check. This also fixes a minor inconsistency: the promote task's PIT-open and search-scan error handlers now treat 500/502/504/408 as retryable too, matching the bulk-item classification, instead of only 503/429. Also factors the repeated uninstall-and-tolerate-not-found block in install.ts into a single uninstallTolerant helper. No behavior change other than the retryable-status broadening above. Co-authored-by: Cursor <cursoragent@cursor.com>
Promote is a recurring task, and Task Manager deletes a recurring task's saved object when a run throws an unrecoverable error: rescheduleFailedRun checks isUnrecoverableError before it looks at the schedule, and processResultForRecurringTask then calls removeTask(). Nothing re-creates this task except startThreatIntel, so a single 403 or socket hang up stopped promotion in every space until Kibana restarted. The transport case was the likely one. ConnectionError, TimeoutError, and RequestAbortedError carry no statusCode at all, only ResponseError does, so the routine 2m task timeout landed in that branch. Replaces the calls with a throwForNextRun helper that retries sooner on a transient status and otherwise throws plainly, and the search and bulk paths now check signal.aborted before classifying, so a timeout is a graceful stop that holds the cursor. Five more fixes in the same area: ioc_tier was frozen at its first citation, because the upsert document is ignored on an update. A value first cited as uncertain stayed out of the precision alias no matter how many later reports called it discriminating. It is now refreshed best-tier-wins in the script, next to the identical severity logic, and the rank map is derived from the same ordered list the membership gate uses so the two cannot drift. Scrub read only update_by_query's updated count. Per-document problems come back in failures[] with a 200, so a run that failed on every document returned updated: 0 and looked like a clean no-op while third-party body text sat past its retention window. Failures are now counted in task state and logged at error. Seed reported version_conflicts as failed. A non-zero failed escalates to a bootstrap rejection, which gates every threat intel route behind a 503 and leaves both tasks unscheduled, so a benign write race could do that. Only real failures[] entries count now, and those were previously read by nothing at all. The workflow install loops were bare awaits. One failing space skipped every install after it on every pass, and the caller's warn swallowed the rejection, so a deployment one space deep looked fully covered. Each install is isolated now, with a summary naming the spaces left without alert attribution. uninstallTolerant logs at debug instead of swallowing, so a 403 or 5xx is not as invisible as an expected not-found. enumerateSpaceIds fetched a single page of 1000 and dropped the tail. Every fix has a test that fails when the fix is reverted, checked by mutating each one in place. Also narrows uninstallTolerant's workflowId off string, which did not type check. Co-authored-by: Cursor <cursoragent@cursor.com>
jonwalstedt
reviewed
Sep 10, 2026
jonwalstedt
left a comment
jonwalstedt
left a comment
Contributor
There was a problem hiding this comment.
Nice work, one finding that needs addressing though
load_sources and check_dedup still filtered on workflow.spaceId plus '*', a leftover from before this workflow moved to a single global install. Now that workflow.spaceId is pinned to '*' for that install, the filter only ever matches '*', so a future space-owned custom source would never be fetched or deduped by this single instance even though the adapters already stamp reports with the source's own space_id. Drop the space_id filter from both steps so enabled is the only gate, the same treatment enrich_threat_report already got. No behavior change today since every source is global; this just stops the global install from silently boxing out space-owned sources later. Co-authored-by: Cursor <cursoragent@cursor.com>
routeSpaceId was already hardcoded to 'default' to avoid an invalid /s/*/... route once this workflow moved to a single global install, but the comment only explained the routing mechanics, not the consequence: the enrich routes resolve their LLM connector through a per-space saved object (Model Settings, namespaceType: single) and the per-space genAi:defaultAIConnector setting, so pinning routeSpaceId to 'default' makes model choice deployment-wide regardless of a report's real space. Spell that out as an accepted MVP tradeoff (consistent with sources being global-only for now) and point at the future upgrade path: resolving routeSpaceId per report inside extract_each_report from that report's own space_id, once per-space model settings matter. Tracked as kibana-6vs. No functional change. Co-authored-by: Cursor <cursoragent@cursor.com>
Skip enumerateSpaceIds and the per-space uninstall loop when neither global threat intel workflow is persisted, collapsing every flag-off boot on a deployment that never enabled threat intel supply from N+3 sequential no-op calls to 2. Co-authored-by: Cursor <cursoragent@cursor.com>
jonwalstedt
approved these changes
Sep 10, 2026
A flag-off boot that only checked the two global workflows skipped leftover per-space attribute_alerts_to_reports instances. Co-authored-by: Cursor <cursoragent@cursor.com>
text_indicator_list stamps one extracted_at across chunks, so gt skipped siblings after a completed PIT. uninstallTolerant only runs after the leftover probe, so debug hid real 403/5xx. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
💛 Build succeeded, but was flaky
Failed CI Steps
Metrics [docs]Async chunks
Page load bundle
Unknown metric groupsasync chunk count
shared chunks total size
total optimizer output size
warm start memory
Test Failures
History
|
stephmilovic
added a commit
to stephmilovic/kibana
that referenced
this pull request
Sep 11, 2026
Two cleanups to workflow yaml that landed in elastic#287207: - Concurrency keys were -global suffixed (threat-intel-enrich-global, threat-intel-ingest-global). If these workflows ever become space aware and a real space is literally named "global", a space running the old global-only version would collide with that space's key. Drop the suffix entirely (threat-intel-enrich, threat-intel-ingest) so there is no string that a future space name can collide with. - enrich_threat_report.yaml routed its kibana.request calls through /s/{{ variables.routeSpaceId }}/..., with routeSpaceId hardcoded to "default". A request path with no /s/{id}/ prefix already resolves to the default space, so this produced identical behavior through an unnecessary variable. Drop routeSpaceId and call the routes directly. Also trimmed the enrich_threat_report.yaml header, which had grown long explaining routeSpaceId specifically; the MVP model-settings tradeoff it described still holds and is kept, just without the removed variable. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires Threat Intelligence behind
threatIntelSupplyEnabled: bootstrap, managed workflow installation, promote/scrub tasks, and route registration. Promote mirrors extracted IOCs into.threat-intel-indicatorsfor Indicator Match rules.Where this sits
#287479 came first in the merge train and is already on
main. The numbered series (#287197–#287205) is merged; #287206 and #287207 now sit directly onmain. #287199 (content parsing) was closed; IOC extraction ships its own section-header classifier.Scope review follow-up
attribute_alerts_to_reportsinstalls per space (it queries.alerts-security.alerts-*and writes per-space hit totals, so a global install would clobber every space's totals onto one shared doc), whileingest_threat_feedsandenrich_threat_reportinstall once globally. Alert analysis and threat intel share a singleready()call so neither install set is dropped, and spaces created after boot are reconciled on the promote task. All three shipenabled: false, and enrich routes its HTTP calls through a fixed space (default) instead of the install-timeworkflow.spaceId(which is'*'globally).Bootstrap fix folded in from #289343
seed_default_sources.tssorted the legacy-source disable scan on_id, which Elasticsearch rejects withillegal_argument_exception: Fielddata access on the _id field is disallowedunlessindices.id_field_data.enabledis set, so it failed bootstrap on a stock cluster. Found this while getting #289343's Scout suite green and moved the fix here since this PR is the one wiring up bootstrap and already needssecurity-threat-huntingreview, so it isn't adding a reviewer #289343 wouldn't otherwise need.Product boundary
threatIntelSupplyEnableddefaults to false. Do not enable until direct-index cross-space isolation is hardened or the administrator trust model is explicitly accepted.To test
PR developed with Cursor + Auto + Sonnet 5 + Opus 4.8