[Threat Intel 10/11] Remaining adapters, dispatcher, and fetch_source step by stephmilovic · Pull Request #287206 · elastic/kibana

stephmilovic · 2026-08-25T22:32:23Z

Summary

Adds the remaining Threat Intelligence adapters (KEV, text indicator list) and the fetch_source workflow step. Dispatch accepts only rss, kev, and text_indicator_list.

Where this sits

#	PR	Depends on	Status
1	#287197 workflow gate correction + fixtures	nothing	✅ merged
2	#287198 contracts, constants, shared libs	nothing	✅ merged
3	#287199 content parsing	#287198	closed
4	#287200 SSRF-guarded HTTP client	#287198	✅ merged
5	#287201 index templates, seeding, inference features	#287198, #287200	✅ merged
6	#287202 indicator alias	#287198, #287201	✅ merged
7	#287203 IOC extraction	#287198, #287200	✅ merged
8	#287204 LLM services, routes, source catalog API	#287198, #287200, #287202, #287203	✅ merged
9	#287205 RSS adapter	#287198, #287200, #287203, #287204	✅ merged
10	#287206 remaining adapters, dispatcher, fetch_source step ← this PR	#287198, #287200, #287203, #287205	✅ merged
11	#287207 promote/scrub tasks and plugin wiring	#287198–#287206	✅ merged
12	#287479 generator fixture article URLs	nothing	✅ merged
13	#289343 Scout API tests for the deterministic routes	#287204	ready for review
14	#289345 enrichment eval suite	#287204	ready for review

#287479 came first in the merge train and is already on main. The numbered series (#287197–#287205) is merged; #287206 and #287207 now sit directly on main. #287199 (content parsing) was closed; IOC extraction ships its own section-header classifier.

Scope review follow-up

Removed the vendor API adapter and generic JSON-list machinery.
Runtime-validates KEV entries and skips malformed rows without failing the whole feed.
Normalizes text-list and KEV provenance through the shared URL policy.

To test

node scripts/jest --config x-pack/solutions/security/plugins/security_solution/server/threat_intel/jest.config.js adapters/kev adapters/text_indicator_list adapters/run_adapter.test.ts

PR developed with Cursor + Auto

Adapters in the RSS/STIX/TAXII (elastic#287205) and vendor_api (elastic#287206) slices import collapseWhitespace from content/text.ts. The content-parsing rewrite made it a private const, which breaks those imports once this PR merges. It's a plain string utility with real external callers, not part of the parsing-architecture change, so restore the export.

… module dependency Transplant the URL canonicalizer into the text-list adapter as adapters/text_indicator_list/canonicalize_url.ts (its only remaining consumer) and repoint the KEV, text-list, and vendor adapters at the services/ severity and report-content helpers now that content/ is gone. Drop the operator-created source shape from the vendor adapter: remove the config.vendor override so a vendor_api source resolves its handler only by the fixed catalog id, and delete the override test. KEV and vendor keep rejecting responses that don't match their declared JSON shape rather than treating them as web pages. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

…de catalog After elastic#287201, feed URLs live in catalog_source_urls rather than source config. Co-authored-by: Cursor <cursoragent@cursor.com>

Removed two box-drawing section banners in the indicator-list tests and a comment restating that the loop skips blank lines.

The vulnerabilities-array check sat inside the try whose catch relabels everything as "not valid JSON", so a well-formed feed that simply lost the array reported a parse error and sent an operator debugging a feed schema change into the parser instead. Splitting the parse from the shape check also drops the second array check below it, which line 205 had already made unreachable, and its catalogVersion error message with it. No test covered that branch. Also use GLOBAL_SPACE_ID for the space fallback, matching the other three call sites. Same value today, but a bare '*' would silently survive a rename of the sentinel.

infra-vault-gh-plugin-prod · 2026-09-04T20:57:59Z

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

infra-vault-gh-plugin-prod · 2026-09-04T20:58:02Z

Pinging @elastic/security-solution (Team: SecuritySolution)

github-actions

Reviewed the new adapters, dispatcher, and fetch_source step. One concrete finding on the KEV adapter: two optional feed fields bypass runtime type-checking and can crash the whole feed run via buildFingerprint, which undercuts the malformed-row resilience this PR is adding. Details inline.

Generated by Claude Reviewer for #287206 · opus · 217.8 AIC · ⌖ 14.4 AIC · ⊞ 5.1K

stephmilovic · 2026-09-08T13:25:39Z

@elasticmachine merge upstream

… param, step handler tests - Guard notes/knownRansomwareCampaignUse against non-strings so a malformed KEV row can't throw in buildFingerprint and fail the whole feed - Carry source._source.name on KEV reports instead of a hardcoded string - Add tierBasis param to parseIndicatorList (default maltrail_indicator_list) - Export IocEntry from fetch_source_common instead of deriving it - Drop the tautological kev enrich isolation test - Add fetch_source_step handler tests (string source, UnknownAdapterError, generic wrap, happy path) - Drop the silent globe icon fallback on the fetch_source step

…ters-rest # Conflicts: # x-pack/solutions/security/plugins/security_solution/server/threat_intel/adapters/fingerprint.test.ts # x-pack/solutions/security/plugins/security_solution/server/threat_intel/adapters/rss/rss_adapter.test.ts # x-pack/solutions/security/plugins/security_solution/server/threat_intel/adapters/types.ts

kibanamachine · 2026-09-08T17:13:14Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: b0cf683
Build duration: 55 mins
Storybooks Preview

Failed CI Steps

Metrics [docs]

✅ unchanged

History

💔 Build #498410 failed 5627e13
💚 Build #497251 succeeded 0995b58
💔 Build #497169 failed a981ff5

jonwalstedt

Nice! all comments addressed, looks good 👍

…d the flag (#287207) ## Summary Wires Threat Intelligence behind `threatIntelSupplyEnabled`: bootstrap, managed workflow installation, promote/scrub tasks, and route registration. Promote mirrors extracted IOCs into `.threat-intel-indicators` for Indicator Match rules. ## Where this sits | # | PR | Depends on | Status | |---|---|---|---| | 1 | #287197 workflow gate correction + fixtures | nothing | ✅ merged | | 2 | #287198 contracts, constants, shared libs | nothing | ✅ merged | | 3 | #287199 content parsing | #287198 | closed | | 4 | #287200 SSRF-guarded HTTP client | #287198 | ✅ merged | | 5 | #287201 index templates, seeding, inference features | #287198, #287200 | ✅ merged | | 6 | #287202 indicator alias | #287198, #287201 | ✅ merged | | 7 | #287203 IOC extraction | #287198, #287200 | ✅ merged | | 8 | #287204 LLM services, routes, source catalog API | #287198, #287200, #287202, #287203 | ✅ merged | | 9 | #287205 RSS adapter | #287198, #287200, #287203, #287204 | ✅ merged | | 10 | #287206 remaining adapters, dispatcher, fetch_source step | #287198, #287200, #287203, #287205 | ✅ merged | | 11 | #287207 promote/scrub tasks and plugin wiring **← this PR** | #287198–#287206 | ready for review | | 12 | #287479 generator fixture article URLs | nothing | ✅ merged | | 13 | #289343 Scout API tests for the deterministic routes | #287204 | ready for review | | 14 | #289345 enrichment eval suite | #287204 | ready for review | | 15 | #290144 read APIs, readiness, space-keyed attribution | #287207 | draft | #287479 came first in the merge train and is already on `main`. The numbered series (#287197–#287205) is merged; #287206 and #287207 now sit directly on `main`. #287199 (content parsing) was closed; IOC extraction ships its own section-header classifier. ## Scope review follow-up - Installs the managed workflows with the right space topology: `attribute_alerts_to_reports` installs per space (it queries `.alerts-security.alerts-*` and writes per-space hit totals, so a global install would clobber every space's totals onto one shared doc), while `ingest_threat_feeds` and `enrich_threat_report` install once globally. Alert analysis and threat intel share a single `ready()` call so neither install set is dropped, and spaces created after boot are reconciled on the promote task. All three ship `enabled: false`, and enrich routes its HTTP calls through a fixed space (`default`) instead of the install-time `workflow.spaceId` (which is `'*'` globally). - Ensures the default-space indicator alias on start. - Chunks promotion bulk writes and treats HTTP 408/500 as retryable bulk failures. - Sanitizes provenance and extracted IOC reference URLs during promotion. - Documents direct-index cross-space isolation as the blocker to enabling the feature. ## Bootstrap fix folded in from #289343 `seed_default_sources.ts` sorted the legacy-source disable scan on `_id`, which Elasticsearch rejects with `illegal_argument_exception: Fielddata access on the _id field is disallowed` unless `indices.id_field_data.enabled` is set, so it failed bootstrap on a stock cluster. Found this while getting #289343's Scout suite green and moved the fix here since this PR is the one wiring up bootstrap and already needs `security-threat-hunting` review, so it isn't adding a reviewer #289343 wouldn't otherwise need. ## Product boundary `threatIntelSupplyEnabled` defaults to false. Do not enable until direct-index cross-space isolation is hardened or the administrator trust model is explicitly accepted. ## To test ``` node scripts/jest --config x-pack/solutions/security/plugins/security_solution/server/threat_intel/jest.config.js wiring.test.ts tasks/promote_threat_indicators.test.ts node scripts/jest x-pack/solutions/security/plugins/security_solution/server/workflows/ node scripts/jest src/platform/packages/shared/kbn-workflows/managed/definitions/threat_intel/ ``` _PR developed with Cursor + Auto + Sonnet 5 + Opus 4.8_ --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Jon Wålstedt <jon.walstedt@elastic.co> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

stephmilovic force-pushed the threat-intel-8-adapters-rest branch from a9d5fc2 to 3ebfa61 Compare August 26, 2026 20:45

jonwalstedt force-pushed the threat-intel-8-adapters-rest branch from 1bdfcc8 to 41d8ba8 Compare August 28, 2026 20:00

stephmilovic force-pushed the threat-intel-8-adapters-rest branch from 41d8ba8 to f75611a Compare September 1, 2026 20:06

stephmilovic mentioned this pull request Sep 1, 2026

[Threat Intel] Move the indicator index out of the .kibana namespace #287479

Merged

stephmilovic force-pushed the threat-intel-8-adapters-rest branch 4 times, most recently from 426380f to 3c3f8bc Compare September 4, 2026 14:47

stephmilovic changed the base branch from main to threat-intel-7-adapters-core September 4, 2026 14:49

stephmilovic force-pushed the threat-intel-7-adapters-core branch from b2b0d71 to a3f1f70 Compare September 4, 2026 15:15

stephmilovic force-pushed the threat-intel-8-adapters-rest branch from 3c3f8bc to 3ad1b0b Compare September 4, 2026 15:15

stephmilovic force-pushed the threat-intel-7-adapters-core branch from a3f1f70 to 671f1cd Compare September 4, 2026 15:25

stephmilovic force-pushed the threat-intel-8-adapters-rest branch from 3ad1b0b to b53abe0 Compare September 4, 2026 15:25

stephmilovic force-pushed the threat-intel-7-adapters-core branch from 671f1cd to 2b32fdf Compare September 4, 2026 15:41

jonwalstedt and others added 10 commits September 4, 2026 14:01

Remove unreachable threat intel adapters

53c2ab4

Remove dead text-list reference metadata

f27260f

Remove dead KEV reference metadata

996c632

Satisfy text-list chunking lint

ad9d322

Remove dead text-list reference classification

7208883

Address scope-review findings from deep PR review

3f61825

Co-authored-by: Cursor <cursoragent@cursor.com>

fix(threat-intel): resolve KEV and text-list adapter URLs from the co…

3edf6ea

…de catalog After elastic#287201, feed URLs live in catalog_source_urls rather than source config. Co-authored-by: Cursor <cursoragent@cursor.com>

Deslop: drop decorative section banners and a no-op comment

5813f23

Removed two box-drawing section banners in the indicator-list tests and a comment restating that the loop skips blank lines.

Fix prettier formatting in the indicator-list adapter tests

86cf8c1

stephmilovic force-pushed the threat-intel-7-adapters-core branch from 4447946 to c309e6c Compare September 4, 2026 20:17

stephmilovic force-pushed the threat-intel-8-adapters-rest branch from a981ff5 to 86cf8c1 Compare September 4, 2026 20:17

stephmilovic changed the base branch from threat-intel-7-adapters-core to main September 4, 2026 20:18

stephmilovic marked this pull request as ready for review September 4, 2026 20:57

stephmilovic requested review from a team as code owners September 4, 2026 20:57

github-actions Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread ...solutions/security/plugins/security_solution/server/threat_intel/adapters/kev/kev_adapter.ts Outdated

jonwalstedt reviewed Sep 8, 2026

View reviewed changes

elasticmachine and others added 4 commits September 8, 2026 15:25

Merge branch 'main' into threat-intel-8-adapters-rest

0b556ab

Changes from node scripts/check

5627e13

jonwalstedt approved these changes Sep 9, 2026

View reviewed changes

stephmilovic merged commit f499daf into elastic:main Sep 9, 2026
43 checks passed

Navigation Menu

Uh oh!

[Threat Intel 10/11] Remaining adapters, dispatcher, and fetch_source step - #287206

[Threat Intel 10/11] Remaining adapters, dispatcher, and fetch_source step#287206
stephmilovic merged 29 commits into
elastic:mainfrom
stephmilovic:threat-intel-8-adapters-rest

stephmilovic commented Aug 25, 2026 •

edited by jonwalstedt

Loading

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 4, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 4, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stephmilovic commented Sep 8, 2026

Uh oh!

kibanamachine commented Sep 8, 2026

Uh oh!

jonwalstedt left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

stephmilovic commented Aug 25, 2026 • edited by jonwalstedt Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Where this sits

Scope review follow-up

To test

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 4, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 4, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stephmilovic commented Sep 8, 2026

Uh oh!

kibanamachine commented Sep 8, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

History

Uh oh!

jonwalstedt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

stephmilovic commented Aug 25, 2026 •

edited by jonwalstedt

Loading