[Threat Intel 2/11] Contracts, constants, and shared libs by stephmilovic · Pull Request #287198 · elastic/kibana

stephmilovic · 2026-08-25T22:30:38Z

First slice of the threat intel supply pipeline. Adds the shared contracts and the small server-side libraries the rest of it builds on, plus the vendored reference data. Nothing imports any of this yet.

common/threat_intel holds the index names, route paths, IOC and severity vocabularies, and the fetch_source step contract. common/experimental_features adds the flag.

server/threat_intel/lib holds four things several later slices need, and that would otherwise force those slices to depend on each other:

ip_ranges classifies special-use IPv4 and IPv6 space. The SSRF guard rejects these and IOC extraction tiers them reference rather than treating them as candidate C2 anchors. Two implementations of this had already drifted apart, which is why it is shared. Covers the transition prefixes that can smuggle a restricted IPv4 address through an IPv6 literal: 6to4, NAT64, and the IPv4-mapped and IPv4-compatible forms.
space_filter is the per-space isolation helper. Every plugin-owned document carries space_id, reads accept the current space plus the global sentinel, writes tag with the current space.
cost_tracker records per-stage token usage and cost for the LLM stages. A model with no pricing row reports null rather than 0, and the trace carries an unpriced-stage count, so real spend is never indistinguishable from no spend.
es_options carries the hidden-index search options. It is here rather than in common because it is an Elasticsearch search option with no meaning in the browser bundle, and rather than in the setup layer because routes and tasks both need it and must not depend on setup.

data/iana_tlds.ts is the IANA TLD registry vendored verbatim and marked linguist-generated, so it collapses in diffs. That is 1,618 of the line count. data/ioc_noise_domains.ts is the benign-domain denylist. Both are consumed by IOC extraction later.

Reviewable logic here is about 550 lines; the rest is vendored data and tests.

Where this sits

#	PR	Depends on	Status
1	#287197 workflow gate correction + fixtures	nothing	✅ merged
2	#287198 contracts, constants, shared libs ← this PR	nothing	✅ merged
3	#287199 content parsing	#287198	closed
4	#287200 SSRF-guarded HTTP client	#287198	✅ merged
5	#287201 index templates, seeding, inference features	#287198, #287200	✅ merged
6	#287202 indicator alias	#287198, #287201	✅ merged
7	#287203 IOC extraction	#287198, #287200	✅ merged
8	#287204 LLM services, routes, source catalog API	#287198, #287200, #287202, #287203	✅ merged
9	#287205 RSS adapter	#287198, #287200, #287203, #287204	✅ merged
10	#287206 remaining adapters, dispatcher, fetch_source step	#287198, #287200, #287203, #287205	✅ merged
11	#287207 promote/scrub tasks and plugin wiring	#287198–#287206	✅ merged
12	#287479 generator fixture article URLs	nothing	✅ merged

#287479 came first in the merge train and is already on main. The numbered series (#287197–#287207) stacks on top in dependency order. #287199 (content parsing) was closed; IOC extraction ships its own section-header classifier.

infra-vault-gh-plugin-prod · 2026-08-25T22:33:29Z

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

infra-vault-gh-plugin-prod · 2026-08-25T22:33:30Z

Pinging @elastic/security-solution (Team: SecuritySolution)

stephmilovic · 2026-08-25T23:18:24Z

@elasticmachine merge upstream

…ply pipeline First slice of the threat intel supply pipeline. Adds the shared contracts and the small server-side libraries the rest of it builds on, plus the vendored reference data. Nothing imports any of this yet, and `threatIntelSupplyEnabled` defaults off, so it is inert. `common/threat_intel` holds the index names, route paths, IOC and severity vocabularies, and the `fetch_source` step contract. `common/experimental_features` gets the flag itself. `server/threat_intel/lib` holds three things that several later slices need and that would otherwise force those slices to depend on each other: - `ip_ranges` classifies special-use IPv4 and IPv6 space. The SSRF guard rejects these and IOC extraction tiers them `reference` rather than treating them as candidate C2 anchors. Two implementations of this had already drifted apart, so it is shared rather than duplicated. Covers the transition prefixes that can smuggle a restricted IPv4 address through an IPv6 literal: 6to4, NAT64, and the IPv4-mapped and IPv4-compatible forms. - `space_filter` is the per-space isolation helper. Every plugin-owned document carries `space_id`, reads accept the current space plus the global sentinel, and writes tag with the current space. - `cost_tracker` records per-stage token usage and cost for the LLM stages. A model with no pricing row reports `null` rather than `0`, and the trace carries an unpriced-stage count, so real spend is never indistinguishable from no spend. - `es_options` carries the hidden-index search options. It is here rather than in `common` because it is an Elasticsearch search option with no meaning in the browser bundle, and rather than in the setup layer because routes and tasks both need it and must not depend on setup. `data/iana_tlds.ts` is the IANA TLD registry vendored verbatim and marked `linguist-generated`, so it collapses in diffs. `data/ioc_noise_domains.ts` is the benign-domain denylist. Both are consumed by IOC extraction in a later slice. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

jonwalstedt

The ip_rages.ts file and its tests needs a small tweak otherwise it looks good!

jonwalstedt · 2026-08-26T13:15:14Z

+  // way to write an embedded 0.0.0.0), and there is no upside: 6to4 is
+  // deprecated by RFC 7526 and no real threat feed is served over either
+  // prefix, so failing closed on the whole range costs nothing.
+  if (/^2002:/i.test(lower)) return true;


The regex /^2001:2:/ matches the full 2001:2::/32 rather than the stated 2001:2::/48 benchmarking block (RFC 5180 §8). Addresses where the third group is non-zero, e.g. 2001:2:1::1 are incorrectly classified as non-routable.

Addresses in 2001:2::/48 appear in exactly two compressed forms: 2001:2::... (third group zeroed and swallowed by ::) and 2001:2:0:... (explicit zero). Replace with:

if (/^2001:2:(?:0:|:)/i.test(lower)) return true;

Also add a routable-boundary case to the test table:

['routable: 2001:2:1::1 is outside benchmarking /48', '2001:2:1::1'],

Good catch, you are right and the fix is in.

Applied your regex exactly. Confirmed the behaviour change rather than assuming it:

address before after

2001:2::1 blocked blocked

2001:2:0:0:0:0:0:1 blocked blocked

2001:2:1::1 blocked routable

2001:2000::1 routable routable

Added your boundary case to the table, plus the explicit-zero form (2001:2:0:0:0:0:0:1) so both in-range spellings are pinned, not just the compressed one.

Worth noting the consequence was not only cosmetic: this list is shared by the SSRF guard and the IOC extractor, so a wrongly-blocked address meant the guard would refuse a legitimate public feed host and the extractor would tier a real indicator reference, which keeps it out of the promoted set entirely.

I also went looking for the same mistake in the neighbouring prefix tests. 2002::/16 and 2001:db8::/32 are correct as written, since /16 and /32 land exactly on group boundaries. The NAT64 one was imprecise in the other direction: the regex matches 64:ff9b::/32 while the comment claimed /96 and /48. I left the wider match, because the rest of that /32 is unassigned so over-blocking fails closed, but I corrected the comment rather than leave it claiming a precision the code did not have.

@jonwalstedt

`/^2001:2:/` matched the whole `2001:2::/32`, not the `2001:2::/48` benchmarking block it was meant to (RFC 5180 §8), so an address with a non-zero third group like `2001:2:1::1` was wrongly classified non-routable. That would make the SSRF guard reject a legitimate public address and the IOC extractor tier it `reference`. The /48 only appears with the third group compressed away or written as an explicit zero, so both forms are matched and nothing wider. Also corrected the NAT64 comment, which named /96 and /48 while the regex covers 64:ff9b::/32. The over-blocking is deliberate since the rest of that /32 is unassigned, but the comment claimed a precision the code did not have. Thanks @jonwalstedt for catching both. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

stephmilovic · 2026-08-26T15:02:10Z

Thanks @jonwalstedt, that was a real bug and not just a tidy-up. Fix and your test case are in, details in the thread.

The 2001:2::/48 match was too wide by 16 bits, so 2001:2:1::1 was wrongly non-routable. Since this list backs both the SSRF guard and the IOC tiering, that meant the guard would refuse a legitimate public feed host and the extractor would tier a real indicator reference. I checked the neighbouring prefixes for the same mistake: 2002::/16 and 2001:db8::/32 are fine, and the NAT64 comment was overclaiming precision so I corrected it.

kibanamachine · 2026-08-26T16:12:17Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 18ac8b2
Build duration: 70 mins

Failed CI Steps

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`shared-plugins`	11.9MB	11.9MB	+28.0B

Unknown metric groups

total optimizer output size

id	before	after	diff
`all`	62.5MB	62.5MB	+28.0B

Test Failures

[job] [logs] FTR Configs #19 / Screenshotting Example should capture a screenshot

History

💛 Build #491551 was flaky fd6a122
💔 Build #491528 failed 0d9ecfb

…ply pipeline (elastic#287198) Second of 11 PRs replacing elastic#285795. Shared contracts plus the small server-side libraries the rest of the pipeline builds on, and the vendored reference data. Nothing imports any of it yet and `threatIntelSupplyEnabled` defaults off, so it is inert. `common/threat_intel` holds the index names, route paths, IOC and severity vocabularies, and the `fetch_source` step contract. `server/threat_intel/lib` holds four things later slices need and that would otherwise force those slices to depend on each other: - `ip_ranges` classifies special-use IPv4 and IPv6 space, shared by the SSRF guard and IOC tiering because two implementations had already drifted apart. Covers the transition prefixes that can smuggle a restricted IPv4 address through an IPv6 literal. - `space_filter` is the per-space isolation helper. - `cost_tracker` records per-stage token usage and cost. An unpriced model reports `null` rather than `0`, so real spend is never indistinguishable from no spend. - `es_options` carries the hidden-index search options. `data/iana_tlds.ts` is the IANA TLD registry vendored verbatim and marked `linguist-generated`; that is 1,618 of the line count. Reviewable logic is about 550 lines.

…og API (#287204) ## Summary Adds LLM enrichment services and Threat Intelligence HTTP routes, including the fixed-catalog source list and enable/disable API. Manual report creation and provenance URLs go through the shared HTTP/HTTPS normalizer. Also wires the routes, inference features, and one-time bootstrap into `plugin.ts`, gated by `threatIntelSupplyEnabled`. This wiring is interim: it exists so the enrichment eval suite (see Evaluation below) has routes to call ahead of the full pipeline wiring. `#287207` replaces it with a proper `wiring.ts` module; that PR's description has a rebase note covering the `plugin.ts` conflict resolution. ## Where this sits | # | PR | Depends on | Status | |---|---|---|---| | 1 | #287197 workflow gate correction + fixtures | nothing | ✅ merged | | 2 | #287198 contracts, constants, shared libs | nothing | ✅ merged | | 3 | #287199 content parsing | #287198 | closed | | 4 | #287200 SSRF-guarded HTTP client | #287198 | ✅ merged | | 5 | #287201 index templates, seeding, inference features | #287198, #287200 | ✅ merged | | 6 | #287202 indicator alias | #287198, #287201 | ✅ merged | | 7 | #287203 IOC extraction | #287198, #287200 | ✅ merged | | 8 | #287204 LLM services, routes, source catalog API **← this PR** | #287198, #287200, #287202, #287203 | 👀 ready | | 9 | #287205 RSS adapter | #287198, #287200, #287203, #287204 | draft | | 10 | #287206 remaining adapters, dispatcher, fetch_source step | #287198, #287200, #287203, #287205 | draft | | 11 | #287207 promote/scrub tasks and plugin wiring | #287198–#287206 | draft | | 12 | #287479 generator fixture article URLs | nothing | ✅ merged | | 13 | #289343 Scout API tests for the deterministic routes | #287204 | draft | | 14 | #289345 enrichment eval suite | #287204 | draft | #287479 came first in the merge train and is already on `main`. The numbered series (#287197–#287207) stacks on top in dependency order. #287199 (content parsing) was closed; IOC extraction ships its own section-header classifier. ## Scope review follow-up - Shared `normalizeProvenanceUrl` helper (HTTP/HTTPS only, strips credentials, bounded length). - Source create/delete routes removed. Updates may change only `enabled`. - Lists and mutations reject catalog IDs outside the approved set. - Ensures the per-space indicator alias on `list_sources` (idempotent). ## Evaluation A `@kbn/evals` suite exercises all four enrichment stages against the BlackHat demo pack article text, posting each input directly to the internal route and scoring the structured response. It runs as a tracked baseline (scores recorded, no build-failing thresholds yet). Latest full run, EIS Claude Sonnet 4.6 as model and judge, 1 repetition, 20 examples: | Stage | Evaluator | Score | |---|---|---| | assess_relevance | IsIntelligenceMatch | 1.00 | | assess_relevance | RelevanceShapeValid | 1.00 | | classify_severity | SeverityExactMatch | 0.83 | | classify_severity | SeverityWithinOneLevel | 1.00 | | enrich_taxonomy | CategoryRecall | 0.80 | | enrich_taxonomy | RegionRecall | 1.00 | | extract_diamond | DiamondNoIocLeak | 1.00 | | extract_diamond | DiamondSignalCount | 1.00 | | extract_diamond | criteria (LLM judge, majority vote) | 1.00 | The suite ships in #289345, a separate eval-only PR that stacks on this one (it needs the routes and the flag-gated wiring added here). The four deterministic routes are covered separately by Scout API tests in #289343. Any prompt or calibration fix the evals surface lands by amending this PR, not the eval PR. ## To test ``` node scripts/jest --config x-pack/solutions/security/plugins/security_solution/server/threat_intel/jest.config.js routes/list_sources.test.ts services/provenance_url.test.ts ``` _PR developed with Cursor + Auto_ --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Jon Wålstedt <jon.walstedt@elastic.co> Co-authored-by: Cursor <cursoragent@cursor.com>

…d the flag (#287207) ## Summary Wires Threat Intelligence behind `threatIntelSupplyEnabled`: bootstrap, managed workflow installation, promote/scrub tasks, and route registration. Promote mirrors extracted IOCs into `.threat-intel-indicators` for Indicator Match rules. ## Where this sits | # | PR | Depends on | Status | |---|---|---|---| | 1 | #287197 workflow gate correction + fixtures | nothing | ✅ merged | | 2 | #287198 contracts, constants, shared libs | nothing | ✅ merged | | 3 | #287199 content parsing | #287198 | closed | | 4 | #287200 SSRF-guarded HTTP client | #287198 | ✅ merged | | 5 | #287201 index templates, seeding, inference features | #287198, #287200 | ✅ merged | | 6 | #287202 indicator alias | #287198, #287201 | ✅ merged | | 7 | #287203 IOC extraction | #287198, #287200 | ✅ merged | | 8 | #287204 LLM services, routes, source catalog API | #287198, #287200, #287202, #287203 | ✅ merged | | 9 | #287205 RSS adapter | #287198, #287200, #287203, #287204 | ✅ merged | | 10 | #287206 remaining adapters, dispatcher, fetch_source step | #287198, #287200, #287203, #287205 | ✅ merged | | 11 | #287207 promote/scrub tasks and plugin wiring **← this PR** | #287198–#287206 | ready for review | | 12 | #287479 generator fixture article URLs | nothing | ✅ merged | | 13 | #289343 Scout API tests for the deterministic routes | #287204 | ready for review | | 14 | #289345 enrichment eval suite | #287204 | ready for review | | 15 | #290144 read APIs, readiness, space-keyed attribution | #287207 | draft | #287479 came first in the merge train and is already on `main`. The numbered series (#287197–#287205) is merged; #287206 and #287207 now sit directly on `main`. #287199 (content parsing) was closed; IOC extraction ships its own section-header classifier. ## Scope review follow-up - Installs the managed workflows with the right space topology: `attribute_alerts_to_reports` installs per space (it queries `.alerts-security.alerts-*` and writes per-space hit totals, so a global install would clobber every space's totals onto one shared doc), while `ingest_threat_feeds` and `enrich_threat_report` install once globally. Alert analysis and threat intel share a single `ready()` call so neither install set is dropped, and spaces created after boot are reconciled on the promote task. All three ship `enabled: false`, and enrich routes its HTTP calls through a fixed space (`default`) instead of the install-time `workflow.spaceId` (which is `'*'` globally). - Ensures the default-space indicator alias on start. - Chunks promotion bulk writes and treats HTTP 408/500 as retryable bulk failures. - Sanitizes provenance and extracted IOC reference URLs during promotion. - Documents direct-index cross-space isolation as the blocker to enabling the feature. ## Bootstrap fix folded in from #289343 `seed_default_sources.ts` sorted the legacy-source disable scan on `_id`, which Elasticsearch rejects with `illegal_argument_exception: Fielddata access on the _id field is disallowed` unless `indices.id_field_data.enabled` is set, so it failed bootstrap on a stock cluster. Found this while getting #289343's Scout suite green and moved the fix here since this PR is the one wiring up bootstrap and already needs `security-threat-hunting` review, so it isn't adding a reviewer #289343 wouldn't otherwise need. ## Product boundary `threatIntelSupplyEnabled` defaults to false. Do not enable until direct-index cross-space isolation is hardened or the administrator trust model is explicitly accepted. ## To test ``` node scripts/jest --config x-pack/solutions/security/plugins/security_solution/server/threat_intel/jest.config.js wiring.test.ts tasks/promote_threat_indicators.test.ts node scripts/jest x-pack/solutions/security/plugins/security_solution/server/workflows/ node scripts/jest src/platform/packages/shared/kbn-workflows/managed/definitions/threat_intel/ ``` _PR developed with Cursor + Auto + Sonnet 5 + Opus 4.8_ --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Jon Wålstedt <jon.walstedt@elastic.co> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

stephmilovic marked this pull request as ready for review August 25, 2026 22:33

stephmilovic requested review from a team as code owners August 25, 2026 22:33

stephmilovic force-pushed the threat-intel-2-foundation branch from e611e78 to fd6a122 Compare August 25, 2026 23:44

jonwalstedt approved these changes Aug 26, 2026

View reviewed changes

stephmilovic merged commit a3fcdec into elastic:main Aug 26, 2026
31 of 32 checks passed

This was referenced Aug 27, 2026

chore: tidy up CODEOWNERS #287096

Merged

Restore Threat Intel CODEOWNERS entries #287711

Merged

This was referenced Sep 1, 2026

[Threat Intel] Move the indicator index out of the .kibana namespace #287479

Merged

[Threat Intel] Scout API tests for the deterministic enrichment routes #289343

Merged

[Threat Intel] Enrichment eval suite for the LLM-backed routes #289345

Merged

Navigation Menu

Uh oh!

[Threat Intel 2/11] Contracts, constants, and shared libs - #287198

[Threat Intel 2/11] Contracts, constants, and shared libs#287198
stephmilovic merged 2 commits into
elastic:mainfrom
stephmilovic:threat-intel-2-foundation

stephmilovic commented Aug 25, 2026 •

edited by jonwalstedt

Loading

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 25, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 25, 2026

Uh oh!

stephmilovic commented Aug 25, 2026

Uh oh!

jonwalstedt left a comment

Uh oh!

jonwalstedt Aug 26, 2026

Uh oh!

stephmilovic Aug 26, 2026

Uh oh!

stephmilovic commented Aug 26, 2026

Uh oh!

kibanamachine commented Aug 26, 2026

total optimizer output size

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

address	before	after
`2001:2::1`	blocked	blocked
`2001:2:0:0:0:0:0:1`	blocked	blocked
`2001:2:1::1`	blocked	routable
`2001:2000::1`	routable	routable

Uh oh!

Conversation

stephmilovic commented Aug 25, 2026 • edited by jonwalstedt Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Where this sits

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 25, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 25, 2026

Uh oh!

stephmilovic commented Aug 25, 2026

Uh oh!

jonwalstedt left a comment

Choose a reason for hiding this comment

Uh oh!

jonwalstedt Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

stephmilovic Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

stephmilovic commented Aug 26, 2026

Uh oh!

kibanamachine commented Aug 26, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Page load bundle

total optimizer output size

Test Failures

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stephmilovic commented Aug 25, 2026 •

edited by jonwalstedt

Loading