[Threat Intel 7/11] IOC extraction with a precision pass and confidence tiering by stephmilovic · Pull Request #287203 · elastic/kibana

stephmilovic · 2026-08-25T22:31:42Z

Summary

Adds structured IOC extraction with section tiering, defang recovery, and adversarial performance bounds. Extraction uses explicit Markdown ## headings via the shared classifyHeader entry point.

Where this sits

#	PR	Depends on	Status
1	#287197 workflow gate correction + fixtures	nothing	✅ merged
2	#287198 contracts, constants, shared libs	nothing	✅ merged
3	#287199 content parsing	#287198	closed
4	#287200 SSRF-guarded HTTP client	#287198	✅ merged
5	#287201 index templates, seeding, inference features	#287198, #287200	✅ merged
6	#287202 indicator alias	#287198, #287201	✅ merged
7	#287203 IOC extraction ← this PR	#287198, #287200	✅ merged
8	#287204 LLM services, routes, source catalog API	#287198, #287200, #287202, #287203	✅ merged
9	#287205 RSS adapter	#287198, #287200, #287203, #287204	✅ merged
10	#287206 remaining adapters, dispatcher, fetch_source step	#287198, #287200, #287203, #287205	✅ merged
11	#287207 promote/scrub tasks and plugin wiring	#287198–#287206	✅ merged
12	#287479 generator fixture article URLs	nothing	✅ merged

#287479 came first in the merge train and is already on main. The numbered series (#287197–#287207) stacks on top in dependency order. #287199 (content parsing) was closed; IOC extraction ships its own section-header classifier.

Scope review follow-up

Linear URL trimming, consumed-span checks, domain suffix dedup, and section tiering.
Rejects domains longer than 253 characters before suffix dedup.
Uses a single lowercased copy of the source text for email defang detection.
Adversarial regressions for 100,000 trailing closers and 10,000 structured domains.

To test

node scripts/jest --config x-pack/solutions/security/plugins/security_solution/server/threat_intel/jest.config.js services/extract_iocs.test.ts

PR developed with Cursor + Auto

infra-vault-gh-plugin-prod · 2026-09-02T00:15:39Z

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

infra-vault-gh-plugin-prod · 2026-09-02T00:15:40Z

Pinging @elastic/security-solution (Team: SecuritySolution)

jonwalstedt

Looks good, two comments on improving the test coverage otherwise good to go

…ce tiering

`extracted.iocs` is a `nested` field, so every entry is its own Lucene document, and the reports index leaves `index.mapping.nested_objects.limit` at the Elasticsearch default of 10,000. Crossing it does not drop the extra entries, it rejects the entire report document. The report then stays `pending` and every enrichment run re-tries it and fails the same way. Extraction bounded each IOC's value length but never how many it returned, and the only path that chunks across documents is `text_indicator_list`. Everything else writes one document, so a single 5,000,000-character analyst paste or a dense feed item could cross the limit on its own. This is the third instance of the same shape in this pipeline, after the over-long document id and the unbounded `sources[]` accumulator: an unbounded producer meeting a hard Elasticsearch limit, where the failure lands on the whole document rather than the offending field. Capped at 5,000, matching the per-document budget `text_indicator_list` already uses and leaving room for the `external_references` entries that share the limit. When it truncates it keeps the most promotable tiers first, so a truncated report loses `reference` and `denied` noise before it loses a `discriminating` indicator, and `count` continues to report what was found rather than what was kept so the truncation is visible to callers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ction_headers helper The content-parsing refactor (elastic#287199) internalized IOC_HEADER_TERMS, TERMINATOR_HEADER_TERMS, TERMINATOR_PREFIXES, and normalizeHeader behind a single classifyHeader() entry point that also added plural-aware and prefix-boundary-aware matching this file's own isIocHeader/isTerminatorHeader lacked. Those raw exports no longer exist, so this file's import breaks once elastic#287199 merges. Delegate to classifyHeader directly instead of re-implementing the same classification locally.

…y, drop HTML path Transplant the heading/section classifier (`classifyHeader`) into the IOC extraction module as `services/section_headers.ts` so extraction no longer depends on the removed `content/` module. `extractIocs` stays string-in and deterministic. Rewrite the tests to feed the bounded structured text the extractor actually consumes (`## heading`, `| cell |`, `- item` lines) instead of routing HTML through `htmlToStructured`, and drop the cases whose only purpose was markup conversion. The markdown-artifact leak gate, section-aware tiering, defanging, and precision-tier coverage are preserved. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

Locks the pushIoc over-long drop to MAX_URL_LENGTH and asserts the defanged field consumers read, including the bracketed URL scheme separator. Co-authored-by: Cursor <cursoragent@cursor.com>

stephmilovic · 2026-09-02T21:21:40Z

@elasticmachine merge upstream

kibanamachine · 2026-09-02T22:26:33Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 4a2ae30
Build duration: 64 mins
Storybooks Preview

Failed CI Steps

Metrics [docs]

✅ unchanged

Test Failures

[job] [logs] FTR Configs #43 / lens serverless - group 1 - subgroup 1 lens smokescreen tests should allow filtering by legend on an xy chart

History

💛 Build #495148 was flaky 7c5b6b3

…ce tiering (elastic#287203)

…og API (#287204) ## Summary Adds LLM enrichment services and Threat Intelligence HTTP routes, including the fixed-catalog source list and enable/disable API. Manual report creation and provenance URLs go through the shared HTTP/HTTPS normalizer. Also wires the routes, inference features, and one-time bootstrap into `plugin.ts`, gated by `threatIntelSupplyEnabled`. This wiring is interim: it exists so the enrichment eval suite (see Evaluation below) has routes to call ahead of the full pipeline wiring. `#287207` replaces it with a proper `wiring.ts` module; that PR's description has a rebase note covering the `plugin.ts` conflict resolution. ## Where this sits | # | PR | Depends on | Status | |---|---|---|---| | 1 | #287197 workflow gate correction + fixtures | nothing | ✅ merged | | 2 | #287198 contracts, constants, shared libs | nothing | ✅ merged | | 3 | #287199 content parsing | #287198 | closed | | 4 | #287200 SSRF-guarded HTTP client | #287198 | ✅ merged | | 5 | #287201 index templates, seeding, inference features | #287198, #287200 | ✅ merged | | 6 | #287202 indicator alias | #287198, #287201 | ✅ merged | | 7 | #287203 IOC extraction | #287198, #287200 | ✅ merged | | 8 | #287204 LLM services, routes, source catalog API **← this PR** | #287198, #287200, #287202, #287203 | 👀 ready | | 9 | #287205 RSS adapter | #287198, #287200, #287203, #287204 | draft | | 10 | #287206 remaining adapters, dispatcher, fetch_source step | #287198, #287200, #287203, #287205 | draft | | 11 | #287207 promote/scrub tasks and plugin wiring | #287198–#287206 | draft | | 12 | #287479 generator fixture article URLs | nothing | ✅ merged | | 13 | #289343 Scout API tests for the deterministic routes | #287204 | draft | | 14 | #289345 enrichment eval suite | #287204 | draft | #287479 came first in the merge train and is already on `main`. The numbered series (#287197–#287207) stacks on top in dependency order. #287199 (content parsing) was closed; IOC extraction ships its own section-header classifier. ## Scope review follow-up - Shared `normalizeProvenanceUrl` helper (HTTP/HTTPS only, strips credentials, bounded length). - Source create/delete routes removed. Updates may change only `enabled`. - Lists and mutations reject catalog IDs outside the approved set. - Ensures the per-space indicator alias on `list_sources` (idempotent). ## Evaluation A `@kbn/evals` suite exercises all four enrichment stages against the BlackHat demo pack article text, posting each input directly to the internal route and scoring the structured response. It runs as a tracked baseline (scores recorded, no build-failing thresholds yet). Latest full run, EIS Claude Sonnet 4.6 as model and judge, 1 repetition, 20 examples: | Stage | Evaluator | Score | |---|---|---| | assess_relevance | IsIntelligenceMatch | 1.00 | | assess_relevance | RelevanceShapeValid | 1.00 | | classify_severity | SeverityExactMatch | 0.83 | | classify_severity | SeverityWithinOneLevel | 1.00 | | enrich_taxonomy | CategoryRecall | 0.80 | | enrich_taxonomy | RegionRecall | 1.00 | | extract_diamond | DiamondNoIocLeak | 1.00 | | extract_diamond | DiamondSignalCount | 1.00 | | extract_diamond | criteria (LLM judge, majority vote) | 1.00 | The suite ships in #289345, a separate eval-only PR that stacks on this one (it needs the routes and the flag-gated wiring added here). The four deterministic routes are covered separately by Scout API tests in #289343. Any prompt or calibration fix the evals surface lands by amending this PR, not the eval PR. ## To test ``` node scripts/jest --config x-pack/solutions/security/plugins/security_solution/server/threat_intel/jest.config.js routes/list_sources.test.ts services/provenance_url.test.ts ``` _PR developed with Cursor + Auto_ --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Jon Wålstedt <jon.walstedt@elastic.co> Co-authored-by: Cursor <cursoragent@cursor.com>

…ce tiering (elastic#287203)

…d the flag (#287207) ## Summary Wires Threat Intelligence behind `threatIntelSupplyEnabled`: bootstrap, managed workflow installation, promote/scrub tasks, and route registration. Promote mirrors extracted IOCs into `.threat-intel-indicators` for Indicator Match rules. ## Where this sits | # | PR | Depends on | Status | |---|---|---|---| | 1 | #287197 workflow gate correction + fixtures | nothing | ✅ merged | | 2 | #287198 contracts, constants, shared libs | nothing | ✅ merged | | 3 | #287199 content parsing | #287198 | closed | | 4 | #287200 SSRF-guarded HTTP client | #287198 | ✅ merged | | 5 | #287201 index templates, seeding, inference features | #287198, #287200 | ✅ merged | | 6 | #287202 indicator alias | #287198, #287201 | ✅ merged | | 7 | #287203 IOC extraction | #287198, #287200 | ✅ merged | | 8 | #287204 LLM services, routes, source catalog API | #287198, #287200, #287202, #287203 | ✅ merged | | 9 | #287205 RSS adapter | #287198, #287200, #287203, #287204 | ✅ merged | | 10 | #287206 remaining adapters, dispatcher, fetch_source step | #287198, #287200, #287203, #287205 | ✅ merged | | 11 | #287207 promote/scrub tasks and plugin wiring **← this PR** | #287198–#287206 | ready for review | | 12 | #287479 generator fixture article URLs | nothing | ✅ merged | | 13 | #289343 Scout API tests for the deterministic routes | #287204 | ready for review | | 14 | #289345 enrichment eval suite | #287204 | ready for review | | 15 | #290144 read APIs, readiness, space-keyed attribution | #287207 | draft | #287479 came first in the merge train and is already on `main`. The numbered series (#287197–#287205) is merged; #287206 and #287207 now sit directly on `main`. #287199 (content parsing) was closed; IOC extraction ships its own section-header classifier. ## Scope review follow-up - Installs the managed workflows with the right space topology: `attribute_alerts_to_reports` installs per space (it queries `.alerts-security.alerts-*` and writes per-space hit totals, so a global install would clobber every space's totals onto one shared doc), while `ingest_threat_feeds` and `enrich_threat_report` install once globally. Alert analysis and threat intel share a single `ready()` call so neither install set is dropped, and spaces created after boot are reconciled on the promote task. All three ship `enabled: false`, and enrich routes its HTTP calls through a fixed space (`default`) instead of the install-time `workflow.spaceId` (which is `'*'` globally). - Ensures the default-space indicator alias on start. - Chunks promotion bulk writes and treats HTTP 408/500 as retryable bulk failures. - Sanitizes provenance and extracted IOC reference URLs during promotion. - Documents direct-index cross-space isolation as the blocker to enabling the feature. ## Bootstrap fix folded in from #289343 `seed_default_sources.ts` sorted the legacy-source disable scan on `_id`, which Elasticsearch rejects with `illegal_argument_exception: Fielddata access on the _id field is disallowed` unless `indices.id_field_data.enabled` is set, so it failed bootstrap on a stock cluster. Found this while getting #289343's Scout suite green and moved the fix here since this PR is the one wiring up bootstrap and already needs `security-threat-hunting` review, so it isn't adding a reviewer #289343 wouldn't otherwise need. ## Product boundary `threatIntelSupplyEnabled` defaults to false. Do not enable until direct-index cross-space isolation is hardened or the administrator trust model is explicitly accepted. ## To test ``` node scripts/jest --config x-pack/solutions/security/plugins/security_solution/server/threat_intel/jest.config.js wiring.test.ts tasks/promote_threat_indicators.test.ts node scripts/jest x-pack/solutions/security/plugins/security_solution/server/workflows/ node scripts/jest src/platform/packages/shared/kbn-workflows/managed/definitions/threat_intel/ ``` _PR developed with Cursor + Auto + Sonnet 5 + Opus 4.8_ --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Jon Wålstedt <jon.walstedt@elastic.co> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

stephmilovic force-pushed the threat-intel-5-ioc-extraction branch 2 times, most recently from 7fad58f to a059e5a Compare August 26, 2026 20:45

jonwalstedt force-pushed the threat-intel-5-ioc-extraction branch from 0ebd039 to fd9196e Compare August 28, 2026 19:59

stephmilovic force-pushed the threat-intel-5-ioc-extraction branch from fd9196e to a7222bf Compare September 1, 2026 20:06

stephmilovic mentioned this pull request Sep 1, 2026

[Threat Intel] Move the indicator index out of the .kibana namespace #287479

Merged

stephmilovic force-pushed the threat-intel-5-ioc-extraction branch from a7222bf to 7c5b6b3 Compare September 2, 2026 00:13

stephmilovic marked this pull request as ready for review September 2, 2026 00:15

stephmilovic requested a review from a team as a code owner September 2, 2026 00:15

jonwalstedt approved these changes Sep 2, 2026

View reviewed changes

Comment thread ...ck/solutions/security/plugins/security_solution/server/threat_intel/services/extract_iocs.ts

Comment thread ...lutions/security/plugins/security_solution/server/threat_intel/services/extract_iocs.test.ts

stephmilovic and others added 4 commits September 2, 2026 14:34

feat(threat-intel): IOC extraction with a precision pass and confiden…

5febe4a

…ce tiering

stephmilovic and others added 7 commits September 2, 2026 14:34

Define the structured heading contract

ee374e3

Remove stale threat intel reference comment

c03b5e7

Remove stale IOC parser references

1b0907f

Bound IOC extraction on adversarial input

b45d6a3

Linearize structured IOC section tiering

327ebde

Satisfy IOC extraction lint

4539c6e

Address scope-review findings from deep PR review

03f4f34

Co-authored-by: Cursor <cursoragent@cursor.com>

stephmilovic force-pushed the threat-intel-5-ioc-extraction branch from 7c5b6b3 to 03f4f34 Compare September 2, 2026 20:38

test(threat-intel): cover defang output and MAX_IOC_VALUE_LENGTH drops

d977327

Locks the pushIoc over-long drop to MAX_URL_LENGTH and asserts the defanged field consumers read, including the bracketed URL scheme separator. Co-authored-by: Cursor <cursoragent@cursor.com>

Merge branch 'main' into threat-intel-5-ioc-extraction

4a2ae30

stephmilovic merged commit 4634848 into elastic:main Sep 3, 2026
41 of 42 checks passed

juliaElastic pushed a commit to juliaElastic/kibana that referenced this pull request Sep 4, 2026

[Threat Intel 7/11] IOC extraction with a precision pass and confiden…

80c3512

…ce tiering (elastic#287203)

This was referenced Sep 4, 2026

[Threat Intel] Scout API tests for the deterministic enrichment routes #289343

Merged

[Threat Intel] Enrichment eval suite for the LLM-backed routes #289345

Merged

Supplementing pushed a commit to Supplementing/kibana that referenced this pull request Sep 8, 2026

[Threat Intel 7/11] IOC extraction with a precision pass and confiden…

62adf5a

…ce tiering (elastic#287203)

Navigation Menu

Uh oh!

[Threat Intel 7/11] IOC extraction with a precision pass and confidence tiering - #287203

[Threat Intel 7/11] IOC extraction with a precision pass and confidence tiering#287203
stephmilovic merged 13 commits into
elastic:mainfrom
stephmilovic:threat-intel-5-ioc-extraction

stephmilovic commented Aug 25, 2026 •

edited by jonwalstedt

Loading

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 2, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 2, 2026

Uh oh!

jonwalstedt left a comment

Uh oh!

Uh oh!

Uh oh!

stephmilovic commented Sep 2, 2026

Uh oh!

kibanamachine commented Sep 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

stephmilovic commented Aug 25, 2026 • edited by jonwalstedt Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Where this sits

Scope review follow-up

To test

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 2, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 2, 2026

Uh oh!

jonwalstedt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

stephmilovic commented Sep 2, 2026

Uh oh!

kibanamachine commented Sep 2, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Test Failures

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

stephmilovic commented Aug 25, 2026 •

edited by jonwalstedt

Loading