[Threat Intel 9/11] RSS adapter with bounded structured-text serialization by stephmilovic · Pull Request #287205 · elastic/kibana

stephmilovic · 2026-08-25T22:32:12Z

Summary

Adds the RSS adapter with bounded structured-text serialization (headings, paragraphs, lists, tables). Entry links are provenance only and are never fetched. STIX and TAXII adapters were removed from the MVP scope.

Where this sits

#	PR	Depends on	Status
1	#287197 workflow gate correction + fixtures	nothing	✅ merged
2	#287198 contracts, constants, shared libs	nothing	✅ merged
3	#287199 content parsing	#287198	closed
4	#287200 SSRF-guarded HTTP client	#287198	✅ merged
5	#287201 index templates, seeding, inference features	#287198, #287200	✅ merged
6	#287202 indicator alias	#287198, #287201	✅ merged
7	#287203 IOC extraction	#287198, #287200	✅ merged
8	#287204 LLM services, routes, source catalog API	#287198, #287200, #287202, #287203	✅ merged
9	#287205 RSS adapter ← this PR	#287198, #287200, #287203, #287204	✅ merged
10	#287206 remaining adapters, dispatcher, fetch_source step	#287198, #287200, #287203, #287205	✅ merged
11	#287207 promote/scrub tasks and plugin wiring	#287198–#287206	✅ merged
12	#287479 generator fixture article URLs	nothing	✅ merged
13	#289343 Scout API tests for the deterministic routes	#287204	ready
14	#289345 enrichment eval suite	#287204	ready

#287479 came first in the merge train and is already on main. The numbered series (#287197–#287204) is merged; #287205 onward now sit directly on main. #287199 (content parsing) was closed; IOC extraction ships its own section-header classifier.

Scope review follow-up

Replaced the RSS HTML helper with a local Cheerio structured serializer.
Skips subtrees with the native hidden attribute.
Sanitizes RSS entry provenance and falls back to the normalized feed URL.
Added a regression proving entry links are never fetched.

Product boundary

RSS fragment handling stays local and bounded. Inline CSS visibility is not parsed.

To test

node scripts/jest --config x-pack/solutions/security/plugins/security_solution/server/threat_intel/jest.config.js adapters/rss

PR developed with Cursor + Auto

Adapters in the RSS/STIX/TAXII (elastic#287205) and vendor_api (elastic#287206) slices import collapseWhitespace from content/text.ts. The content-parsing rewrite made it a private const, which breaks those imports once this PR merges. It's a plain string utility with real external callers, not part of the parsing-architecture change, so restore the export.

…og API (#287204) ## Summary Adds LLM enrichment services and Threat Intelligence HTTP routes, including the fixed-catalog source list and enable/disable API. Manual report creation and provenance URLs go through the shared HTTP/HTTPS normalizer. Also wires the routes, inference features, and one-time bootstrap into `plugin.ts`, gated by `threatIntelSupplyEnabled`. This wiring is interim: it exists so the enrichment eval suite (see Evaluation below) has routes to call ahead of the full pipeline wiring. `#287207` replaces it with a proper `wiring.ts` module; that PR's description has a rebase note covering the `plugin.ts` conflict resolution. ## Where this sits | # | PR | Depends on | Status | |---|---|---|---| | 1 | #287197 workflow gate correction + fixtures | nothing | ✅ merged | | 2 | #287198 contracts, constants, shared libs | nothing | ✅ merged | | 3 | #287199 content parsing | #287198 | closed | | 4 | #287200 SSRF-guarded HTTP client | #287198 | ✅ merged | | 5 | #287201 index templates, seeding, inference features | #287198, #287200 | ✅ merged | | 6 | #287202 indicator alias | #287198, #287201 | ✅ merged | | 7 | #287203 IOC extraction | #287198, #287200 | ✅ merged | | 8 | #287204 LLM services, routes, source catalog API **← this PR** | #287198, #287200, #287202, #287203 | 👀 ready | | 9 | #287205 RSS adapter | #287198, #287200, #287203, #287204 | draft | | 10 | #287206 remaining adapters, dispatcher, fetch_source step | #287198, #287200, #287203, #287205 | draft | | 11 | #287207 promote/scrub tasks and plugin wiring | #287198–#287206 | draft | | 12 | #287479 generator fixture article URLs | nothing | ✅ merged | | 13 | #289343 Scout API tests for the deterministic routes | #287204 | draft | | 14 | #289345 enrichment eval suite | #287204 | draft | #287479 came first in the merge train and is already on `main`. The numbered series (#287197–#287207) stacks on top in dependency order. #287199 (content parsing) was closed; IOC extraction ships its own section-header classifier. ## Scope review follow-up - Shared `normalizeProvenanceUrl` helper (HTTP/HTTPS only, strips credentials, bounded length). - Source create/delete routes removed. Updates may change only `enabled`. - Lists and mutations reject catalog IDs outside the approved set. - Ensures the per-space indicator alias on `list_sources` (idempotent). ## Evaluation A `@kbn/evals` suite exercises all four enrichment stages against the BlackHat demo pack article text, posting each input directly to the internal route and scoring the structured response. It runs as a tracked baseline (scores recorded, no build-failing thresholds yet). Latest full run, EIS Claude Sonnet 4.6 as model and judge, 1 repetition, 20 examples: | Stage | Evaluator | Score | |---|---|---| | assess_relevance | IsIntelligenceMatch | 1.00 | | assess_relevance | RelevanceShapeValid | 1.00 | | classify_severity | SeverityExactMatch | 0.83 | | classify_severity | SeverityWithinOneLevel | 1.00 | | enrich_taxonomy | CategoryRecall | 0.80 | | enrich_taxonomy | RegionRecall | 1.00 | | extract_diamond | DiamondNoIocLeak | 1.00 | | extract_diamond | DiamondSignalCount | 1.00 | | extract_diamond | criteria (LLM judge, majority vote) | 1.00 | The suite ships in #289345, a separate eval-only PR that stacks on this one (it needs the routes and the flag-gated wiring added here). The four deterministic routes are covered separately by Scout API tests in #289343. Any prompt or calibration fix the evals surface lands by amending this PR, not the eval PR. ## To test ``` node scripts/jest --config x-pack/solutions/security/plugins/security_solution/server/threat_intel/jest.config.js routes/list_sources.test.ts services/provenance_url.test.ts ``` _PR developed with Cursor + Auto_ --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Jon Wålstedt <jon.walstedt@elastic.co> Co-authored-by: Cursor <cursoragent@cursor.com>

`rss`, `stix`, and `taxii` all import `redactUrl` and use it in their thrown errors, but each still interpolated the raw URL into at least one log line. Source URLs may embed `user:password@`, so those lines put feed credentials into Kibana's logs. Six sites across the three adapters. The raw URL is still what gets fetched; only the log text changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Missed in the previous commit: the zero-reportable-objects log line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ugin indices The conclusion was right and the mechanism was wrong. These are not system or restricted indices: `.kibana-*` with a hyphen is not in Elasticsearch's restricted set, which covers `.kibana` and `.kibana_*` with an underscore, and an ordinary role granted `read` on one of them reads it fine without `allow_restricted_indices`. Verified against a live cluster rather than reasoned about. The real reason a non-superuser gets a security_exception is simpler: a Kibana feature privilege is not an Elasticsearch privilege, and nothing in this plugin grants an index privilege on these indices. Worth correcting rather than leaving as a harmless inaccuracy, because the two explanations imply opposite fixes. "Restricted index" reads as "no role can be given access, stop trying"; the truth is "no role has been given access yet", which is something an operator can act on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…parse_rss Turns on xml2js's namespace and child-order tracking so this file decides, once, what a feed entry's body actually is, instead of leaving that to be re-derived by hand downstream in content/text.ts (companion change on threat-intel-3-content). parseAtom/parseRss2/parseRdf now return a tagged EntryBody (`{ kind: 'markup', html }` or `{ kind: 'text', text }`) instead of an ambiguous bodyHtml string. rss_adapter.ts branches on body.kind instead of always calling stripHtml and always writing body_html. Fixes two real gaps along the way: RDF feeds had no content:encoded support at all, and Atom type="xhtml" content was silently dropped instead of walked as markup. A type="text" (or untyped, which RFC 4287 defaults to text) construct is also no longer mislabeled into body_html. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…dapter Condensed multi-paragraph docblocks to the one non-obvious fact each, including the xml2js option choices and the xmlns:true rejection. No behavior change; parse_rss.test.ts still green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The surrounding comment explains that the scoped client cannot reach the plugin indices, which is still true. `.threat-intel-indicators` is the one deliberately placed where Elasticsearch's reserved roles can grant it, so it is worth saying that grantable is not granted: this client still has no privilege on it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… storage Add a small RSS-local helper, adapters/rss/html_fragment_to_text.ts, that turns an embedded RSS/Atom HTML fragment into bounded plain text with Cheerio: it strips script/style/template/iframe/noembed/noframes/title/ textarea subtrees (leaving a space so tokens can't reassemble), decodes entities, keeps block/break/cell/list boundaries, bounds input and output, and fails closed to empty text. It is deliberately not a general HTML API. Update rss_adapter to convert markup entries through that helper and store only body_text (no body_html); the configured feed URL is the only thing fetched, and an entry link is kept as source.url provenance only. Repoint the RSS, STIX, and TAXII adapters at the transplanted services/ severity and report-content helpers now that the content/ module is gone. Co-authored-by: Cursor <cursoragent@cursor.com>

After elastic#287201, feed URLs live in catalog_source_urls rather than source config. Co-authored-by: Cursor <cursoragent@cursor.com>

infra-vault-gh-plugin-prod · 2026-09-04T20:57:58Z

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

infra-vault-gh-plugin-prod · 2026-09-04T20:57:59Z

Pinging @elastic/security-solution (Team: SecuritySolution)

…ents

jonwalstedt

Looks good! not sure if a custom rss parser is to prefer rather than using rss-parser but I'll leave that up to you to decide :)

stephmilovic · 2026-09-08T13:25:28Z

@elasticmachine merge upstream

kibanamachine · 2026-09-08T14:29:25Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: d3af712
Build duration: 63 mins
Storybooks Preview

Failed CI Steps

FTR Configs #4

Metrics [docs]

✅ unchanged

Test Failures

[job] [logs] FTR Configs #4 / lens app - group 1 lens dimension editor should correctly optimize multiple percentile metrics

History

💛 Build #497262 was flaky ad575c5
💔 Build #497252 failed c309e6c
💔 Build #497168 failed 4447946

…ation (elastic#287205)

…d the flag (#287207) ## Summary Wires Threat Intelligence behind `threatIntelSupplyEnabled`: bootstrap, managed workflow installation, promote/scrub tasks, and route registration. Promote mirrors extracted IOCs into `.threat-intel-indicators` for Indicator Match rules. ## Where this sits | # | PR | Depends on | Status | |---|---|---|---| | 1 | #287197 workflow gate correction + fixtures | nothing | ✅ merged | | 2 | #287198 contracts, constants, shared libs | nothing | ✅ merged | | 3 | #287199 content parsing | #287198 | closed | | 4 | #287200 SSRF-guarded HTTP client | #287198 | ✅ merged | | 5 | #287201 index templates, seeding, inference features | #287198, #287200 | ✅ merged | | 6 | #287202 indicator alias | #287198, #287201 | ✅ merged | | 7 | #287203 IOC extraction | #287198, #287200 | ✅ merged | | 8 | #287204 LLM services, routes, source catalog API | #287198, #287200, #287202, #287203 | ✅ merged | | 9 | #287205 RSS adapter | #287198, #287200, #287203, #287204 | ✅ merged | | 10 | #287206 remaining adapters, dispatcher, fetch_source step | #287198, #287200, #287203, #287205 | ✅ merged | | 11 | #287207 promote/scrub tasks and plugin wiring **← this PR** | #287198–#287206 | ready for review | | 12 | #287479 generator fixture article URLs | nothing | ✅ merged | | 13 | #289343 Scout API tests for the deterministic routes | #287204 | ready for review | | 14 | #289345 enrichment eval suite | #287204 | ready for review | | 15 | #290144 read APIs, readiness, space-keyed attribution | #287207 | draft | #287479 came first in the merge train and is already on `main`. The numbered series (#287197–#287205) is merged; #287206 and #287207 now sit directly on `main`. #287199 (content parsing) was closed; IOC extraction ships its own section-header classifier. ## Scope review follow-up - Installs the managed workflows with the right space topology: `attribute_alerts_to_reports` installs per space (it queries `.alerts-security.alerts-*` and writes per-space hit totals, so a global install would clobber every space's totals onto one shared doc), while `ingest_threat_feeds` and `enrich_threat_report` install once globally. Alert analysis and threat intel share a single `ready()` call so neither install set is dropped, and spaces created after boot are reconciled on the promote task. All three ship `enabled: false`, and enrich routes its HTTP calls through a fixed space (`default`) instead of the install-time `workflow.spaceId` (which is `'*'` globally). - Ensures the default-space indicator alias on start. - Chunks promotion bulk writes and treats HTTP 408/500 as retryable bulk failures. - Sanitizes provenance and extracted IOC reference URLs during promotion. - Documents direct-index cross-space isolation as the blocker to enabling the feature. ## Bootstrap fix folded in from #289343 `seed_default_sources.ts` sorted the legacy-source disable scan on `_id`, which Elasticsearch rejects with `illegal_argument_exception: Fielddata access on the _id field is disallowed` unless `indices.id_field_data.enabled` is set, so it failed bootstrap on a stock cluster. Found this while getting #289343's Scout suite green and moved the fix here since this PR is the one wiring up bootstrap and already needs `security-threat-hunting` review, so it isn't adding a reviewer #289343 wouldn't otherwise need. ## Product boundary `threatIntelSupplyEnabled` defaults to false. Do not enable until direct-index cross-space isolation is hardened or the administrator trust model is explicitly accepted. ## To test ``` node scripts/jest --config x-pack/solutions/security/plugins/security_solution/server/threat_intel/jest.config.js wiring.test.ts tasks/promote_threat_indicators.test.ts node scripts/jest x-pack/solutions/security/plugins/security_solution/server/workflows/ node scripts/jest src/platform/packages/shared/kbn-workflows/managed/definitions/threat_intel/ ``` _PR developed with Cursor + Auto + Sonnet 5 + Opus 4.8_ --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Jon Wålstedt <jon.walstedt@elastic.co> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

stephmilovic force-pushed the threat-intel-7-adapters-core branch 3 times, most recently from 52c2671 to fefcec8 Compare August 26, 2026 20:45

jonwalstedt force-pushed the threat-intel-7-adapters-core branch from f0e5f25 to ce056f2 Compare August 28, 2026 19:59

stephmilovic force-pushed the threat-intel-7-adapters-core branch from ce056f2 to df63099 Compare September 1, 2026 20:06

stephmilovic changed the title ~~[Threat Intel 9/11] RSS, STIX, and TAXII source adapters~~ [Threat Intel 9/11] RSS adapter with bounded structured-text serialization Sep 1, 2026

stephmilovic mentioned this pull request Sep 1, 2026

[Threat Intel] Move the indicator index out of the .kibana namespace #287479

Merged

stephmilovic force-pushed the threat-intel-7-adapters-core branch 2 times, most recently from 2714d11 to 0af170c Compare September 2, 2026 20:38

stephmilovic force-pushed the threat-intel-7-adapters-core branch from 5715d9e to 4447946 Compare September 4, 2026 17:33

This was referenced Sep 4, 2026

[Threat Intel] Scout API tests for the deterministic enrichment routes #289343

Merged

[Threat Intel] Enrichment eval suite for the LLM-backed routes #289345

Merged

stephmilovic and others added 12 commits September 4, 2026 14:01

feat(threat-intel): RSS, STIX, and TAXII source adapters

3c14ba6

fix(threat-intel): redact the last raw feed URL in the taxii adapter

6b62139

Missed in the previous commit: the zero-reportable-objects log line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Narrow ingest adapters to structured RSS

8325c7d

Narrow the fetch adapter source contract

5dd6ba0

Exclude hidden RSS fragment content

c2af0e7

fix(threat-intel): resolve RSS adapter URLs from the code catalog

c309e6c

After elastic#287201, feed URLs live in catalog_source_urls rather than source config. Co-authored-by: Cursor <cursoragent@cursor.com>

stephmilovic force-pushed the threat-intel-7-adapters-core branch from 4447946 to c309e6c Compare September 4, 2026 20:17

stephmilovic changed the base branch from threat-intel-6-llm-and-catalog to main September 4, 2026 20:18

stephmilovic marked this pull request as ready for review September 4, 2026 20:57

stephmilovic requested a review from a team as a code owner September 4, 2026 20:57

refactor(threat-intel): fix buildContext type cast and trim test comm…

ad575c5

…ents

jonwalstedt approved these changes Sep 8, 2026

View reviewed changes

Comment thread ...k/solutions/security/plugins/security_solution/server/threat_intel/adapters/rss/parse_rss.ts

Merge branch 'main' into threat-intel-7-adapters-core

d3af712

stephmilovic merged commit 2c3d5f9 into elastic:main Sep 8, 2026
42 checks passed

alexreal1314 pushed a commit to alexreal1314/kibana that referenced this pull request Sep 8, 2026

[Threat Intel 9/11] RSS adapter with bounded structured-text serializ…

aa52e0f

…ation (elastic#287205)

Navigation Menu

Uh oh!

[Threat Intel 9/11] RSS adapter with bounded structured-text serialization - #287205

[Threat Intel 9/11] RSS adapter with bounded structured-text serialization#287205
stephmilovic merged 14 commits into
elastic:mainfrom
stephmilovic:threat-intel-7-adapters-core

stephmilovic commented Aug 25, 2026 •

edited by jonwalstedt

Loading

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 4, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 4, 2026

Uh oh!

jonwalstedt left a comment

Uh oh!

Uh oh!

stephmilovic commented Sep 8, 2026

Uh oh!

kibanamachine commented Sep 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

stephmilovic commented Aug 25, 2026 • edited by jonwalstedt Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Where this sits

Scope review follow-up

Product boundary

To test

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 4, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 4, 2026

Uh oh!

jonwalstedt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

stephmilovic commented Sep 8, 2026

Uh oh!

kibanamachine commented Sep 8, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Test Failures

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

stephmilovic commented Aug 25, 2026 •

edited by jonwalstedt

Loading