[Threat Intel] Enrichment eval suite for the LLM-backed routes

[stephmilovic]

Summary

Adds the @kbn/evals suite for the four LLM-backed threat intel enrichment routes: assess_relevance, classify_severity, enrich_taxonomy, and extract_diamond.

Each example posts article text directly to the internal route and scores the structured response, so the suite measures the route contract rather than a chat transcript. It runs as a tracked baseline: scores are recorded and watched over time, with no build-failing thresholds yet.

The suite is split out from #287204 because the two halves are reviewed and gated differently. This is 1,335 lines of datasets, evaluators, and judges, reviewed for whether the scoring is sound, and it runs on the weekly and on-demand eval pipelines against a real connector. #287204 carries the routes themselves, which are gated by unit tests on every commit.

What it covers

Stage	Evaluators
assess_relevance	IsIntelligenceMatch, RelevanceShapeValid
classify_severity	SeverityExactMatch, SeverityWithinOneLevel
enrich_taxonomy	CategoryRecall, RegionRecall
extract_diamond	DiamondNoIocLeak, DiamondSignalCount, criteria (LLM judge, majority vote)

DiamondNoIocLeak is the one worth calling out: the Diamond model summarizes an intrusion, and leaking a raw indicator into that prose would put unvetted values in front of an analyst as though they were confirmed. The evaluator fails the example when that happens rather than scoring the summary on readability.

The Scout server config disables searchInferenceEndpoints on purpose, so resolveScopedModel takes the genAiSettings:defaultAIConnector path and the eval exercises the same resolution branch a deployment without a preconfigured endpoint would hit.

Baseline scores

From the recorded run on #287204, EIS Claude Sonnet 4.6 as the model under test, 1 repetition, 20 examples. That run predates the judge pin below, so its LLM-judged row was scored by Claude Sonnet 4.6:

Stage	Evaluator	Score
assess_relevance	IsIntelligenceMatch	1.00
assess_relevance	RelevanceShapeValid	1.00
classify_severity	SeverityExactMatch	0.83
classify_severity	SeverityWithinOneLevel	1.00
enrich_taxonomy	CategoryRecall	0.80
enrich_taxonomy	RegionRecall	1.00
extract_diamond	DiamondNoIocLeak	1.00
extract_diamond	DiamondSignalCount	1.00
extract_diamond	criteria (LLM judge, majority vote)	1.00

SeverityExactMatch at 0.83 with SeverityWithinOneLevel at 1.00 is the expected shape: every miss is off by one level, none inverts high and low. CategoryRecall at 0.80 is the weakest signal and the one to watch as the baseline accumulates runs.

Any prompt or calibration change these scores motivate belongs in #287204 with the route it affects, not here.

Judge

The suite runs with models:judge:eis/google-gemini-3.1-pro, so the LLM-judged criteria on extract_diamond are scored by Gemini 3.1 Pro rather than by whichever model is under test. Pinning one judge keeps that row comparable across the model matrix, since a per-model judge has each column scoring itself. The judge label maps to EVAL_CONNECTOR_ID in eval_pipeline.ts, so it applies on both the PR and on-demand runs.

The criteria evaluator keeps its majority vote over several judge samples. Voting handles per-run verdict flips on the conjunctive criteria, which is a separate problem from which model does the judging. The baseline table above will need a re-record under the new judge.

Testing

Verified locally in a bootstrapped worktree on this branch:

• node scripts/check.js --scope=branch exits 0 (moon, lint, tsc across 1498 projects, 828 jest tests)

The scores above come from the earlier run recorded on #287204, not from a run on this branch. playwright --list cannot enumerate an eval suite without a configured connector, since createPlaywrightEvalsConfig validates EVAL_CONNECTOR_ID at config load, so a fresh full run needs a provisioned EIS or LiteLLM connector.

Where this sits

#	PR	Depends on	Status
1	#287197 workflow gate correction + fixtures	nothing	✅ merged
2	#287198 contracts, constants, shared libs	nothing	✅ merged
3	#287199 content parsing	#287198	closed
4	#287200 SSRF-guarded HTTP client	#287198	✅ merged
5	#287201 index templates, seeding, inference features	#287198, #287200	✅ merged
6	#287202 indicator alias	#287198, #287201	✅ merged
7	#287203 IOC extraction	#287198, #287200	✅ merged
8	#287204 LLM services, routes, source catalog API	#287198, #287200, #287202, #287203	✅ merged
9	#287205 RSS adapter	#287198, #287200, #287203, #287204	✅ merged
10	#287206 remaining adapters, dispatcher, fetch_source step	#287198, #287200, #287203, #287205	✅ merged
11	#287207 promote/scrub tasks and plugin wiring	#287198–#287206	ready for review
12	#287479 generator fixture article URLs	nothing	✅ merged
13	#289343 Scout API tests for the deterministic routes	#287204	ready for review
14	#289345 enrichment eval suite ← this PR	#287204	ready for review

#287479 came first in the merge train and is already on main. The numbered series (#287197–#287205) is merged; #287206 and #287207 now sit directly on main. #287199 (content parsing) was closed; IOC extraction ships its own section-header classifier. Rows 13 and 14 are the test-only PRs and carry no production code.

Both test-only PRs cover code that merged in #287204 and now sit directly on main.

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-solution (Team: SecuritySolution)

[stephmilovic]

@elasticmachine merge upstream

[patrykkopycinski]

Reviewed the full diff against the PR head (1ba2488), and re-derived each finding from the committed tree rather than the description.

Solid suite overall: real ground truth instead of placeholder strings, CODE evaluators alongside the judge, a serverConfigSet that actually resolves, and an honest "tracked baseline, no build-failing thresholds yet" framing. The commit-2 taxonomy relabel is the right instinct — fixing bad ground truth beats inflating a score.

Three things I'd want addressed before the score table is treated as a baseline. Two are inline suggestions; the third is a request for a test.

1. withMajorityVote drops judge-model attribution (blocking)

evaluators.criteria() is assembled in kbn-evals/src/evaluate.ts as { ...createCriteriaEvaluator(...), getModel }. The wrapper rebuilds the object literal field-by-field, so getModel (and getVersion) are lost. kibana_evals_executor/client.ts:276 reads model: evaluator.getModel?.() when writing the score, so every criteria score from this suite lands with model undefined and the judge becomes unattributable in the scores index. Inline suggestion below.

2. Empty-set recall is a guaranteed 1.00 (blocking)

recall() returns 1 when expected is empty. Commit 2 set regions: [] for aws-iam and github-actions, so RegionRecall scores a hard 1.00 on 2 of 5 examples regardless of what the model returns — which is where the reported RegionRecall 1.00 comes from.

buildStatsAggregation (kbn-evals-common/impl/query_builders.ts:241) aggregates evaluator.score with extended_stats, which skips nulls but averages in 1s. So "no labels to recall" must be score: null (dropped from the mean), not 1 (inflates it). The existing comment — "recall is trivially satisfied — there is nothing to recall" — describes exactly the condition that should be N/A. Inline suggestion below; same latent issue for CategoryRecall if a pack ever gets an empty category set.

3. DiamondSignalCount may not be able to fail (blocking)

signal_count is computed server-side as countNonNone(output) (#287204), the floors are >= 3 / >= 3 / >= 2 on three packs dense in all four vertices, and scoring is binary count >= min. Together with DiamondNoIocLeak both reporting 1.00, this is consistent with a check that cannot go red.

Could you add a mutation test that proves it bites — force a vertex to NONE (or use a genuinely sparse article) and assert the score goes to 0? If it can't be made to fail, two of the three extract_diamond evaluators aren't measuring anything.

Non-blocking

• No jest.config.js and no unit tests. 11 of 16 sibling security eval suites ship one, and withMajorityVote is real logic — vote tallying, tie-resolution (passes * 2 >= results.length), N/A handling, the byId.size === 0 fallback — currently untested. Matching kbn-evals-suite-alerts-rag verbatim:

  module.exports = {
    preset: '@kbn/test',
    rootDir: '../../../../..',
    roots: ['<rootDir>/x-pack/solutions/security/packages/kbn-evals-suite-threat-intel-enrichment'],
  };

  Worth noting tsconfig.json already declares "types": ["jest"] with no jest config present. For the fixes above, the regression tests should fail on the current committed code — otherwise they're not gates.
• packs.ts carries dead fields. After commit 2, regions and mitre have zero references in the package, and categories survives only as classify_severity input. Two sources of taxonomy truth invites a future edit to the wrong one.
• withMajorityVote is arguably platform code. createMultiJudgeEvaluator is already exported from @kbn/evals with a 'majority' strategy and unit tests. It aggregates across different judges rather than repeated samples of one, so it isn't a drop-in — but a per-criterion self-consistency wrapper looks generic. If it moves, it should be its own PR, not folded in here.
• .eslintrc.js turns off import/no-nodejs-modules, but the package imports no Node builtins.
• samples = 3 triples judge cost per example — worth a comment on the intended trade-off.

On the reported scores

I'd hold off on reading the table as a baseline: RegionRecall 1.00 is partly arithmetic (#2), DiamondSignalCount / DiamondNoIocLeak are unproven-to-fail (#3), and the criteria rows are unattributed to a judge model (#1). SeverityExactMatch 0.83 and CategoryRecall 0.80 look like the only numbers carrying real signal today — and they're the two that can actually move.

Separately, the run uses Claude Sonnet 4.6 as both model and judge. Self-judging measurably biases scores, and the 1.00s sit at the ceiling where saturation hides variance. A multi-model run with a cross-family judge would make this a much stronger baseline. Also worth flagging for anyone reading the green checkmark: kibana-evals has no score threshold in its path, so a green check says nothing about suite quality.

[patrykkopycinski]

Suggested change

	name: base.name,
	export const withMajorityVote = (base: Evaluator, samples = 3): Evaluator => ({
	...base,

Spreading base preserves getModel / getVersion, which kibana_evals_executor/client.ts:276 reads (model: evaluator.getModel?.()) to attribute the score to the judge model. Rebuilding the literal field-by-field silently drops them, so every criteria score writes with model undefined.

[stephmilovic]

Fixed in 03657e6. withMajorityVote now spreads base, so getModel / getVersion survive and every criteria score attributes to the judge model again instead of writing model undefined. Added a unit test asserting the wrapped evaluator keeps both.

[patrykkopycinski]

Suggested change

	/** Fraction of the labelled set that appears in the model output (recall). */
	const recall = (expected: string[] | undefined, actual: string[] | undefined): number => {
	if (!expected || expected.length === 0) return 1;
	/** Fraction of the labelled set that appears in the model output (recall). */
	const recall = (expected: string[] | undefined, actual: string[] | undefined): number | null => {
	if (!expected || expected.length === 0) return null;

extended_stats skips nulls but averages 1s, so returning 1 for "nothing to recall" inflates the evaluator mean (this is what makes RegionRecall read 1.00 on the regions: [] packs). Returning null drops those examples from the mean instead. Callers need to handle it, e.g.:

evaluate: async ({ output, expected }) => {
  const score = recall(expected?.regions, output?.regions);
  if (score === null) {
    return { score: null, label: 'N/A', explanation: 'No labelled regions to recall' };
  }
  return { score, label: `recall_${score.toFixed(2)}` };
},

[stephmilovic]

Fixed in 03657e6. recall() returns null (not 1) when the labelled set is empty, and both CategoryRecall and RegionRecall surface it as { score: null, label: 'N/A' } so extended_stats drops the example from the mean rather than averaging in a 1. A unit test covers the empty regions: [] case that was producing the RegionRecall 1.00.

[patrykkopycinski]

Is there a case where this can score 0? signal_count is server-computed as countNonNone(output), and the dataset floors are 3/3/2 on packs dense in all four vertices — so count >= min may hold unconditionally.

A mutation test would settle it: force a vertex to NONE (or add a genuinely sparse article) and assert the score drops to 0. If it can't be driven red, this evaluator and DiamondNoIocLeak are both reporting 1.00 without measuring anything.

[stephmilovic]

Added the mutation test in 03657e6: it forces the victim vertex to NONE with signal_count below the floor and asserts DiamondSignalCount scores 0, so the check demonstrably bites rather than being inert. To be precise about what that proves: it shows the evaluator logic is live, not that the current three packs ever drive it red. On those packs a passing 1.00 is the expected result for genuinely dense articles; the evaluator is the floor guard against an all-NONE collapse on a report the pipeline sent to extraction.

[stephmilovic]

Addressed the review in 03657e6.

Blocking

1. withMajorityVote now spreads base, restoring getModel / getVersion attribution.
2. Empty-set recall() returns null and both recall evaluators emit an N/A result, so those examples drop out of the extended_stats mean instead of inflating it.
3. Added a mutation test that drives DiamondSignalCount to 0 on a NONE vertex below the floor.

Non-blocking

• Added jest.config.js and unit tests. They cover withMajorityVote (vote tally, tie-to-PASS, N/A, byId.size === 0 fallback, model/version preservation), the recall N/A behavior, and the signal-count mutation. The vote and recall tests assert the fixed behavior, so they fail on the pre-fix code.
• Dropped the dead regions / mitre fields from packs.ts. categories stays since classify_severity still consumes it; taxonomy recall ground truth already lives in enrich_taxonomy_dataset's curated set.
• Removed the import/no-nodejs-modules override. The package imports no Node builtins. It matches a sibling-suite convention, so tell me if you would rather keep it for consistency and I will restore it.
• Documented the samples = 3 judge-cost trade-off.

On moving withMajorityVote to platform
Leaving it in the suite for now. As you noted it is not a drop-in for createMultiJudgeEvaluator (per-criterion self-consistency over repeated samples of one judge vs aggregating different judges), so if it graduates to @kbn/evals it should be its own PR rather than folded in here.

On the baseline and re-running
Agreed the current table should not be read as a baseline: fix #2 changes RegionRecall, fix #1 changes judge attribution, and the run used the model as its own judge. Planning to regenerate the numbers with a cross-family judge so the baseline carries real signal, and will post the refreshed table rather than treating the current one as the bar.

[stephmilovic]

Refreshed baseline with a cross-family judge

Re-ran the suite locally after the fixes, with the judge on a different family from the model under test.

• Model under test: anthropic-claude-4.6-sonnet
• Judge: openai-gpt-5.4 (cross-family, no self-judging)
• 3 repetitions, local Scout stateful (evals_threat_intel config), EIS connectors

Stage	Evaluator	Mean	Notes
assess_relevance	IsIntelligenceMatch	1.00
assess_relevance	RelevanceShapeValid	1.00
classify_severity	SeverityExactMatch	0.83	std 0.37, min 0, real signal
classify_severity	SeverityWithinOneLevel	1.00
enrich_taxonomy	CategoryRecall	0.80	std 0.24, min 0.50, real signal
enrich_taxonomy	RegionRecall	1.00	over the 3 labelled packs; 2/5 packs carry no region and now score N/A (excluded from the mean)
extract_diamond	DiamondNoIocLeak	1.00
extract_diamond	DiamondSignalCount	1.00	on these dense packs; the added mutation test proves it scores 0 on a NONE vertex
extract_diamond	criteria (majority-3x)	0.84	std 0.08, cross-family judge

What changed versus the original table:

• RegionRecall is no longer a free 1.00. The two empty-region packs score N/A and drop out of the mean, so the 1.00 now reflects only the 3 packs that actually carry a region label.
• criteria moved off the ceiling: 0.84 with the cross-family judge instead of the near-1.00 self-judged number, and it now attributes to openai-gpt-5.4 in the scores index (fix links to styles and js #1 verified end to end).
• SeverityExactMatch (0.83) and CategoryRecall (0.80) remain the two numbers carrying the most signal, as you called out.

Still saturated: DiamondNoIocLeak and DiamondSignalCount sit at 1.00 on the three dense packs. The mutation test now demonstrates DiamondSignalCount can go red, so it is a live check rather than an inert one, but the current dataset does not exercise the floor. A genuinely sparse article would be the follow-up to make that number move.

Caveats: single model under test, local run, small dataset (20 examples x 3 reps). Treat this as a directional baseline, not a CI gate. kibana-evals has no score threshold in its path, so a green check still says nothing about these numbers.

[cla-checker-service]

💚 CLA has been signed

[stephmilovic]

Refreshed the baseline after the review fixes. 3 reps, anthropic-claude-4.6-sonnet under test, google-gemini-3.1-pro as the judge.

Dataset	Evaluator	N	Mean	Min	Max
assess_relevance	IsIntelligenceMatch	18	1.00	1	1
assess_relevance	RelevanceShapeValid	18	1.00	1	1
classify_severity	SeverityExactMatch	18	0.83	0	1
classify_severity	SeverityWithinOneLevel	18	1.00	1	1
enrich_taxonomy	CategoryRecall	15	0.80	0.5	1
enrich_taxonomy	RegionRecall	9	1.00	1	1
extract_diamond	DiamondNoIocLeak	12	1.00	1	1
extract_diamond	DiamondSignalCount	12	1.00	1	1
extract_diamond	criteria	12	0.95	0.8	1

On the saturated diamond checks: I added a deliberately sparse extract_diamond example, a technique-only advisory that genuinely supports one vertex (capability) and nothing else, at min_signal_count: 1. The model returned capability = PARTIAL and adversary/infrastructure/victim = NONE, so signal_count = 1 against a floor of 1. Zero margin, drop that one vertex and DiamondSignalCount goes to 0. It passed all 3 reps because the model doesn't collapse, but it's a real boundary on live data now instead of saturated by construction (the dense packs clear their 3/3/2 floors by 1-2).

Two other things from the fixes show up here: RegionRecall averages over N=9 because empty-expected examples now return N/A instead of a free 1.0, and CategoryRecall (0.80, min 0.5) and SeverityExactMatch (0.83, min 0) are genuinely discriminating rather than pinned at 1.0.

[patrykkopycinski]

🙇

[kibanamachine]

PR run: bk-01a0929f-d13d-429d-841d-65fbc9ba3684::smoke-tests::anthropic-claude-4.5-haiku | Baseline (main): bk-01a086a9-0774-4273-8efa-e2b095b5e434::smoke-tests::anthropic-claude-4.5-haiku
Baseline: commit 144b3c5, 2 days ago
Significance threshold: p < 0.05

Summary
No significant regressions detected (5 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (5 rows)

Dataset	Evaluator	N	Mean (PR)	Mean (main)	Diff	p-value	Sig	Outcome
smoke tests: es-snapshot-loader	SnapshotRestored	1	1.00	1.00	0.00	-	n/a	-
smoke tests: llm-judge	Criteria	1	1.00	1.00	0.00	-	n/a	-
smoke tests: score ingestion and code evaluator	ContainsKibana	1	1.00	1.00	0.00	-	n/a	-
smoke tests: trace-retrieval	Input Tokens	1	12.00	12.00	0.00	-	n/a	-
smoke tests: trace-retrieval	Output Tokens	1	5.00	5.00	0.00	-	n/a	-

[kibanamachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: ab5c6a0
• Build duration: 107 mins
• Storybooks Preview

Failed CI Steps

• FTR Configs #76
• Serverless Rule Management - Prebuilt Rules Upgrade - Security Solution Cypress Tests #1

Metrics [docs]

Unknown metric groups

warm start memory

id	before	after	diff
post forced gc heap baseline	-	835592722	+835592722
post forced gc heap delta	-	-2996264	-2996264
post forced gc heap delta standard deviation	-	2191710	+2191710
post forced gc heap target	-	832596458	+832596458
tail heap delta	-	-26199403	-26199403
total			+1641185223

History

• 💔 Build #500456 failed 2d4d3ce
• 💔 Build #499214 failed 8d826d3
• 💛 Build #498531 was flaky 180ad87
• 💛 Build #498467 was flaky ccc8555
• 💛 Build #498322 was flaky 1ba2488