[Evals] Security LLM performance matrix generator in @kbn/evals-extensions

[patrykkopycinski]

Summary

Consolidated PR: the security LLM performance matrix generator plus every persona-matrix suite fix, so that merging this single PR leaves the matrix and the suite fully working together. Supersedes #283586; folds in (now-closed) #286165, #286201, #286421.

1. Matrix generator (@kbn/evals-extensions)

• Matrix generation: build_matrix, query_matrix_scores, query_matrix_traces; HTML rendering; matrix.ts CLI command
• Per-category columns via examplePrefixes (slices the 21-example persona-matrix dataset into its 7 categories), grouped under a shared header
• Trace detail lookup via the per-example scores route (the per-experiment route strips task.output/example.input, which made every trace render "Trace unavailable"); graceful degradation when one example's scores exceed the transport limit
• Per-variant trace cards: category columns aggregate a/b/c variants — previously only the first-wins representative rendered, hiding 14 of 21 prompts
• Golden cluster privileges: create_doc → create for serverless; EvalsClient restored to upstream API + additive listExperiments/getExampleScores

2. Persona-matrix suite fixes

• ExpectedToolCalled scores the full declared tool set (was expectedTools[0] only): 16 of 21 examples had later declared tools unenforced. Reports missingToolIds. Offline re-scoring confirms no model-ordering flip (4.8-opus < 4.5-opus under first/any/all/fraction).
• Threat-hunting skill: generate_esql is load-bearing (required before execute_esql; findings only from executed queries) — frontier models were hand-writing ES|QL and skipping the schema-validating generator.
• SkillInvoked matches load_skill IDs (folded [Security Solution][kbn-evals] Match load_skill IDs in persona matrix #286165): recognizes both {"skill":"<id>"} arguments and SKILL.md paths.
• Environment-truth seeding (folded [Security Solution][kbn-evals] Persona matrix: seed environment-truth data so every prompt is satisfiable #286421): seeds endpoint telemetry, Security Labs content, TI mock, entity-store records so every prompt is satisfiable; idempotent custom-tool reinstall with correct esql params contract.
• Scout maxPayload 50MB for the suite's config set (folded [Scout] Raise server.maxPayload to 50MB in evals_security_persona_matrix config set #286201): judge payloads on long trajectories exceed the 1.6MB default.
• Configurable Playwright timeout (PERSONA_MATRIX_TIMEOUT_MINUTES, default 30 unchanged): determinism runs at 3 repetitions (~90min) died at the fixed ceiling.

Verification

• type_check: 0 errors on both touched projects (kbn-evals-extensions, kbn-evals-suite-security-persona-matrix)
• Jest: matrix suites 64/64; persona-matrix evaluator suite 16/16
• ESLint clean on all changed files; pre-push hook PASS
• E2E evidence from live runs against the golden cluster: haiku smoke 21/21 with 252/252 scores after env-seeding; 5/5 models render question/tool trail/step trace/answer in matrix.html; sweep VM runs exercise the overlaid evaluator + timeout config end-to-end

3. Suite-aware rejudge + reference-grounding correctness (this update)

• Per-suite reference adapters (14d7690bc1b7): rejudge juries now load ground truth through a per-suite adapter instead of assuming a single reference shape.
• Suite-aware rejudge juries + newest-execution selection (6d96ffa11ba7): the AD jury recomputes Criteria/Rubric; replay selects the newest execution rather than first-found.
• Attack-discovery references joined by scenario key (e85d529fea08): prose references are keyed per-scenario so the jury grades against the right alert profile.
• Structured ground truth keyed like the prose references (db8bc1277b12): buildStructuredReferences and the prose map derive the same exampleKey, so both lookups agree. Mutation-proven: the drift test fails when the bug is reintroduced.
• Judge provenance derived, not asserted (d04a2f760d01): new deriveJudgeProvenance counts the judge on every admitted run and refuses to name a single judge unless unanimous; unrecorded judges read "unknown" instead of inheriting one. 6 tests, mutation-verified (restoring the hardcoded id turns 3 red; counting unrecorded judges as a judge turns 2 red).
• Golden extraction is now reproducible (d04a2f760d01): the published board previously depended on an untracked scratch file for its extraction step, so its provenance could not be checked against the data it summarised. extract_golden_aggregate.ts is committed, source-filters to the fields it aggregates (full _source pulls every transcript into memory: ~2GB RSS, no forward progress), and logs per-suite progress. Verified live: 30,534 docs across 3 suites, 41 models, exit 0, byte-identical across two runs.

Known limitations (honest disclosures)

• The board is mixed-judge across every column, and the artifact previously mis-declared this. render_from_golden.ts hardcoded judgeModelId: 'google-gemini-3.1-pro' into the published provenance, and separately hardcoded a judge census into the disclosure notes; nothing checked either against the scores they described. Both are now derived at render time by deriveJudgeProvenance (per-suite mix in the table below), so the note cannot drift from the data it summarises. The persona columns are not graded by one shared instrument, which is the assumption the published spread/CI figures rest on. Since re-scoring identical trajectories with a different judge reorders this board (Spearman rho=0.55), cross-row comparisons mix instruments and must not be read as model-quality differences. Re-deriving the spread/CI statistics per-judge is v2 scope — this corrects the label, not the underlying mixed-instrument comparison.
• AD-column rejudge is incomplete, with two distinct unresolved failure modes. A full-column rejudge (9 models × 27 criteria × 3 shards) is not done. Observed across identical invocations of the same single-execution rejudge (judge stack confirmed healthy throughout: CCM enabled, full gemini catalog, Kibana 200): (1) selection variability — the newest-execution selection lands on different golden experiment_ids run-to-run, sometimes fetching 0 docs, sometimes 4-but-ungradable ("missing gradable attack-discovery output"); and (2) jury non-production — even with 4 gradable cells planned, the jury intermittently emits nothing gradable into Criteria/Rubric (0 cells, 4 failed). Mode (2) means judge-side behaviour cannot yet be ruled out. The v2 fix must first capture the jury's raw output on a 0-cell verdict to distinguish a judge flake from a grading-contract mismatch, then constrain/pin replay to executions with gradable AD output and surface "no gradable output" as a first-class data gap.
• CCM (Elastic Inference Service model sync) is in-memory only on the judge stack. PUT /_inference/_ccm enables it, but it lives in ES's in-memory cache and reverts to enabled:false on restart. Mitigated on the m1max judge host: the boot script now runs a background helper that PUTs the CCM key and gates on the gemini endpoint materializing after every ES restart (verified live on a fresh boot). Any other judge host needs the same step or it silently regresses to "connector not found".
• Golden docs are keyed on experiment_id, not execution_id — merging re-judged scores back into golden requires a dedicated write path that does not yet exist.

Judge attribution is confounded — the board does not support cross-model ranking

Counted from the golden extract the board is built from (42 graded models), derived at render time rather than asserted:

Suite	models	judge mix	graded by >1 judge
security-persona-matrix	39	haiku 22, sonnet-4.6 15, gemini-3.1-pro 3, gpt-5.4 1	2
attack-discovery-agent-builder	32	sonnet-4.6 18, gemini-3.1-pro 15	1
automatic-migrations	9	gemini-3.1-pro 8, kimi-k3 1	0

Board-wide: 24 of 42 models were graded by more than one judge, 18 by exactly one (10 under sonnet-4.6, 5 under gemini-3.1-pro, 3 under haiku). An earlier revision of this PR stated the opposite — "no persona-matrix model is graded by more than one judge (zero overlap)" — from a narrower extract. That claim is withdrawn: it is false on the current data, and the renderer no longer hardcodes any judge census. Every figure above is now derived by deriveJudgeProvenance at render time and mutation-tested (fabricating the judge mix in the extract changes every number in the published note).

The confound is real but is not clean block separation: for the 18 single-judge models, judge severity is collinear with model identity and cross-block comparison is invalid; for the 24 multi-judge models, the cell mean itself averages over graders. Since re-scoring identical trajectories with a different judge reorders this board (Spearman rho 0.55), neither group supports reading row order as model quality.

Within the largest single-judge block, the LLM-judged composite correlates only Spearman 0.483 / Pearson 0.505 with the judge-independent deterministic evaluators (ExpectedToolCalled, MinExpectedSteps, FinalAnswerPresent, SkillInvoked). gpt-5.5 ranks 1st by judge but 11th of 25 deterministically. The judged axis is a separate instrument from what the model actually did.

Sep 7 published board mislabelled its judge. target/llm_matrix_golden/matrix.json declares judgeModelId: google-gemini-3.1-pro, but golden shows its persona scores that day were graded 3842 docs by claude-4.5-haiku, 2122 by claude-4.6-sonnet and only 420 by gemini-3.1-pro. It is a mixed-judge aggregate labelled single-judge. Re-rendering it under the declared judge populates only 6.3% of cells, so the published numbers cannot be reproduced single-judge; the fidelity guard now refuses that render instead of emitting a near-empty board.

The judge notes are now unconditional. They previously rendered only when optional caveat-stats env vars were set, so a plain render published a board with no judge disclosure — which is how the false claim survived.

Judge-overlap experiment — the confound is now measured, not assumed

That data-collection run has since been executed. Six models sharing an identical
7-column persona block were re-judged blind by all four judges: 507 graded
cells, 124 of them graded by every judge, 1 transient toolValidationError and
1 unreplayable cell, both recorded in the artifacts rather than dropped.

Analysis is committed (src/matrix/judge_overlap.ts, 8 tests, 3 mutations verified
red) so the numbers below come from the same code path CI runs, not a scratch script.

Judge severity splits into two parts. A uniform offset shifts every model
equally and cancels out of any within-judge ranking. The residual that survives
offset removal is what actually reorders a board.

Judge	mean	offset
anthropic-claude-4.5-haiku	0.794	+0.064
google-gemini-3.1-pro	0.726	−0.004
anthropic-claude-4.6-sonnet	0.713	−0.018
openai-gpt-5.4	0.688	−0.042

The decisive comparison: residual disagreement is 0.170 mean / 0.303 p90,
against a model spread of only 0.155. Judge noise exceeds the signal it is
meant to measure, so analyzeJudgeOverlap returns rankable: false — that
verdict is about reading any single judge's column as a ranking, which is what
the shipped board does.

Consequences for reading this board:

• eis-openai-gpt-5-5 is 1st under all four judges — the only rank that is
  judge-independent under any single judge. (Averaging the judges recovers more
  than this; see the ensemble column below.)
• Every position below the top reorders by judge. glm-5.3-flash moves 4th→last,
  gemini-3.1-pro 2nd→5th. Mean pairwise Spearman on model order is +0.533
  (min +0.371).
• Haiku is the most lenient judge (+0.064) and graded 67.6% of the shipped
  board. That leniency is baked into the largest column.
• No self-preference detected: judge-family vs model-family deltas span
  −0.027 to +0.006, all within noise. The earlier family-bias hypothesis for the
  attack-discovery column is unsupported.

Caveat stated plainly: 6 models × ~20 cells is a small block, so individual
pairwise rho values carry wide confidence intervals. The robust finding is the
residual magnitude relative to model spread, not any specific correlation.

Ensemble column — what the board can now rank

Averaging the four judges over the 124 cells they all graded is the remedy the
overlap data implies: judge-specific scatter is independent, model quality is
not, so the mean of N judges suppresses the noise without touching the signal.
Committed as src/matrix/ensemble_column.ts (7 tests, 3 mutations verified red)
and rendered into the board from the real artifacts.

#	Model	Ensemble	Judge spread
1	eis-openai-gpt-5-5	0.779	0.153
2	openai-gpt-5.4	0.746	0.165
3	google-gemini-3.1-pro	0.735	0.083
4	anthropic-claude-4.8-opus	0.709	0.113
5	z-ai/glm-5.3-flash	0.706	0.139
6	google-gemini-2.5-flash	0.704	0.056

Measured: 8 of 15 model pairs separate by paired bootstrap over shared
examples (95% CI excluding zero), 2 of them borderline. That figure does not
survive contact with run-to-run variance, and this PR no longer claims any
persona separation is real.

Retraction (supersedes the corrections below). The bootstrap resamples
cells within a single run, so it measures judge and example variance but is
structurally blind to run-to-run variance — a one-run sample has none to see.

The golden data contains 7 models that were run twice under two naming
conventions (eis-* on Sep 7, bare ids on Sep 6). Those pairs are the same
model on the same columns, and they are the only re-run evidence available:

mean |Δ| across 7 re-run pairs (0–1 quality evaluators)	0.099
cells reproducing identically	~70%
ensemble pairwise gaps	median 0.033, max 0.075
gaps exceeding the re-run noise floor	0 of 15

Every pairwise gap on the persona ensemble — including all 8 "separable" and
all 6 "robust" — is smaller than the disagreement between two runs of the same
model. The correct reading is that the persona column orders nothing yet.

Measured directly (2026-09-09). The upper bound above has since been
confirmed by a dedicated seed-only re-run, so the retraction no longer rests on
inference from differently-named pairs. Three models were re-run on the
identical suite, examples and judge, with the seed as the only difference
(VM_NAME_SUFFIX=seedrep, 9/9 shards PASS, 98/98 docs each, 882 score docs):

model	cells	mean |Δ|	identical
eis-anthropic-claude-5-sonnet	194	0.1072	61%
eis-google-gemini-3-1-pro	194	0.0873	62%
eis-openai-gpt-5-5	194	0.0945	68%
combined	582	0.0963	~64%

The seed-only floor is 0.0963, within 0.003 of the 0.099 upper bound — the
naming-convention confounds contributed almost nothing, and this is genuine
sampling variance. 0 of 15 ensemble gaps clear it; the largest gap in the
column is 0.0750. About a third of cells change value on a pure re-run, which
is the discrimination limit of a single-run persona sweep. Averaging more
judges cannot recover it, because the variance originates in the model's own
sampling rather than in judge disagreement.

A doc-count gate cannot catch a wrong-suite run

While closing the migrations coverage gap, a sweep launched with
--suite security-automatic-migrations reported 8/8 units PASS at
docs=86/80 and filled zero board cells.

The board's two migrations columns key on dataset ids
05fd1e03-… / 07a6c75d-…, written by the agent-builder suite
(kbn-evals-suite-agent-builder, experiments agent builder: automatic-migration-start-end-to-end / -update-grounded). The similarly named
security-automatic-migrations suite is a different package with a different
playwright.config.ts, and it wrote 688 docs under three unrelated datasets
(standard-dashboards, qradar, splunk-spl). Migrations coverage is therefore
still 8/41 — the run was green and useless.

The floor gate was structurally blind to this: 86 >= 80 holds regardless of
which datasets produced the 86, so a gate that only detects undercounts cannot
detect running the wrong thing. check_expected_datasets now asserts dataset
identity for suites that declare expected_dataset_ids, and is verified by
replay against the real mig2 execution ids — every unit that passed at 86/80
fails with wrong suite: unexpected datasets …; missing datasets …. Suites that
declare no identity (persona, AD) keep their existing behaviour.

One mutation initially survived here: deleting the unexpected-dataset check
entirely still passed, because the wrong-suite fixture also triggered the
missing branch and passed for the wrong reason. A superset case now isolates
the unexpected-only path; 5/5 mutations caught.

Correction. An earlier revision of this PR claimed 8 of 15. That figure came
from an ad-hoc script with two defects, both now fixed in committed code:

1. Index pairing. countSeparablePairs paired the two models by array
   position. Each model's cell list is built independently, so position i is a
   different example per model — the "paired" bootstrap compared unrelated cells
   and leaked example difficulty, the dominant variance component, into every CI.
   It now pairs on exampleId.
2. Evaluators counted as independent. The ad-hoc version treated each
   evaluator as its own observation, inflating n by 4x. Within-example evaluator
   spread is 0.636 against a between-example sd of 0.094, so evaluators on one
   example measure different constructs and are not independent draws.

The board and the module now derive both counts from the artifact, so a re-run
that separates differently cannot leave a stale number on the page.

Third correction, found by a lint failure. no-bitwise rejected the
Mulberry32 PRNG, so the bootstrap moved to a plain-arithmetic LCG — and the
count moved again, 7 → 8. Same data, same estimator, different generator.
The cause is real and worth stating: one pair (opus-4.8 vs
claude-5-sonnet) sits 0.0019 from its CI edge and separates in only 4%
of seeds. A bare "N pairs separate" was reporting a coin flip as a verdict.
buildEnsembleColumn now returns borderlinePairs (CI edge within 0.005 of
zero) and the board publishes "only N separate robustly" beside the headline.
8 separate, 2 borderline, 6 robust.

The 7 pairs that remain tied are not ordered, and the board says so:

• opus-4.8 vs gemini-2.5-flash: +0.0053 CI[−0.0368, +0.0429] — a coin flip.
• gpt-5.4 vs gemini-3.1-pro: +0.0115 CI[−0.0172, +0.0402] — not distinguishable.

Honest reading: #1 is real, #2–3 are a tier, #4–6 are a tie. Ranks 4–6 span
0.005 points and must not be quoted as an ordering.

Each ensemble score publishes its judgeSpread alongside, because a 0.5 produced
by judges at 1 and 0 is not the same claim as a 0.5 they agreed on. Only cells
graded by every judge are averaged, so a judge that failed on a model's hard
examples cannot make that model look strong through absence. The sqrt(N) noise
reduction is reported as an upper bound — judges are correlated, not
independent.

Evaluator health gate — the rubric defect was not isolated

The attack-discovery rubric was found by hand after 240 observations had already
been spent on it. Auditing the rest of the golden extract the same way turns up
more of the same class, so this adds checkEvaluatorHealth to make the check
mechanical rather than retrospective.

Across the 32 scored evaluators in the golden extract (12,053 observations), only
38% actually discriminate between models:

finding	count	evidence
constant 1.000 for every model	3	Sequence Accuracy, RequiredAlertIdsInResponse, DocVersionReleaseDate — 159 observations each, min = max = 1
gates that never pass for anyone	3	Skill Invoked (data-exploration) scores 0 across all 237 observations
saturated graders	1	persona Sequence Accuracy, 94.6% at ceiling

This distorts the published board. Recomputing composites from discriminating
evaluators only:

suite	models changing rank	largest move
attack-discovery-agent-builder	26/38	14 places
security-persona-matrix	47/58	7 places

gemini-3.1-flash-lite moves 14 places on attack-discovery depending purely on
whether near-constant evaluators are averaged in.

Gates are deliberately held to a different standard: "did it avoid the forbidden
tool?" is supposed to sit at the ceiling, and failing it for saturation would
train people to ignore the warning. A gate is only flagged when nobody passes it.

Two caveats on this analysis. The composite used here is an unweighted mean over
evaluators, which is not exactly how the board ranks, so treat the rank moves as
evidence that evaluator selection matters, not as a corrected ranking. And role
(gate vs grader) is currently inferred from evaluator names in the audit script
— that belongs in the evaluator definition, since a name list can be widened
until the audit passes, which is the failure this gate exists to prevent.

Filed for the suites outside this PR's scope: #289935.

Attack Discovery golden-path slices could not be rejudged — now fixed forward

The AD column has 10 slices in golden. The four scenario-registry (clean profile)
slices were rejudged under the per-item rubric (99 cells, all four judges 99/99).
The five golden-path slices could not be, and the cause was a missing join key
rather than a scheduling gap:

golden-path (live-retrieval)      n=2969  scenarioKey: 100% missing
golden-path (provided-alerts)     n=2378  scenarioKey: 100% missing
golden-path (missing-alert-r...)  n=2226  scenarioKey: 100% missing
golden-path (multiple-alert-...)  n=2210  scenarioKey: 100% missing
golden-path (status-only)         n=2210  scenarioKey: 100% missing
scenario-registry (clean profile) n=1749  scenarioKey: encoded-powershell, ...

A replay joins recorded scores to dataset references on
example.metadata.scenarioKey. The golden-path documents carry no metadata, and
example.id is '0' for 11,512 of 11,993 documents, so nothing distinguished
one scenario from another. Replaying them would have graded all five against the
first scenario's ground truth — the run would have succeeded and the numbers
would have been meaningless.

This PR gives each golden-path scenario a scenarioKey and makes the field
required on the metadata type, so the compiler now rejects a scenario that
cannot be replayed. Four tests pin the contract (present, unique, agreeing with
the fixture the spec slices on, non-empty); all four fail without the fix and
three targeted mutations are caught.

This does not retroactively fix existing golden data. Documents already
recorded have no scenarioKey and stay unreplayable; the board's five golden-path
AD cells remain on old-rubric scores, labelled not comparable. The next suite run
records the key and those slices become rejudgeable from then on.

Evaluator roles are declared, not inferred

The health gate above shipped with roles inferred from evaluator names. That is
the wrong mechanism for a correctness check — a pattern list can be widened until
the audit passes, which is the failure the audit exists to catch.

It was already happening: RequiredAlertIdsInResponse and DocVersionReleaseDate
matched the gate patterns, so the audit stopped asking whether they could ever
fail, while both return a constant 1.000 across all 159 golden observations.
Replacing the regexes with a declared registry surfaces both as dead graders
rather than excusing them.

To be precise about what changed: the constant count in the table above stays at
3, because those two were already being counted from raw golden data in the
retrospective analysis. What changed is the tool's verdict — it now reports
them instead of classifying them as healthy gates. The one count that moved is
saturated graders, 2 to 1, because ToolUsageOnly genuinely is a gate and the
regex had it right by accident.

Unregistered evaluators resolve to unknown rather than defaulting: guessing
grader fails a healthy gate, guessing gate excuses a dead one. Each
declaration carries a rationale, and a test pins the two known-constant
evaluators as graders so they cannot be relabelled as gates to silence the audit
— the cheapest way to make this tool useless.

Attack Discovery column — corrected twice, root cause found

Two earlier revisions of this description were wrong about AD, and the second
wrong answer was produced while fixing the first. Both are recorded here because
the board's credibility rests on how these numbers were obtained.

1. First claim: "AD is a 22-model, single-example inventory column needing a
   suite re-run." Wrong — that was an artifact of reading one slice.
2. Second claim: "the matrix config named three of the nine AD datasets."
   Also wrong. The config does not drive extraction at all; the extractor takes
   no config and discovers suites itself.

Actual root cause, in extract_golden_aggregate.ts: the extractor selected
the newest experiment_id per suite. Persona runs all of its columns inside
one experiment, so a suite-wide filter is harmless there and the bug stayed
invisible. Attack discovery runs each slice as its own experiment, so the
filter kept one slice and silently discarded the other eight. count: 1 was the
symptom of a selection bug, not a property of the data.

Selecting the newest experiment per column instead:

Suite	Before	After
attack-discovery	31 models × 1 dataset	38 models × up to 10 slices
persona-matrix	7 columns	21 columns
automatic-migrations	2 columns	21 columns

Golden holds 2,881 AD documents graded on Criteria/Rubric across 38 models;
30 models carry all nine slices. AD never needed new examples or a suite re-run.

A suite can now carry several judges, so judgeModelIds is reported and
judgeModelId goes undefined rather than naming one of several. Judge
provenance counts the full set — counting only the single-judge field made mixed
runs vanish from the breakdown built to expose them. AD is in fact mixed:
16 runs gemini-3.1-pro, 16 claude-4.6-sonnet, 3 others.

Two harness bugs blocked reaching this column at all, both fixed here:

• The attack-discovery reference adapter listed every dataset export name except
  goldenPathExamples, the one the agent-builder suite actually uses, so
  --dataset on that suite failed with "does not export an examples array".
• JUDGE_KBN_AUTH is raw user:pass and is base64-encoded by the CLI. A
  pre-encoded value double-encoded into a 401 that only surfaced after all 99
  planned cells had been judged and discarded. buildJudgeAuthHeader rejects it
  up front. The zero-graded guard added earlier in this PR is what made the
  failure visible instead of writing an empty "refreshed" column.

First AD rejudge: not rankable under the OLD rubric (superseded below)

Four judges were run blind over the 99-cell AD block. Three completed
(gemini 99/99, gpt 99/99, sonnet 97/99); haiku failed all 99 on
structured-output validation. That failure was later traced to the rubric
itself, not to haiku — see the measured section below, where it grades 99/99
under the per-item form. 170 cells were graded by all three surviving judges, over 23 models.

The judges agree well here — far better than on persona:

	persona block	AD block
per-cell judge spread	0.170	0.052
pairwise exact agreement	—	0.93 – 0.98

But only 47 of 231 model pairs (20.3%) separate under a paired bootstrap.
The blocker is not the judge — it is rubric saturation:

• 87.8% of all scores are exactly 1.0; 2.4% are 0.
• 16 of 23 models sit within 0.02 of the ceiling.
• Excluding the two 2.5-era models, the entire top-21 spans 0.0595.

Judges disagree by 0.052 while the models they are scoring differ by less than
that. Rejudging cannot fix this and neither can more judges — the AD rubric
does not discriminate among current frontier models, and only a harder or
finer-grained rubric would. That verdict is computed by saturation.ts, which
reports limitingFactor: 'saturation' | 'judge-noise' | 'neither' rather than
letting a reader infer it, so a future re-run cannot quietly turn a saturated
rubric into an apparent ranking.

The AD column therefore stays inventory on this board — now an accurate
38-model inventory rather than a 22-model artifact of a selection bug.

The rubric fix (folded in)

Diagnosing saturation was worth little without fixing it, so this PR now also
fixes the rubric that causes it.

createAttackDiscoveryRubricEvaluator passed all seven requirements to the
criteria judge as one string ending "answer Y or N", with a 5-of-7 threshold
buried in the prose. The judge returned a single binary score, so a discovery
missing two requirements scored identically to a flawless one. The saturation is
structural, not a property of the models.

The evidence separates the two evaluators cleanly over the same 3-judge rejudge:

evaluator	n	mean	at 1.0	sd
Rubric (collapsed Y/N)	295	0.956	95.6%	0.205
Criteria (already per-item)	295	0.963	81.4%	0.088

The shared criteria evaluator already implements weighted per-item scoring
(totalScore / maxScore); the AD rubric was discarding it. The fix passes the
seven requirements as seven criteria, so missing two now scores 5/7 instead of 0.

The 5-of-7 threshold is dropped rather than reimplemented — a threshold on
top of per-item scores would re-collapse exactly the signal this recovers.

This changes AD scores for every model. New AD runs are not comparable with
previously recorded AD golden data, and the board's current AD column is still
the old-rubric, saturated one — it is labelled as such rather than silently
re-baselined. Re-running the AD suite to produce a comparable column is follow-up
work, not part of this PR.

A regression test asserts seven distinct criteria reach the judge and is
mutation-proven: restoring the collapsed single-string rubric turns it red.

Measured: the rubric fix works, and haiku was never the problem

The AD block was rejudged on the per-item rubric, four judges, blind, same 99
cells. Every judge completed:

judge	graded	rubric loaded from suite
gemini-3.1-pro	99/99	yes
claude-4.6-sonnet	99/99	yes
claude-4.5-haiku	99/99	yes
gpt-5.4	99/99	yes

haiku's failure was the rubric, not the model. It previously failed all 99
cells on structured-output validation across 4 attempts each, and was reported
here as a model limitation. Asked the same seven requirements as seven separate
criteria, it graded all 99 without a single retry. The collapsed prompt — seven
items plus a threshold rule, answered in one shot — was what it could not
satisfy. The earlier "haiku cannot grade the AD rubric" claim was wrong and is
retracted.

The distribution changed exactly as intended:

	old (collapsed Y/N)	new (per-item)
distinct score values	2	6
scores at 1.0	95.6%	57.8%
sd	0.205 (binary)	0.139
models within 0.02 of ceiling	16 / 23	0 / 23
separable pairs (paired bootstrap)	47 / 231 (20.3%)	89 / 251 (35.5%)
borderline separations	—	0

analyzeSaturation on the new artifacts returns limitingFactor: "none" —
against "saturation" before. The AD column can now rank, where previously
no amount of rejudging could have made it.

One honest caveat: per-cell judge spread rose from 0.052 to 0.223. That is
expected — judges agreeing on a binary question is not agreement about quality —
and it is still below the 0.369 model spread, which is why the column ranks. Judge
noise is now the next constraint, where saturation used to be the binding one.

These numbers come from the 87 cells graded by all four judges; the artifacts are
/tmp/ad_rj3_local/{gemini,sonnet,haiku,gpt}.json. The board's AD column is
still the old-rubric data — re-running the suite to publish a comparable ranked
column is follow-up work, not part of this PR.

[github-actions]

One security finding: the HTML report renderer rewrites markdown links from untrusted evaluated-model answer text into <a href> without scheme validation, allowing (redacted) URI XSS in the generated matrix.html`. Details inline. The rest of the matrix pipeline (config schema, build/aggregation, query paths, CSV/markdown/JSON rendering) is well-tested and reads cleanly.

Generated by Claude Reviewer for #285833 · 162.3 AIC · ⌖ 16.6 AIC · ⊞ 4.6K

[infra-vault-gh-plugin-prod]

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

• Click to trigger kibana-pull-request for this PR!
• Click to trigger kibana-deploy-project-from-pr for this PR!
• Click to trigger kibana-deploy-cloud-from-pr for this PR!
• Click to trigger kibana-entity-store-performance-from-pr for this PR!
• Click to trigger kibana-storybooks-from-pr for this PR!

[patrykkopycinski]

buildkite test this

[patrykkopycinski]

buildkite test this

[patrykkopycinski]

buildkite test this

[patrykkopycinski]

buildkite test this

[patrykkopycinski]

buildkite test this

[patrykkopycinski]

buildkite test this

[patrykkopycinski]

buildkite test this

[patrykkopycinski]

buildkite test this

[patrykkopycinski]

buildkite test this

[patrykkopycinski]

buildkite test this