Skip to content

[kbn-evals] Fix CI LLM triage silently dropping suites: forced tool call, truncation detection, raw-output logging - #290495

Merged
arturoliduena merged 5 commits into
elastic:mainfrom
arturoliduena:kbn-evals-6094-bug-LLM-Evals-Triage-Returns-Non-JSON
Sep 12, 2026
Merged

[kbn-evals] Fix CI LLM triage silently dropping suites: forced tool call, truncation detection, raw-output logging#290495
arturoliduena merged 5 commits into
elastic:mainfrom
arturoliduena:kbn-evals-6094-bug-LLM-Evals-Triage-Returns-Non-JSON

Conversation

@arturoliduena

@arturoliduena arturoliduena commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Closes https://github.com/elastic/observability-dev/issues/6094

Summary

The per-suite LLM triage in the evals CI notify step parsed the model's free-text reply with JSON.parse. Any imperfection in that reply produced the same opaque fallback in Slack, the PR comment and the Buildkite annotation:

Triage summary could not be generated: Triage model did not return valid JSON. See the suite owner notify Buildkite step for details.

Weekly builds 72 and 73 both lost some triages in this way.

Changes

Structured output via a forced tool call

  • New report_triage tool whose parameters schema mirrors { groups: [{ error, location, models, rootCause }] }.
  • runTriageModelStructured sends it with tool_choice: { type: 'function', function: { name: 'report_triage' } }, so the model must fill tool arguments rather than answer in text.
  • buildOpenrouterChatRequest gains an options argument (tools, toolChoice, maxTokens). Defaults are unchanged, so the weekly cross-suite rollup (runTriageModel) still receives markdown bullets.

@arturoliduena arturoliduena added the release_note:skip Skip the PR/issue when compiling release notes label Sep 11, 2026
@arturoliduena
arturoliduena requested review from a team as code owners September 11, 2026 10:02
@arturoliduena arturoliduena added backport:skip This PR does not require backporting kbn-evals Issue related to the work on Kibana's LLM evaluation framework. Team:nightshift-context-and-research Nightshift Context & Research Team labels Sep 11, 2026
@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Sep 11, 2026

Copy link
Copy Markdown

Pinging @elastic/nightshift-context-and-research-team (Team:nightshift-context-and-research)

@arturoliduena
arturoliduena force-pushed the kbn-evals-6094-bug-LLM-Evals-Triage-Returns-Non-JSON branch from 2f84642 to 74fdadc Compare September 11, 2026 10:24
@arturoliduena arturoliduena added evals:all Run all @kbn/evals models:weekly-eis-models Run evals against the weekly EIS model set (see eval_pipeline.ts) labels Sep 11, 2026
viduni94
viduni94 approved these changes Sep 11, 2026
View reviewed changes

@viduni94 viduni94 left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM @arturoliduena
Thanks for fixing this.
I left a couple of comments, please take a look before merging. Thanks

Comment thread x-pack/platform/packages/shared/kbn-evals/scripts/ci/failure_context_helpers.js Outdated
console.error(
`Raw triage model reply: ${redactSecrets(JSON.stringify(message ?? responseJson)).slice(
0,
500

@viduni94 viduni94 Sep 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider 1000 here? 500 is a bit too low I feel and may lose some information.

@arturoliduena arturoliduena Sep 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread x-pack/platform/packages/shared/kbn-evals/scripts/ci/failure_context_helpers.js Outdated
return { groups: parseTriageGroups(raw), modelId };
const message = responseJson?.choices?.[0]?.message;
try {
return { groups: parseTriageGroups(message?.tool_calls?.[0]?.function?.arguments), modelId };

@viduni94 viduni94 Sep 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the model returns a text response instead of a tool call, message?.tool_calls?.[0]?.function?.arguments is undefined. parseTriageGroups(undefined) then throws "Triage model did not return valid JSON", which is accurate for the JSON parse failure but misleading about why. The issue is that there was no tool call at all, not that the arguments were malformed JSON. Distinguishing these two cases in the logged output would make CI debugging faster.

Something like:

const args = message?.tool_calls?.[0]?.function?.arguments;
if (args === undefined) {
  console.error(`Triage model ignored the tool call. Raw reply: ${redactSecrets(JSON.stringify(message ?? responseJson)).slice(0, 500)}`);
  throw new Error('Triage model did not call the report_triage tool');
}
return { groups: parseTriageGroups(args), modelId };

This also removes the need for a try/catch around the normal path, making the code cleaner.

WDYT?

@arturoliduena arturoliduena Sep 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great suggestion, applied: 7fbf136

@elastic elastic deleted a comment from kibanamachine Sep 12, 2026
@elastic elastic deleted a comment from kibanamachine Sep 12, 2026
@elastic elastic deleted a comment from kibanamachine Sep 12, 2026
@elastic elastic deleted a comment from kibanamachine Sep 12, 2026
@elastic elastic deleted a comment from kibanamachine Sep 12, 2026
@elastic elastic deleted a comment from kibanamachine Sep 12, 2026
@elastic elastic deleted a comment from kibanamachine Sep 12, 2026
@elastic elastic deleted a comment from kibanamachine Sep 12, 2026
@elastic elastic deleted a comment from kibanamachine Sep 12, 2026
@elastic elastic deleted a comment from kibanamachine Sep 12, 2026
@elastic elastic deleted a comment from kibanamachine Sep 12, 2026
@elastic elastic deleted a comment from kibanamachine Sep 12, 2026
@elastic elastic deleted a comment from kibanamachine Sep 12, 2026
@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::llm-tasks::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::llm-tasks::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
llmTasks: retrieve-documentation-task HasElasticDocsUrl 2 1.00 1.00 0.00 1.00 No -
llmTasks: retrieve-documentation-task NonEmptyDocuments 2 1.00 1.00 0.00 1.00 No -
llmTasks: retrieve-documentation-task RequiredTermsInRetrievedContent 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::llm-tasks::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::llm-tasks::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
llmTasks: retrieve-documentation-task HasElasticDocsUrl 2 1.00 1.00 0.00 1.00 No -
llmTasks: retrieve-documentation-task NonEmptyDocuments 2 1.00 1.00 0.00 1.00 No -
llmTasks: retrieve-documentation-task RequiredTermsInRetrievedContent 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::llm-tasks::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::llm-tasks::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
llmTasks: retrieve-documentation-task HasElasticDocsUrl 2 1.00 1.00 0.00 1.00 No -
llmTasks: retrieve-documentation-task NonEmptyDocuments 2 1.00 1.00 0.00 1.00 No -
llmTasks: retrieve-documentation-task RequiredTermsInRetrievedContent 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::llm-tasks::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::llm-tasks::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
llmTasks: retrieve-documentation-task HasElasticDocsUrl 2 1.00 1.00 0.00 1.00 No -
llmTasks: retrieve-documentation-task NonEmptyDocuments 2 1.00 1.00 0.00 1.00 No -
llmTasks: retrieve-documentation-task RequiredTermsInRetrievedContent 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::llm-tasks::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::llm-tasks::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
llmTasks: retrieve-documentation-task HasElasticDocsUrl 2 1.00 1.00 0.00 1.00 No -
llmTasks: retrieve-documentation-task NonEmptyDocuments 2 1.00 1.00 0.00 1.00 No -
llmTasks: retrieve-documentation-task RequiredTermsInRetrievedContent 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::llm-tasks::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::llm-tasks::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
llmTasks: retrieve-documentation-task HasElasticDocsUrl 2 1.00 1.00 0.00 1.00 No -
llmTasks: retrieve-documentation-task NonEmptyDocuments 2 1.00 1.00 0.00 1.00 No -
llmTasks: retrieve-documentation-task RequiredTermsInRetrievedContent 2 1.00 1.00 0.00 1.00 No -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::smoke-tests::anthropic-claude-4.6-sonnet | Baseline (main): 3a5213b38d56fe12::smoke-tests::anthropic-claude-4.6-sonnet
Baseline: commit 489158c, 10 days ago

Warning: Baseline is 10 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (5 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (5 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
smoke tests: es-snapshot-loader SnapshotRestored 1 1.00 1.00 0.00 - n/a -
smoke tests: llm-judge Criteria 1 1.00 1.00 0.00 - n/a -
smoke tests: score ingestion and code evaluator ContainsKibana 1 1.00 1.00 0.00 - n/a -
smoke tests: trace-retrieval Input Tokens 1 12.00 12.00 0.00 - n/a -
smoke tests: trace-retrieval Output Tokens 1 8.00 5.00 +3.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::smoke-tests::google-gemini-3.1-pro | Baseline (main): bk-019fa8d7-5840-4b94-96ff-b651ee8206f0::smoke-tests::google-gemini-3.1-pro
Baseline: commit 32d83cb, 45 days ago

Warning: Baseline is 45 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (5 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (5 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
smoke tests: es-snapshot-loader SnapshotRestored 1 1.00 1.00 0.00 - n/a -
smoke tests: llm-judge Criteria 1 1.00 1.00 0.00 - n/a -
smoke tests: score ingestion and code evaluator ContainsKibana 1 1.00 1.00 0.00 - n/a -
smoke tests: trace-retrieval Input Tokens 1 5.00 5.00 0.00 - n/a -
smoke tests: trace-retrieval Output Tokens 1 111.00 81.00 +30.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::smoke-tests::openai-gpt-5.4 | Baseline (main): bk-01a03501-86fe-4d7d-894d-3e95df37ed0d::smoke-tests::openai-gpt-5.4
Baseline: commit 9b25677, 18 days ago

Warning: Baseline is 18 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (5 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (5 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
smoke tests: es-snapshot-loader SnapshotRestored 1 1.00 1.00 0.00 - n/a -
smoke tests: llm-judge Criteria 1 1.00 1.00 0.00 - n/a -
smoke tests: score ingestion and code evaluator ContainsKibana 1 1.00 1.00 0.00 - n/a -
smoke tests: trace-retrieval Input Tokens 1 11.00 11.00 0.00 - n/a -
smoke tests: trace-retrieval Output Tokens 1 5.00 5.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::smoke-tests::openai-gpt-oss-120b | Baseline (main): bk-01a03501-86fe-4d7d-894d-3e95df37ed0d::smoke-tests::openai-gpt-oss-120b
Baseline: commit 9b25677, 18 days ago

Warning: Baseline is 18 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (5 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (5 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
smoke tests: es-snapshot-loader SnapshotRestored 1 1.00 1.00 0.00 - n/a -
smoke tests: llm-judge Criteria 1 1.00 1.00 0.00 - n/a -
smoke tests: score ingestion and code evaluator ContainsKibana 1 1.00 1.00 0.00 - n/a -
smoke tests: trace-retrieval Input Tokens 1 74.00 72.00 +2.00 - n/a -
smoke tests: trace-retrieval Output Tokens 1 76.00 35.00 +41.00 - n/a -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::agent-builder-dashboards::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::agent-builder-dashboards::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 50 comparisons.

Note: 1 unpaired (ran in only one experiment), 1 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder dashboards: grid row layout Input Tokens 2 184554.50 153298.00 +31256.50 0.01 Yes Regression
No significant changes (49 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder dashboards: dashboard section creation Cached Tokens 1 10166.00 14198.00 -4032.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard attachment exists 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard attachment title 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard grid bounds 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard panel count 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard section shape 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard skill activated 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Input Tokens 1 82996.00 97229.00 -14233.00 - n/a -
agent builder dashboards: dashboard section creation Latency 1 11.99 10.87 +1.12 - n/a -
agent builder dashboards: dashboard section creation Output Tokens 1 971.00 1492.00 -521.00 - n/a -
agent builder dashboards: dashboard section creation Tool Calls 1 2.00 3.00 -1.00 - n/a -
agent builder dashboards: dashboard section creation trajectory 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Cached Tokens 1 74879.00 86015.00 -11136.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard attachment exists 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard attachment title 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard grid bounds 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard panel count 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard skill activated 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Input Tokens 1 429033.00 456722.00 -27689.00 - n/a -
agent builder dashboards: dashboard skill activation Latency 1 23.30 25.58 -2.27 - n/a -
agent builder dashboards: dashboard skill activation Output Tokens 1 6350.00 6806.00 -456.00 - n/a -
agent builder dashboards: dashboard skill activation Tool Calls 1 4.00 4.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation trajectory 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: data exploration without dashboard Cached Tokens 1 14736.00 29644.00 -14908.00 - n/a -
agent builder dashboards: data exploration without dashboard Dashboard attachment exists 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: data exploration without dashboard Dashboard skill not activated 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: data exploration without dashboard Input Tokens 2 37527.50 29985.00 +7542.50 0.72 No -
agent builder dashboards: data exploration without dashboard Latency 2 4.18 4.41 -0.23 0.25 No -
agent builder dashboards: data exploration without dashboard Output Tokens 2 265.00 236.50 +28.50 0.81 No -
agent builder dashboards: data exploration without dashboard Tool Calls 2 3.00 2.50 +0.50 0.72 No -
agent builder dashboards: data exploration without dashboard trajectory 2 0.50 0.50 0.00 1.00 No -
agent builder dashboards: grid row layout Cached Tokens 2 45737.00 33484.00 +12253.00 0.28 No -
agent builder dashboards: grid row layout Dashboard attachment exists 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard attachment title 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard grid bounds 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard grid row layout 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard skill activated 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Latency 2 14.21 14.31 -0.10 0.71 No -
agent builder dashboards: grid row layout Output Tokens 2 2383.00 1793.00 +590.00 0.10 No -
agent builder dashboards: grid row layout Tool Calls 2 3.00 2.00 +1.00 1.00 No -
agent builder dashboards: grid row layout trajectory 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: visualization without dashboard Cached Tokens 1 10165.00 10147.00 +18.00 - n/a -
agent builder dashboards: visualization without dashboard Dashboard attachment exists 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: visualization without dashboard Input Tokens 1 53898.00 62426.00 -8528.00 - n/a -
agent builder dashboards: visualization without dashboard Latency 1 5.54 7.90 -2.36 - n/a -
agent builder dashboards: visualization without dashboard Output Tokens 1 405.00 656.00 -251.00 - n/a -
agent builder dashboards: visualization without dashboard Tool Calls 1 2.00 3.00 -1.00 - n/a -
agent builder dashboards: visualization without dashboard trajectory 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: visualization without dashboard Visualization skill activated without dashboard 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::agent-builder-dashboards::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::agent-builder-dashboards::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (48 evaluator comparisons).

Note: 1 unpaired (ran in only one experiment), 6 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (48 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder dashboards: dashboard section creation Dashboard attachment exists 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard attachment title 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard grid bounds 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard panel count 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard section shape 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard skill activated 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Input Tokens 1 99995.00 14953.00 +85042.00 - n/a -
agent builder dashboards: dashboard section creation Latency 1 7.83 1.81 +6.02 - n/a -
agent builder dashboards: dashboard section creation Output Tokens 1 1125.00 90.00 +1035.00 - n/a -
agent builder dashboards: dashboard section creation Tool Calls 1 2.00 1.00 +1.00 - n/a -
agent builder dashboards: dashboard section creation trajectory 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Cached Tokens 1 10166.00 4049.00 +6117.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard attachment exists 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard attachment title 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard grid bounds 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard panel count 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard skill activated 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Input Tokens 1 510108.00 420568.00 +89540.00 - n/a -
agent builder dashboards: dashboard skill activation Latency 1 17.41 17.17 +0.24 - n/a -
agent builder dashboards: dashboard skill activation Output Tokens 1 7520.00 5924.00 +1596.00 - n/a -
agent builder dashboards: dashboard skill activation Tool Calls 1 5.00 3.00 +2.00 - n/a -
agent builder dashboards: dashboard skill activation trajectory 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: data exploration without dashboard Cached Tokens 1 4065.00 20316.00 -16251.00 - n/a -
agent builder dashboards: data exploration without dashboard Dashboard attachment exists 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: data exploration without dashboard Dashboard skill not activated 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: data exploration without dashboard Input Tokens 2 40252.00 63878.50 -23626.50 0.39 No -
agent builder dashboards: data exploration without dashboard Latency 2 2.25 2.56 -0.31 0.37 No -
agent builder dashboards: data exploration without dashboard Output Tokens 2 270.00 367.00 -97.00 0.39 No -
agent builder dashboards: data exploration without dashboard Tool Calls 2 2.00 2.50 -0.50 0.39 No -
agent builder dashboards: data exploration without dashboard trajectory 2 0.50 0.50 0.00 1.00 No -
agent builder dashboards: grid row layout Cached Tokens 1 32466.00 8097.00 +24369.00 - n/a -
agent builder dashboards: grid row layout Dashboard attachment exists 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard attachment title 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard grid bounds 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard grid row layout 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard skill activated 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Input Tokens 2 82925.50 112935.50 -30010.00 0.75 No -
agent builder dashboards: grid row layout Latency 2 6.32 8.76 -2.44 0.48 No -
agent builder dashboards: grid row layout Output Tokens 2 909.50 1272.00 -362.50 0.65 No -
agent builder dashboards: grid row layout Tool Calls 2 2.00 1.50 +0.50 0.39 No -
agent builder dashboards: grid row layout trajectory 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: visualization without dashboard Dashboard attachment exists 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: visualization without dashboard Input Tokens 1 35742.00 34934.00 +808.00 - n/a -
agent builder dashboards: visualization without dashboard Latency 1 3.07 2.64 +0.42 - n/a -
agent builder dashboards: visualization without dashboard Output Tokens 1 320.00 301.00 +19.00 - n/a -
agent builder dashboards: visualization without dashboard Tool Calls 1 2.00 2.00 0.00 - n/a -
agent builder dashboards: visualization without dashboard trajectory 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: visualization without dashboard Visualization skill activated without dashboard 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::agent-builder-dashboards::openai-gpt-5.2 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::agent-builder-dashboards::openai-gpt-5.2
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
3 significant difference(s) detected out of 50 comparisons.

Note: 1 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder dashboards: data exploration without dashboard Input Tokens 2 11661.00 54875.00 -43214.00 0.04 Yes Improvement
agent builder dashboards: data exploration without dashboard Output Tokens 2 44.50 142.00 -97.50 0.02 Yes Improvement
agent builder dashboards: grid row layout Latency 2 10.70 7.61 +3.08 0.02 Yes Regression
No significant changes (47 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder dashboards: dashboard section creation Cached Tokens 1 21150.00 8912.00 +12238.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard attachment exists 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard attachment title 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard grid bounds 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard panel count 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard section shape 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard skill activated 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Input Tokens 1 72571.00 69371.00 +3200.00 - n/a -
agent builder dashboards: dashboard section creation Latency 1 9.91 3.94 +5.97 - n/a -
agent builder dashboards: dashboard section creation Output Tokens 1 456.00 414.00 +42.00 - n/a -
agent builder dashboards: dashboard section creation Tool Calls 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation trajectory 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Cached Tokens 1 123776.00 60617.00 +63159.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard attachment exists 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard attachment title 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard grid bounds 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard panel count 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard skill activated 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Input Tokens 1 326762.00 332043.00 -5281.00 - n/a -
agent builder dashboards: dashboard skill activation Latency 1 22.04 12.18 +9.86 - n/a -
agent builder dashboards: dashboard skill activation Output Tokens 1 6512.00 4687.00 +1825.00 - n/a -
agent builder dashboards: dashboard skill activation Tool Calls 1 3.00 2.00 +1.00 - n/a -
agent builder dashboards: dashboard skill activation trajectory 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: data exploration without dashboard Cached Tokens 1 11264.00 56960.00 -45696.00 - n/a -
agent builder dashboards: data exploration without dashboard Dashboard attachment exists 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: data exploration without dashboard Dashboard skill not activated 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: data exploration without dashboard Latency 2 3.79 3.88 -0.09 0.94 No -
agent builder dashboards: data exploration without dashboard Tool Calls 2 1.00 4.50 -3.50 0.06 No -
agent builder dashboards: data exploration without dashboard trajectory 2 0.50 0.50 0.00 1.00 No -
agent builder dashboards: grid row layout Cached Tokens 2 90095.50 47012.00 +43083.50 0.10 No -
agent builder dashboards: grid row layout Dashboard attachment exists 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard attachment title 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard grid bounds 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard grid row layout 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard skill activated 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Input Tokens 2 165350.50 103086.50 +62264.00 0.31 No -
agent builder dashboards: grid row layout Output Tokens 2 1976.00 1538.50 +437.50 0.60 No -
agent builder dashboards: grid row layout Tool Calls 2 1.50 1.50 0.00 1.00 No -
agent builder dashboards: grid row layout trajectory 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: visualization without dashboard Cached Tokens 1 491776.00 47488.00 +444288.00 - n/a -
agent builder dashboards: visualization without dashboard Dashboard attachment exists 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: visualization without dashboard Input Tokens 1 523088.00 77489.00 +445599.00 - n/a -
agent builder dashboards: visualization without dashboard Latency 1 7.17 3.66 +3.51 - n/a -
agent builder dashboards: visualization without dashboard Output Tokens 1 2541.00 337.00 +2204.00 - n/a -
agent builder dashboards: visualization without dashboard Tool Calls 1 2.00 2.00 0.00 - n/a -
agent builder dashboards: visualization without dashboard trajectory 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: visualization without dashboard Visualization skill activated without dashboard 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::agent-builder-dashboards::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::agent-builder-dashboards::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
3 significant difference(s) detected out of 50 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder dashboards: grid row layout Cached Tokens 2 14400.00 75333.00 -60933.00 0.01 Yes -
agent builder dashboards: grid row layout Input Tokens 2 25566.00 163664.00 -138098.00 0.01 Yes Improvement
agent builder dashboards: grid row layout Output Tokens 2 125.50 1538.50 -1413.00 0.01 Yes Improvement
No significant changes (47 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder dashboards: dashboard section creation Cached Tokens 1 36328.00 15011.00 +21317.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard attachment exists 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard attachment title 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard grid bounds 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard panel count 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard section shape 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Dashboard skill activated 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard section creation Input Tokens 1 87591.00 75011.00 +12580.00 - n/a -
agent builder dashboards: dashboard section creation Latency 1 5.81 4.78 +1.03 - n/a -
agent builder dashboards: dashboard section creation Output Tokens 1 969.00 541.00 +428.00 - n/a -
agent builder dashboards: dashboard section creation Tool Calls 1 3.00 2.00 +1.00 - n/a -
agent builder dashboards: dashboard section creation trajectory 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Cached Tokens 1 155459.00 87122.00 +68337.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard attachment exists 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard attachment title 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard grid bounds 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard panel count 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Dashboard skill activated 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: dashboard skill activation Input Tokens 1 463480.00 379774.00 +83706.00 - n/a -
agent builder dashboards: dashboard skill activation Latency 1 9.55 10.45 -0.90 - n/a -
agent builder dashboards: dashboard skill activation Output Tokens 1 5155.00 4429.00 +726.00 - n/a -
agent builder dashboards: dashboard skill activation Tool Calls 1 5.00 6.00 -1.00 - n/a -
agent builder dashboards: dashboard skill activation trajectory 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: data exploration without dashboard Cached Tokens 2 11264.00 11264.00 0.00 1.00 No -
agent builder dashboards: data exploration without dashboard Dashboard attachment exists 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: data exploration without dashboard Dashboard skill not activated 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: data exploration without dashboard Input Tokens 2 15026.00 11661.00 +3365.00 0.39 No -
agent builder dashboards: data exploration without dashboard Latency 2 3.49 3.76 -0.27 0.65 No -
agent builder dashboards: data exploration without dashboard Output Tokens 2 178.50 137.50 +41.00 0.38 No -
agent builder dashboards: data exploration without dashboard Tool Calls 2 0.50 0.50 0.00 1.00 No -
agent builder dashboards: data exploration without dashboard trajectory 2 0.50 0.50 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard attachment exists 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard attachment title 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard grid bounds 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard grid row layout 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Dashboard skill activated 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: grid row layout Latency 2 3.36 9.16 -5.80 0.17 No -
agent builder dashboards: grid row layout Tool Calls 2 2.00 2.00 0.00 1.00 No -
agent builder dashboards: grid row layout trajectory 2 1.00 1.00 0.00 1.00 No -
agent builder dashboards: visualization without dashboard Cached Tokens 1 162176.00 37586.00 +124590.00 - n/a -
agent builder dashboards: visualization without dashboard Dashboard attachment exists 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: visualization without dashboard Input Tokens 1 184945.00 77942.00 +107003.00 - n/a -
agent builder dashboards: visualization without dashboard Latency 1 5.90 4.09 +1.81 - n/a -
agent builder dashboards: visualization without dashboard Output Tokens 1 1163.00 491.00 +672.00 - n/a -
agent builder dashboards: visualization without dashboard Tool Calls 1 3.00 2.00 +1.00 - n/a -
agent builder dashboards: visualization without dashboard trajectory 1 1.00 1.00 0.00 - n/a -
agent builder dashboards: visualization without dashboard Visualization skill activated without dashboard 1 1.00 1.00 0.00 - n/a -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::agent-builder-visualizations::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::agent-builder-visualizations::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (12 evaluator comparisons).

Note: 4 unpaired (ran in only one experiment), 15 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (12 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder visualizations: standalone visualization creation Cached Tokens 6 20159.50 19917.00 +242.50 0.98 No -
agent builder visualizations: standalone visualization creation Chart Compatible Result 17 1.00 1.00 0.00 1.00 No -
agent builder visualizations: standalone visualization creation Chart Type vs Intent 17 1.00 1.00 0.00 1.00 No -
agent builder visualizations: standalone visualization creation ES|QL Execution Validity 17 1.00 1.00 0.00 1.00 No -
agent builder visualizations: standalone visualization creation ES|QL Functional Equivalence 17 1.00 0.97 +0.03 0.32 No -
agent builder visualizations: standalone visualization creation Input Tokens 17 51600.35 57564.29 -5963.94 0.36 No -
agent builder visualizations: standalone visualization creation Latency 17 5.68 6.72 -1.04 0.12 No -
agent builder visualizations: standalone visualization creation Output Tokens 17 422.76 503.06 -80.29 0.24 No -
agent builder visualizations: standalone visualization creation Renderer vs Intent 17 1.00 1.00 0.00 1.00 No -
agent builder visualizations: standalone visualization creation Tool Calls 17 2.12 2.24 -0.12 0.53 No -
agent builder visualizations: standalone visualization creation trajectory 17 1.00 1.00 0.00 1.00 No -
agent builder visualizations: standalone visualization creation Visualization Config Validity 17 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::agent-builder-visualizations::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::agent-builder-visualizations::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
3 significant difference(s) detected out of 12 comparisons.

Note: 3 unpaired (ran in only one experiment), 19 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder visualizations: standalone visualization creation Input Tokens 17 58604.94 47269.65 +11335.29 0.04 Yes Regression
agent builder visualizations: standalone visualization creation Latency 17 3.46 2.89 +0.57 0.00 Yes Regression
agent builder visualizations: standalone visualization creation Output Tokens 17 460.24 385.82 +74.41 0.04 Yes Regression
No significant changes (9 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder visualizations: standalone visualization creation Cached Tokens 6 13210.33 7777.50 +5432.83 0.15 No -
agent builder visualizations: standalone visualization creation Chart Compatible Result 17 1.00 1.00 0.00 1.00 No -
agent builder visualizations: standalone visualization creation Chart Type vs Intent 17 1.00 1.00 0.00 1.00 No -
agent builder visualizations: standalone visualization creation ES|QL Execution Validity 17 1.00 1.00 0.00 1.00 No -
agent builder visualizations: standalone visualization creation ES|QL Functional Equivalence 17 0.79 0.91 -0.12 0.32 No -
agent builder visualizations: standalone visualization creation Renderer vs Intent 17 1.00 1.00 0.00 1.00 No -
agent builder visualizations: standalone visualization creation Tool Calls 17 2.24 2.06 +0.18 0.18 No -
agent builder visualizations: standalone visualization creation trajectory 17 1.00 1.00 0.00 1.00 No -
agent builder visualizations: standalone visualization creation Visualization Config Validity 17 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::agent-builder-visualizations::openai-gpt-5.2 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::agent-builder-visualizations::openai-gpt-5.2
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
9 significant difference(s) detected out of 12 comparisons.

Note: 1 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder visualizations: standalone visualization creation Cached Tokens 16 470651.00 51268.25 +419382.75 0.00 Yes -
agent builder visualizations: standalone visualization creation Chart Compatible Result 17 0.00 0.94 -0.94 0.00 Yes Regression
agent builder visualizations: standalone visualization creation Chart Type vs Intent 17 0.12 0.94 -0.82 0.00 Yes Regression
agent builder visualizations: standalone visualization creation ES|QL Execution Validity 17 0.00 0.94 -0.94 0.00 Yes Regression
agent builder visualizations: standalone visualization creation ES|QL Functional Equivalence 17 0.00 0.91 -0.91 0.00 Yes Regression
agent builder visualizations: standalone visualization creation Input Tokens 17 465887.53 72078.88 +393808.65 0.00 Yes Regression
agent builder visualizations: standalone visualization creation Latency 17 7.17 5.27 +1.90 0.01 Yes Regression
agent builder visualizations: standalone visualization creation Output Tokens 17 2455.35 378.65 +2076.71 0.00 Yes Regression
agent builder visualizations: standalone visualization creation Visualization Config Validity 17 0.00 0.94 -0.94 0.00 Yes Regression
No significant changes (3 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder visualizations: standalone visualization creation Renderer vs Intent 17 0.94 1.00 -0.06 0.32 No -
agent builder visualizations: standalone visualization creation Tool Calls 17 2.00 2.24 -0.24 0.15 No -
agent builder visualizations: standalone visualization creation trajectory 17 0.88 0.94 -0.06 0.32 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::agent-builder-visualizations::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::agent-builder-visualizations::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
7 significant difference(s) detected out of 12 comparisons.

Note: 1 unpaired (ran in only one experiment), 1 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder visualizations: standalone visualization creation Cached Tokens 16 90379.13 44846.63 +45532.50 0.00 Yes -
agent builder visualizations: standalone visualization creation Chart Compatible Result 17 0.06 0.88 -0.82 0.00 Yes Regression
agent builder visualizations: standalone visualization creation Chart Type vs Intent 17 0.12 0.94 -0.82 0.00 Yes Regression
agent builder visualizations: standalone visualization creation ES|QL Execution Validity 17 0.06 0.88 -0.82 0.00 Yes Regression
agent builder visualizations: standalone visualization creation ES|QL Functional Equivalence 17 0.03 0.88 -0.85 0.00 Yes Regression
agent builder visualizations: standalone visualization creation Input Tokens 17 106193.06 61349.06 +44844.00 0.00 Yes Regression
agent builder visualizations: standalone visualization creation Visualization Config Validity 17 0.00 0.88 -0.88 0.00 Yes Regression
No significant changes (5 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder visualizations: standalone visualization creation Latency 17 5.53 6.08 -0.55 0.78 No -
agent builder visualizations: standalone visualization creation Output Tokens 17 684.06 1005.29 -321.24 0.60 No -
agent builder visualizations: standalone visualization creation Renderer vs Intent 17 0.94 0.94 0.00 1.00 No -
agent builder visualizations: standalone visualization creation Tool Calls 17 2.35 2.59 -0.24 0.64 No -
agent builder visualizations: standalone visualization creation trajectory 17 1.00 0.91 +0.09 0.18 No -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alert-analysis-workflow::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-analysis-workflow::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

Note: 16 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security: alert-analysis-workflow-classification ClassificationAccuracy 8 1.00 1.00 0.00 1.00 No -
security: alert-analysis-workflow-classification criteria 8 0.75 0.67 +0.08 0.15 No -
security: alert-analysis-workflow-classification ValidVerdict 8 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alert-analysis-workflow::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-analysis-workflow::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

Note: 16 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security: alert-analysis-workflow-classification ClassificationAccuracy 8 1.00 1.00 0.00 1.00 No -
security: alert-analysis-workflow-classification criteria 8 0.71 0.67 +0.04 0.32 No -
security: alert-analysis-workflow-classification ValidVerdict 8 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alert-analysis-workflow::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-analysis-workflow::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

Note: 16 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security: alert-analysis-workflow-classification ClassificationAccuracy 8 1.00 1.00 0.00 1.00 No -
security: alert-analysis-workflow-classification criteria 8 0.75 0.71 +0.04 0.57 No -
security: alert-analysis-workflow-classification ValidVerdict 8 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alert-analysis-workflow::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-analysis-workflow::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

Note: 16 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security: alert-analysis-workflow-classification ClassificationAccuracy 8 1.00 1.00 0.00 1.00 No -
security: alert-analysis-workflow-classification criteria 8 0.67 0.63 +0.04 0.32 No -
security: alert-analysis-workflow-classification ValidVerdict 8 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alert-analysis-workflow::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-analysis-workflow::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

Note: 16 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security: alert-analysis-workflow-classification ClassificationAccuracy 8 1.00 1.00 0.00 1.00 No -
security: alert-analysis-workflow-classification criteria 8 0.75 0.79 -0.04 0.57 No -
security: alert-analysis-workflow-classification ValidVerdict 8 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alert-analysis-workflow::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-analysis-workflow::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

Note: 16 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security: alert-analysis-workflow-classification ClassificationAccuracy 8 1.00 1.00 0.00 1.00 No -
security: alert-analysis-workflow-classification criteria 8 0.58 0.67 -0.08 0.15 No -
security: alert-analysis-workflow-classification ValidVerdict 8 1.00 1.00 0.00 1.00 No -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::attack-discovery::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::attack-discovery::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (7 evaluator comparisons).

Note: 6 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (7 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Attack Discovery All Scenarios AttackDiscoveryBasic 1 0.00 0.00 0.00 - n/a -
Attack Discovery All Scenarios AttackDiscoveryRubric 1 0.00 0.00 0.00 - n/a -
Attack Discovery All Scenarios Criteria 1 0.14 0.14 0.00 - n/a -
Attack Discovery All Scenarios Latency 1 0.03 0.03 +0.01 - n/a -
Attack Discovery All Scenarios Tool Calls 1 0.00 0.00 0.00 - n/a -
attack discovery: graphState smoke Ran 1 1.00 1.00 0.00 - n/a -
attack discovery: searchAlerts smoke Ran 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::attack-discovery::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::attack-discovery::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (7 evaluator comparisons).

Note: 6 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (7 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Attack Discovery All Scenarios AttackDiscoveryBasic 1 0.00 0.00 0.00 - n/a -
Attack Discovery All Scenarios AttackDiscoveryRubric 1 0.00 0.00 0.00 - n/a -
Attack Discovery All Scenarios Criteria 1 0.14 0.43 -0.29 - n/a -
Attack Discovery All Scenarios Latency 1 0.03 0.03 -0.01 - n/a -
Attack Discovery All Scenarios Tool Calls 1 0.00 0.00 0.00 - n/a -
attack discovery: graphState smoke Ran 1 1.00 1.00 0.00 - n/a -
attack discovery: searchAlerts smoke Ran 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::attack-discovery::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::attack-discovery::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (9 evaluator comparisons).

Note: 2 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (9 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Attack Discovery All Scenarios AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
Attack Discovery All Scenarios AttackDiscoveryRubric 1 0.00 0.00 0.00 - n/a -
Attack Discovery All Scenarios Criteria 1 1.00 1.00 0.00 - n/a -
Attack Discovery All Scenarios Input Tokens 1 34511.00 34511.00 0.00 - n/a -
Attack Discovery All Scenarios Latency 1 0.03 0.02 +0.01 - n/a -
Attack Discovery All Scenarios Output Tokens 1 9162.00 3327.00 +5835.00 - n/a -
Attack Discovery All Scenarios Tool Calls 1 0.00 0.00 0.00 - n/a -
attack discovery: graphState smoke Ran 1 1.00 1.00 0.00 - n/a -
attack discovery: searchAlerts smoke Ran 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::attack-discovery::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::attack-discovery::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (7 evaluator comparisons).

Note: 6 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (7 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Attack Discovery All Scenarios AttackDiscoveryBasic 1 0.00 0.00 0.00 - n/a -
Attack Discovery All Scenarios AttackDiscoveryRubric 1 0.00 0.00 0.00 - n/a -
Attack Discovery All Scenarios Criteria 1 0.14 0.43 -0.29 - n/a -
Attack Discovery All Scenarios Latency 1 0.03 0.03 +0.00 - n/a -
Attack Discovery All Scenarios Tool Calls 1 0.00 0.00 0.00 - n/a -
attack discovery: graphState smoke Ran 1 1.00 1.00 0.00 - n/a -
attack discovery: searchAlerts smoke Ran 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::attack-discovery::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::attack-discovery::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (9 evaluator comparisons).

Note: 2 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (9 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Attack Discovery All Scenarios AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
Attack Discovery All Scenarios AttackDiscoveryRubric 1 0.00 0.00 0.00 - n/a -
Attack Discovery All Scenarios Criteria 1 1.00 1.00 0.00 - n/a -
Attack Discovery All Scenarios Input Tokens 1 26105.00 26105.00 0.00 - n/a -
Attack Discovery All Scenarios Latency 1 0.04 0.03 +0.01 - n/a -
Attack Discovery All Scenarios Output Tokens 1 6100.00 5108.00 +992.00 - n/a -
Attack Discovery All Scenarios Tool Calls 1 0.00 0.00 0.00 - n/a -
attack discovery: graphState smoke Ran 1 1.00 1.00 0.00 - n/a -
attack discovery: searchAlerts smoke Ran 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::attack-discovery::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::attack-discovery::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (9 evaluator comparisons).

Note: 1 unpaired (ran in only one experiment), 1 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (9 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Attack Discovery All Scenarios AttackDiscoveryBasic 1 0.00 1.00 -1.00 - n/a -
Attack Discovery All Scenarios AttackDiscoveryRubric 1 0.00 0.00 0.00 - n/a -
Attack Discovery All Scenarios Criteria 1 0.14 0.86 -0.71 - n/a -
Attack Discovery All Scenarios Input Tokens 1 26192.00 26178.00 +14.00 - n/a -
Attack Discovery All Scenarios Latency 1 1.94 0.02 +1.92 - n/a -
Attack Discovery All Scenarios Output Tokens 1 39.00 5928.00 -5889.00 - n/a -
Attack Discovery All Scenarios Tool Calls 1 0.00 0.00 0.00 - n/a -
attack discovery: graphState smoke Ran 1 1.00 1.00 0.00 - n/a -
attack discovery: searchAlerts smoke Ran 1 0.00 1.00 -1.00 - n/a -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::streams::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::streams::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (141 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (141 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Content-Based - no existing llm_partition_quality 1 0.67 0.67 0.00 - n/a -
Content-Based - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Content-Based - no existing partition_coverage 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app pattern_quality_score 1 0.67 0.67 0.00 - n/a -
Dissect - logs.csv llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.csv parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.csv pattern_quality_score 1 0.75 0.75 0.00 - n/a -
Dissect - logs.db_pool llm_extraction_quality 1 0.60 0.80 -0.20 - n/a -
Dissect - logs.db_pool parse_rate 1 0.90 1.00 -0.10 - n/a -
Dissect - logs.db_pool pattern_quality_score 1 0.82 0.93 -0.10 - n/a -
Dissect - logs.db_query llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.db_query parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.db_query pattern_quality_score 1 0.87 0.87 0.00 - n/a -
Dissect - logs.firewall llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.firewall parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.firewall pattern_quality_score 1 0.93 0.93 0.00 - n/a -
Dissect - logs.haproxy llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.haproxy parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.haproxy pattern_quality_score 1 0.94 0.94 0.00 - n/a -
Dissect - logs.health_check llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Dissect - logs.health_check parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.health_check pattern_quality_score 1 0.83 0.88 -0.05 - n/a -
Dissect - logs.keyvalue llm_extraction_quality 1 0.40 0.60 -0.20 - n/a -
Dissect - logs.keyvalue parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.keyvalue pattern_quality_score 1 0.78 0.78 0.00 - n/a -
Dissect - logs.pipeline llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.pipeline parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.pipeline pattern_quality_score 1 0.76 0.76 0.00 - n/a -
Dissect - logs.spring_boot llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.spring_boot parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.spring_boot pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Dissect - logs.syslog_bsd llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_bsd parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_bsd pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Dissect - logs.syslog_rfc5424 llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_rfc5424 parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_rfc5424 pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Dissect - logs.system_metrics llm_extraction_quality 1 0.40 0.40 0.00 - n/a -
Dissect - logs.system_metrics parse_rate 1 0.00 0.00 0.00 - n/a -
Dissect - logs.system_metrics pattern_quality_score 1 0.15 0.15 0.00 - n/a -
Dissect - logs.tsv llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.tsv parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.tsv pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Dissect - logs.vpn llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.vpn parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.vpn pattern_quality_score 1 0.95 0.95 0.00 - n/a -
Dissect - logs.web_access llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.web_access parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.web_access pattern_quality_score 1 0.93 0.93 0.00 - n/a -
Grok - logs.apache llm_extraction_quality 1 0.20 0.20 0.00 - n/a -
Grok - logs.apache parse_rate 1 0.70 0.70 0.00 - n/a -
Grok - logs.apache pattern_quality_score 1 0.75 0.78 -0.03 - n/a -
Grok - logs.auth llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.auth parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.auth pattern_quality_score 1 0.94 0.94 0.00 - n/a -
Grok - logs.docker llm_extraction_quality 1 0.40 0.40 0.00 - n/a -
Grok - logs.docker parse_rate 1 0.00 0.00 0.00 - n/a -
Grok - logs.docker pattern_quality_score 1 0.00 0.00 0.00 - n/a -
Grok - logs.hdfs llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.hdfs parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.hdfs pattern_quality_score 1 0.80 0.80 0.00 - n/a -
Grok - logs.kafka llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Grok - logs.kafka parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.kafka pattern_quality_score 1 0.72 0.72 0.00 - n/a -
Grok - logs.mongodb llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.mongodb parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.mongodb pattern_quality_score 1 0.69 0.69 0.00 - n/a -
Grok - logs.mysql_slowlog llm_extraction_quality 1 0.40 0.40 0.00 - n/a -
Grok - logs.mysql_slowlog parse_rate 1 0.60 0.60 0.00 - n/a -
Grok - logs.mysql_slowlog pattern_quality_score 1 0.49 0.49 -0.00 - n/a -
Grok - logs.nginx llm_extraction_quality 1 0.20 0.20 0.00 - n/a -
Grok - logs.nginx parse_rate 1 0.22 0.22 0.00 - n/a -
Grok - logs.nginx pattern_quality_score 1 0.65 0.65 0.00 - n/a -
Grok - logs.nginx_error llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Grok - logs.nginx_error parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.nginx_error pattern_quality_score 1 0.75 0.75 +0.01 - n/a -
Grok - logs.openssh llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.openssh parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.openssh pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Grok - logs.postgresql llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.postgresql parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.postgresql pattern_quality_score 1 0.82 0.82 0.00 - n/a -
Grok - logs.rabbitmq llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.rabbitmq parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.rabbitmq pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Grok - logs.redis llm_extraction_quality 1 0.40 0.40 0.00 - n/a -
Grok - logs.redis parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.redis pattern_quality_score 1 0.74 0.74 0.00 - n/a -
Grok - logs.spark llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.spark parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.spark pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Grok - logs.syslog llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Grok - logs.syslog parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.syslog pattern_quality_score 1 0.94 0.94 0.00 - n/a -
Grok - logs.tomcat llm_extraction_quality 1 0.20 0.20 0.00 - n/a -
Grok - logs.tomcat parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.tomcat pattern_quality_score 1 0.60 0.60 0.00 - n/a -
Grok - logs.zookeeper llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.zookeeper parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.zookeeper pattern_quality_score 1 0.72 0.75 -0.04 - n/a -
Homogeneous - no existing llm_partition_quality 1 0.33 0.33 0.00 - n/a -
Homogeneous - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Homogeneous - no existing partition_coverage 1 0.95 0.92 +0.03 - n/a -
Multi-System - no existing llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Multi-System - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Multi-System - no existing partition_coverage 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" overlap_score 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" partition_coverage 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" overlap_score 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" partition_coverage 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Apache llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Apache pipeline_quality_score 1 0.98 0.98 0.00 - n/a -
Pipeline Suggestion - HDFS llm_pipeline_quality 1 1.00 0.80 +0.20 - n/a -
Pipeline Suggestion - HDFS pipeline_quality_score 1 0.95 0.95 0.00 - n/a -
Pipeline Suggestion - OpenSSH llm_pipeline_quality 1 1.00 0.80 +0.20 - n/a -
Pipeline Suggestion - OpenSSH pipeline_quality_score 1 0.88 1.00 -0.12 - n/a -
Pipeline Suggestion - Spark llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Spark pipeline_quality_score 1 0.97 0.97 0.00 - n/a -
Pipeline Suggestion - structured llm_pipeline_quality 1 0.00 0.00 0.00 - n/a -
Pipeline Suggestion - structured pipeline_quality_score 1 0.00 0.00 0.00 - n/a -
Pipeline Suggestion - Zookeeper llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Zookeeper pipeline_quality_score 1 0.95 0.95 0.00 - n/a -
Refinement - 1 existing llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Refinement - 1 existing overlap_score 1 1.00 1.00 0.00 - n/a -
Refinement - 1 existing partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" llm_partition_quality 1 0.67 0.67 0.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" partition_coverage 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::streams::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::streams::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (141 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (141 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Content-Based - no existing llm_partition_quality 1 0.67 0.67 0.00 - n/a -
Content-Based - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Content-Based - no existing partition_coverage 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app pattern_quality_score 1 0.67 0.67 0.00 - n/a -
Dissect - logs.csv llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.csv parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.csv pattern_quality_score 1 0.80 0.80 0.00 - n/a -
Dissect - logs.db_pool llm_extraction_quality 1 0.80 0.60 +0.20 - n/a -
Dissect - logs.db_pool parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.db_pool pattern_quality_score 1 0.89 0.82 +0.06 - n/a -
Dissect - logs.db_query llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.db_query parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.db_query pattern_quality_score 1 0.87 0.87 0.00 - n/a -
Dissect - logs.firewall llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.firewall parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.firewall pattern_quality_score 1 0.95 0.95 0.00 - n/a -
Dissect - logs.haproxy llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.haproxy parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.haproxy pattern_quality_score 1 0.91 0.91 0.00 - n/a -
Dissect - logs.health_check llm_extraction_quality 1 0.40 0.60 -0.20 - n/a -
Dissect - logs.health_check parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.health_check pattern_quality_score 1 0.79 0.80 -0.01 - n/a -
Dissect - logs.keyvalue llm_extraction_quality 1 0.80 1.00 -0.20 - n/a -
Dissect - logs.keyvalue parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.keyvalue pattern_quality_score 1 0.80 0.83 -0.03 - n/a -
Dissect - logs.pipeline llm_extraction_quality 1 0.80 1.00 -0.20 - n/a -
Dissect - logs.pipeline parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.pipeline pattern_quality_score 1 0.76 0.76 0.00 - n/a -
Dissect - logs.spring_boot llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.spring_boot parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.spring_boot pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Dissect - logs.syslog_bsd llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_bsd parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_bsd pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Dissect - logs.syslog_rfc5424 llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_rfc5424 parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_rfc5424 pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Dissect - logs.system_metrics llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Dissect - logs.system_metrics parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.system_metrics pattern_quality_score 1 0.75 0.75 0.00 - n/a -
Dissect - logs.tsv llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.tsv parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.tsv pattern_quality_score 1 0.76 0.76 0.00 - n/a -
Dissect - logs.vpn llm_extraction_quality 1 0.80 1.00 -0.20 - n/a -
Dissect - logs.vpn parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.vpn pattern_quality_score 1 1.00 1.00 0.00 - n/a -
Dissect - logs.web_access llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.web_access parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.web_access pattern_quality_score 1 0.93 0.93 0.00 - n/a -
Grok - logs.apache llm_extraction_quality 1 0.20 0.20 0.00 - n/a -
Grok - logs.apache parse_rate 1 0.70 0.70 0.00 - n/a -
Grok - logs.apache pattern_quality_score 1 0.74 0.74 0.00 - n/a -
Grok - logs.auth llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.auth parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.auth pattern_quality_score 1 0.94 0.94 0.00 - n/a -
Grok - logs.docker llm_extraction_quality 1 0.40 0.40 0.00 - n/a -
Grok - logs.docker parse_rate 1 0.00 0.00 0.00 - n/a -
Grok - logs.docker pattern_quality_score 1 0.00 0.00 0.00 - n/a -
Grok - logs.hdfs llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.hdfs parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.hdfs pattern_quality_score 1 0.80 0.80 0.00 - n/a -
Grok - logs.kafka llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Grok - logs.kafka parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.kafka pattern_quality_score 1 0.72 0.72 0.00 - n/a -
Grok - logs.mongodb llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.mongodb parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.mongodb pattern_quality_score 1 0.69 0.69 0.00 - n/a -
Grok - logs.mysql_slowlog llm_extraction_quality 1 0.40 0.20 +0.20 - n/a -
Grok - logs.mysql_slowlog parse_rate 1 0.60 0.60 0.00 - n/a -
Grok - logs.mysql_slowlog pattern_quality_score 1 0.50 0.49 +0.01 - n/a -
Grok - logs.nginx llm_extraction_quality 1 0.40 0.40 0.00 - n/a -
Grok - logs.nginx parse_rate 1 0.22 0.22 0.00 - n/a -
Grok - logs.nginx pattern_quality_score 1 0.65 0.64 +0.00 - n/a -
Grok - logs.nginx_error llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Grok - logs.nginx_error parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.nginx_error pattern_quality_score 1 0.67 0.75 -0.08 - n/a -
Grok - logs.openssh llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.openssh parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.openssh pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Grok - logs.postgresql llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.postgresql parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.postgresql pattern_quality_score 1 0.82 0.82 0.00 - n/a -
Grok - logs.rabbitmq llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.rabbitmq parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.rabbitmq pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Grok - logs.redis llm_extraction_quality 1 0.40 0.80 -0.40 - n/a -
Grok - logs.redis parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.redis pattern_quality_score 1 0.59 0.55 +0.04 - n/a -
Grok - logs.spark llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.spark parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.spark pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Grok - logs.syslog llm_extraction_quality 1 0.40 0.80 -0.40 - n/a -
Grok - logs.syslog parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.syslog pattern_quality_score 1 0.94 0.94 0.00 - n/a -
Grok - logs.tomcat llm_extraction_quality 1 0.20 0.20 0.00 - n/a -
Grok - logs.tomcat parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.tomcat pattern_quality_score 1 0.60 0.60 0.00 - n/a -
Grok - logs.zookeeper llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.zookeeper parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.zookeeper pattern_quality_score 1 0.72 0.72 0.00 - n/a -
Homogeneous - no existing llm_partition_quality 1 0.33 0.33 0.00 - n/a -
Homogeneous - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Homogeneous - no existing partition_coverage 1 0.94 0.90 +0.03 - n/a -
Multi-System - no existing llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Multi-System - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Multi-System - no existing partition_coverage 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" overlap_score 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" partition_coverage 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" overlap_score 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" partition_coverage 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Apache llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Apache pipeline_quality_score 1 0.98 0.98 0.00 - n/a -
Pipeline Suggestion - HDFS llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - HDFS pipeline_quality_score 1 0.95 0.95 0.00 - n/a -
Pipeline Suggestion - OpenSSH llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - OpenSSH pipeline_quality_score 1 0.88 1.00 -0.12 - n/a -
Pipeline Suggestion - Spark llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Spark pipeline_quality_score 1 0.97 0.97 0.00 - n/a -
Pipeline Suggestion - structured llm_pipeline_quality 1 0.00 0.00 0.00 - n/a -
Pipeline Suggestion - structured pipeline_quality_score 1 0.00 0.00 0.00 - n/a -
Pipeline Suggestion - Zookeeper llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Zookeeper pipeline_quality_score 1 0.95 0.95 0.00 - n/a -
Refinement - 1 existing llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Refinement - 1 existing overlap_score 1 1.00 1.00 0.00 - n/a -
Refinement - 1 existing partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" llm_partition_quality 1 0.33 0.33 0.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" overlap_score 1 0.00 0.00 0.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" partition_coverage 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::streams::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::streams::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (141 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (141 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Content-Based - no existing llm_partition_quality 1 0.67 0.67 0.00 - n/a -
Content-Based - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Content-Based - no existing partition_coverage 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app pattern_quality_score 1 0.67 0.67 0.00 - n/a -
Dissect - logs.csv llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.csv parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.csv pattern_quality_score 1 0.80 0.75 +0.05 - n/a -
Dissect - logs.db_pool llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Dissect - logs.db_pool parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.db_pool pattern_quality_score 1 0.84 0.84 0.00 - n/a -
Dissect - logs.db_query llm_extraction_quality 1 0.40 1.00 -0.60 - n/a -
Dissect - logs.db_query parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.db_query pattern_quality_score 1 0.77 0.82 -0.05 - n/a -
Dissect - logs.firewall llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Dissect - logs.firewall parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.firewall pattern_quality_score 1 0.88 0.88 +0.01 - n/a -
Dissect - logs.haproxy llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.haproxy parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.haproxy pattern_quality_score 1 0.91 0.91 0.00 - n/a -
Dissect - logs.health_check llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Dissect - logs.health_check parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.health_check pattern_quality_score 1 0.73 0.73 0.00 - n/a -
Dissect - logs.keyvalue llm_extraction_quality 1 1.00 0.60 +0.40 - n/a -
Dissect - logs.keyvalue parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.keyvalue pattern_quality_score 1 0.67 0.67 0.00 - n/a -
Dissect - logs.pipeline llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.pipeline parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.pipeline pattern_quality_score 1 0.76 0.76 0.00 - n/a -
Dissect - logs.spring_boot llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.spring_boot parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.spring_boot pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Dissect - logs.syslog_bsd llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_bsd parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_bsd pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Dissect - logs.syslog_rfc5424 llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_rfc5424 parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_rfc5424 pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Dissect - logs.system_metrics llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Dissect - logs.system_metrics parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.system_metrics pattern_quality_score 1 0.75 0.75 0.00 - n/a -
Dissect - logs.tsv llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.tsv parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.tsv pattern_quality_score 1 0.76 0.76 0.00 - n/a -
Dissect - logs.vpn llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.vpn parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.vpn pattern_quality_score 1 0.98 0.98 0.00 - n/a -
Dissect - logs.web_access llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.web_access parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.web_access pattern_quality_score 1 0.93 0.93 0.00 - n/a -
Grok - logs.apache llm_extraction_quality 1 0.20 0.20 0.00 - n/a -
Grok - logs.apache parse_rate 1 0.70 0.70 0.00 - n/a -
Grok - logs.apache pattern_quality_score 1 0.78 0.78 0.00 - n/a -
Grok - logs.auth llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.auth parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.auth pattern_quality_score 1 0.94 0.94 0.00 - n/a -
Grok - logs.docker llm_extraction_quality 1 0.40 0.40 0.00 - n/a -
Grok - logs.docker parse_rate 1 0.00 0.00 0.00 - n/a -
Grok - logs.docker pattern_quality_score 1 0.00 0.00 0.00 - n/a -
Grok - logs.hdfs llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.hdfs parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.hdfs pattern_quality_score 1 0.80 0.80 0.00 - n/a -
Grok - logs.kafka llm_extraction_quality 1 0.60 0.80 -0.20 - n/a -
Grok - logs.kafka parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.kafka pattern_quality_score 1 0.72 0.78 -0.05 - n/a -
Grok - logs.mongodb llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.mongodb parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.mongodb pattern_quality_score 1 0.69 0.69 0.00 - n/a -
Grok - logs.mysql_slowlog llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Grok - logs.mysql_slowlog parse_rate 1 0.60 0.60 0.00 - n/a -
Grok - logs.mysql_slowlog pattern_quality_score 1 0.52 0.52 +0.00 - n/a -
Grok - logs.nginx llm_extraction_quality 1 0.00 0.40 -0.40 - n/a -
Grok - logs.nginx parse_rate 1 0.22 0.22 0.00 - n/a -
Grok - logs.nginx pattern_quality_score 1 0.70 0.65 +0.05 - n/a -
Grok - logs.nginx_error llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Grok - logs.nginx_error parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.nginx_error pattern_quality_score 1 0.77 0.77 0.00 - n/a -
Grok - logs.openssh llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.openssh parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.openssh pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Grok - logs.postgresql llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.postgresql parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.postgresql pattern_quality_score 1 0.82 0.82 0.00 - n/a -
Grok - logs.rabbitmq llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.rabbitmq parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.rabbitmq pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Grok - logs.redis llm_extraction_quality 1 0.40 0.40 0.00 - n/a -
Grok - logs.redis parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.redis pattern_quality_score 1 0.63 0.48 +0.15 - n/a -
Grok - logs.spark llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.spark parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.spark pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Grok - logs.syslog llm_extraction_quality 1 0.60 0.80 -0.20 - n/a -
Grok - logs.syslog parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.syslog pattern_quality_score 1 0.94 0.94 0.00 - n/a -
Grok - logs.tomcat llm_extraction_quality 1 0.20 0.20 0.00 - n/a -
Grok - logs.tomcat parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.tomcat pattern_quality_score 1 0.76 0.76 0.00 - n/a -
Grok - logs.zookeeper llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.zookeeper parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.zookeeper pattern_quality_score 1 0.75 0.75 0.00 - n/a -
Homogeneous - no existing llm_partition_quality 1 0.33 0.33 0.00 - n/a -
Homogeneous - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Homogeneous - no existing partition_coverage 1 0.90 0.96 -0.06 - n/a -
Multi-System - no existing llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Multi-System - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Multi-System - no existing partition_coverage 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" overlap_score 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" partition_coverage 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" overlap_score 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" partition_coverage 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Apache llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Apache pipeline_quality_score 1 0.98 0.98 0.00 - n/a -
Pipeline Suggestion - HDFS llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - HDFS pipeline_quality_score 1 0.95 0.95 0.00 - n/a -
Pipeline Suggestion - OpenSSH llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - OpenSSH pipeline_quality_score 1 0.88 0.88 0.00 - n/a -
Pipeline Suggestion - Spark llm_pipeline_quality 1 1.00 0.80 +0.20 - n/a -
Pipeline Suggestion - Spark pipeline_quality_score 1 0.97 0.97 0.00 - n/a -
Pipeline Suggestion - structured llm_pipeline_quality 1 0.00 0.00 0.00 - n/a -
Pipeline Suggestion - structured pipeline_quality_score 1 0.00 0.00 0.00 - n/a -
Pipeline Suggestion - Zookeeper llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Zookeeper pipeline_quality_score 1 0.95 0.95 0.00 - n/a -
Refinement - 1 existing llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Refinement - 1 existing overlap_score 1 1.00 1.00 0.00 - n/a -
Refinement - 1 existing partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" llm_partition_quality 1 0.67 0.33 +0.33 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" overlap_score 1 1.00 0.00 +1.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" partition_coverage 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::streams::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::streams::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (141 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (141 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Content-Based - no existing llm_partition_quality 1 0.67 0.67 0.00 - n/a -
Content-Based - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Content-Based - no existing partition_coverage 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app pattern_quality_score 1 0.67 0.67 0.00 - n/a -
Dissect - logs.csv llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.csv parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.csv pattern_quality_score 1 0.75 0.80 -0.05 - n/a -
Dissect - logs.db_pool llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Dissect - logs.db_pool parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.db_pool pattern_quality_score 1 0.84 0.84 0.00 - n/a -
Dissect - logs.db_query llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.db_query parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.db_query pattern_quality_score 1 0.82 0.82 0.00 - n/a -
Dissect - logs.firewall llm_extraction_quality 1 0.60 1.00 -0.40 - n/a -
Dissect - logs.firewall parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.firewall pattern_quality_score 1 0.75 0.95 -0.20 - n/a -
Dissect - logs.haproxy llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.haproxy parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.haproxy pattern_quality_score 1 0.91 0.91 0.00 - n/a -
Dissect - logs.health_check llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Dissect - logs.health_check parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.health_check pattern_quality_score 1 0.80 0.73 +0.07 - n/a -
Dissect - logs.keyvalue llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.keyvalue parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.keyvalue pattern_quality_score 1 0.83 0.67 +0.15 - n/a -
Dissect - logs.pipeline llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.pipeline parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.pipeline pattern_quality_score 1 0.75 0.75 0.00 - n/a -
Dissect - logs.spring_boot llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.spring_boot parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.spring_boot pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Dissect - logs.syslog_bsd llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_bsd parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_bsd pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Dissect - logs.syslog_rfc5424 llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_rfc5424 parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_rfc5424 pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Dissect - logs.system_metrics llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Dissect - logs.system_metrics parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.system_metrics pattern_quality_score 1 0.75 0.75 0.00 - n/a -
Dissect - logs.tsv llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.tsv parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.tsv pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Dissect - logs.vpn llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Dissect - logs.vpn parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.vpn pattern_quality_score 1 0.85 0.85 0.00 - n/a -
Dissect - logs.web_access llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.web_access parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.web_access pattern_quality_score 1 0.93 0.93 0.00 - n/a -
Grok - logs.apache llm_extraction_quality 1 0.20 0.20 0.00 - n/a -
Grok - logs.apache parse_rate 1 0.70 0.70 0.00 - n/a -
Grok - logs.apache pattern_quality_score 1 0.80 0.80 0.00 - n/a -
Grok - logs.auth llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.auth parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.auth pattern_quality_score 1 0.94 0.94 0.00 - n/a -
Grok - logs.docker llm_extraction_quality 1 0.40 0.60 -0.20 - n/a -
Grok - logs.docker parse_rate 1 0.00 0.00 0.00 - n/a -
Grok - logs.docker pattern_quality_score 1 0.00 0.00 0.00 - n/a -
Grok - logs.hdfs llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.hdfs parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.hdfs pattern_quality_score 1 0.80 0.80 0.00 - n/a -
Grok - logs.kafka llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Grok - logs.kafka parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.kafka pattern_quality_score 1 0.65 0.78 -0.13 - n/a -
Grok - logs.mongodb llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.mongodb parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.mongodb pattern_quality_score 1 0.69 0.69 0.00 - n/a -
Grok - logs.mysql_slowlog llm_extraction_quality 1 0.40 0.20 +0.20 - n/a -
Grok - logs.mysql_slowlog parse_rate 1 0.60 0.60 0.00 - n/a -
Grok - logs.mysql_slowlog pattern_quality_score 1 0.50 0.50 0.00 - n/a -
Grok - logs.nginx llm_extraction_quality 1 0.20 0.20 0.00 - n/a -
Grok - logs.nginx parse_rate 1 0.22 0.22 0.00 - n/a -
Grok - logs.nginx pattern_quality_score 1 0.74 0.69 +0.05 - n/a -
Grok - logs.nginx_error llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Grok - logs.nginx_error parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.nginx_error pattern_quality_score 1 0.74 0.74 0.00 - n/a -
Grok - logs.openssh llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.openssh parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.openssh pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Grok - logs.postgresql llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.postgresql parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.postgresql pattern_quality_score 1 0.82 0.82 0.00 - n/a -
Grok - logs.rabbitmq llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.rabbitmq parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.rabbitmq pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Grok - logs.redis llm_extraction_quality 1 0.40 0.40 0.00 - n/a -
Grok - logs.redis parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.redis pattern_quality_score 1 0.59 0.59 0.00 - n/a -
Grok - logs.spark llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.spark parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.spark pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Grok - logs.syslog llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Grok - logs.syslog parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.syslog pattern_quality_score 1 0.94 0.89 +0.05 - n/a -
Grok - logs.tomcat llm_extraction_quality 1 0.20 0.20 0.00 - n/a -
Grok - logs.tomcat parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.tomcat pattern_quality_score 1 0.44 0.44 0.00 - n/a -
Grok - logs.zookeeper llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.zookeeper parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.zookeeper pattern_quality_score 1 0.75 0.75 0.00 - n/a -
Homogeneous - no existing llm_partition_quality 1 0.50 0.50 0.00 - n/a -
Homogeneous - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Homogeneous - no existing partition_coverage 1 1.00 1.00 0.00 - n/a -
Multi-System - no existing llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Multi-System - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Multi-System - no existing partition_coverage 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" overlap_score 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" partition_coverage 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" overlap_score 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" partition_coverage 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Apache llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Apache pipeline_quality_score 1 0.98 0.98 0.00 - n/a -
Pipeline Suggestion - HDFS llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - HDFS pipeline_quality_score 1 0.95 0.95 0.00 - n/a -
Pipeline Suggestion - OpenSSH llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - OpenSSH pipeline_quality_score 1 0.88 0.88 0.00 - n/a -
Pipeline Suggestion - Spark llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Spark pipeline_quality_score 1 0.97 0.97 0.00 - n/a -
Pipeline Suggestion - structured llm_pipeline_quality 1 0.00 0.00 0.00 - n/a -
Pipeline Suggestion - structured pipeline_quality_score 1 0.00 0.00 0.00 - n/a -
Pipeline Suggestion - Zookeeper llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Zookeeper pipeline_quality_score 1 0.95 0.95 0.00 - n/a -
Refinement - 1 existing llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Refinement - 1 existing overlap_score 1 1.00 1.00 0.00 - n/a -
Refinement - 1 existing partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" llm_partition_quality 1 0.83 0.83 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" partition_coverage 1 1.00 1.00 0.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" llm_partition_quality 1 0.67 0.67 0.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" partition_coverage 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::streams::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::streams::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (141 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (141 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Content-Based - no existing llm_partition_quality 1 0.50 0.50 0.00 - n/a -
Content-Based - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Content-Based - no existing partition_coverage 1 0.00 0.00 0.00 - n/a -
Dissect - logs.app llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app pattern_quality_score 1 0.67 0.67 0.00 - n/a -
Dissect - logs.csv llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.csv parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.csv pattern_quality_score 1 0.75 0.75 0.00 - n/a -
Dissect - logs.db_pool llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Dissect - logs.db_pool parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.db_pool pattern_quality_score 1 0.93 0.93 0.00 - n/a -
Dissect - logs.db_query llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.db_query parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.db_query pattern_quality_score 1 0.82 0.82 0.00 - n/a -
Dissect - logs.firewall llm_extraction_quality 1 0.80 1.00 -0.20 - n/a -
Dissect - logs.firewall parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.firewall pattern_quality_score 1 0.95 0.95 0.00 - n/a -
Dissect - logs.haproxy llm_extraction_quality 1 0.80 1.00 -0.20 - n/a -
Dissect - logs.haproxy parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.haproxy pattern_quality_score 1 0.91 0.91 0.00 - n/a -
Dissect - logs.health_check llm_extraction_quality 1 0.40 0.40 0.00 - n/a -
Dissect - logs.health_check parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.health_check pattern_quality_score 1 0.86 0.79 +0.07 - n/a -
Dissect - logs.keyvalue llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.keyvalue parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.keyvalue pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Dissect - logs.pipeline llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.pipeline parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.pipeline pattern_quality_score 1 0.75 0.75 0.00 - n/a -
Dissect - logs.spring_boot llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.spring_boot parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.spring_boot pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Dissect - logs.syslog_bsd llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_bsd parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_bsd pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Dissect - logs.syslog_rfc5424 llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_rfc5424 parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_rfc5424 pattern_quality_score 1 0.90 0.90 0.00 - n/a -
Dissect - logs.system_metrics llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.system_metrics parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.system_metrics pattern_quality_score 1 0.96 0.92 +0.04 - n/a -
Dissect - logs.tsv llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.tsv parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.tsv pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Dissect - logs.vpn llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.vpn parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.vpn pattern_quality_score 1 1.00 1.00 0.00 - n/a -
Dissect - logs.web_access llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.web_access parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.web_access pattern_quality_score 1 0.93 0.93 0.00 - n/a -
Grok - logs.apache llm_extraction_quality 1 0.20 0.20 0.00 - n/a -
Grok - logs.apache parse_rate 1 0.70 0.70 0.00 - n/a -
Grok - logs.apache pattern_quality_score 1 0.79 0.79 0.00 - n/a -
Grok - logs.auth llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.auth parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.auth pattern_quality_score 1 0.94 0.94 0.00 - n/a -
Grok - logs.docker llm_extraction_quality 1 0.20 0.40 -0.20 - n/a -
Grok - logs.docker parse_rate 1 0.00 0.00 0.00 - n/a -
Grok - logs.docker pattern_quality_score 1 0.00 0.00 0.00 - n/a -
Grok - logs.hdfs llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.hdfs parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.hdfs pattern_quality_score 1 0.80 0.80 0.00 - n/a -
Grok - logs.kafka llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Grok - logs.kafka parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.kafka pattern_quality_score 1 0.72 0.72 0.00 - n/a -
Grok - logs.mongodb llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.mongodb parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.mongodb pattern_quality_score 1 0.69 0.69 0.00 - n/a -
Grok - logs.mysql_slowlog llm_extraction_quality 1 0.20 0.40 -0.20 - n/a -
Grok - logs.mysql_slowlog parse_rate 1 0.60 0.60 0.00 - n/a -
Grok - logs.mysql_slowlog pattern_quality_score 1 0.51 0.51 +0.00 - n/a -
Grok - logs.nginx llm_extraction_quality 1 0.40 0.40 0.00 - n/a -
Grok - logs.nginx parse_rate 1 0.22 0.22 0.00 - n/a -
Grok - logs.nginx pattern_quality_score 1 0.65 0.65 0.00 - n/a -
Grok - logs.nginx_error llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Grok - logs.nginx_error parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.nginx_error pattern_quality_score 1 0.75 0.75 0.00 - n/a -
Grok - logs.openssh llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.openssh parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.openssh pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Grok - logs.postgresql llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.postgresql parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.postgresql pattern_quality_score 1 0.82 0.82 0.00 - n/a -
Grok - logs.rabbitmq llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Grok - logs.rabbitmq parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.rabbitmq pattern_quality_score 1 0.67 0.67 0.00 - n/a -
Grok - logs.redis llm_extraction_quality 1 0.40 0.40 0.00 - n/a -
Grok - logs.redis parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.redis pattern_quality_score 1 0.63 0.63 0.00 - n/a -
Grok - logs.spark llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.spark parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.spark pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Grok - logs.syslog llm_extraction_quality 1 1.00 0.80 +0.20 - n/a -
Grok - logs.syslog parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.syslog pattern_quality_score 1 0.94 0.94 +0.00 - n/a -
Grok - logs.tomcat llm_extraction_quality 1 0.20 0.20 0.00 - n/a -
Grok - logs.tomcat parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.tomcat pattern_quality_score 1 0.38 0.38 0.00 - n/a -
Grok - logs.zookeeper llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.zookeeper parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.zookeeper pattern_quality_score 1 0.77 0.77 0.00 - n/a -
Homogeneous - no existing llm_partition_quality 1 0.83 0.83 0.00 - n/a -
Homogeneous - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Homogeneous - no existing partition_coverage 1 0.00 0.00 0.00 - n/a -
Multi-System - no existing llm_partition_quality 1 0.50 0.50 0.00 - n/a -
Multi-System - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Multi-System - no existing partition_coverage 1 0.00 0.00 0.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" llm_partition_quality 1 0.50 0.50 0.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" overlap_score 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" partition_coverage 1 0.00 0.00 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" llm_partition_quality 1 0.50 0.50 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" overlap_score 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" partition_coverage 1 0.00 0.00 0.00 - n/a -
Pipeline Suggestion - Apache llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Apache pipeline_quality_score 1 0.98 0.98 0.00 - n/a -
Pipeline Suggestion - HDFS llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - HDFS pipeline_quality_score 1 0.95 0.95 0.00 - n/a -
Pipeline Suggestion - OpenSSH llm_pipeline_quality 1 1.00 0.60 +0.40 - n/a -
Pipeline Suggestion - OpenSSH pipeline_quality_score 1 0.88 0.88 0.00 - n/a -
Pipeline Suggestion - Spark llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Spark pipeline_quality_score 1 0.97 0.97 0.00 - n/a -
Pipeline Suggestion - structured llm_pipeline_quality 1 0.00 0.00 0.00 - n/a -
Pipeline Suggestion - structured pipeline_quality_score 1 0.00 0.00 0.00 - n/a -
Pipeline Suggestion - Zookeeper llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - Zookeeper pipeline_quality_score 1 0.95 0.95 0.00 - n/a -
Refinement - 1 existing llm_partition_quality 1 0.50 0.50 0.00 - n/a -
Refinement - 1 existing overlap_score 1 1.00 1.00 0.00 - n/a -
Refinement - 1 existing partition_coverage 1 0.00 0.00 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" llm_partition_quality 1 0.50 0.50 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" partition_coverage 1 0.00 0.00 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" llm_partition_quality 1 0.50 0.50 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" partition_coverage 1 0.00 0.00 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" llm_partition_quality 1 0.50 0.50 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" partition_coverage 1 0.00 0.00 0.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" llm_partition_quality 1 0.50 0.50 0.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" partition_coverage 1 0.00 0.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::streams::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::streams::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (131 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (131 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Content-Based - no existing llm_partition_quality 1 0.50 0.67 -0.17 - n/a -
Content-Based - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Content-Based - no existing partition_coverage 1 0.00 1.00 -1.00 - n/a -
Dissect - logs.app llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.app pattern_quality_score 1 0.67 0.67 0.00 - n/a -
Dissect - logs.csv llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.csv parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.csv pattern_quality_score 1 0.80 0.80 0.00 - n/a -
Dissect - logs.db_pool llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.db_pool parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.db_pool pattern_quality_score 1 0.97 1.00 -0.03 - n/a -
Dissect - logs.db_query llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.db_query parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.db_query pattern_quality_score 1 0.78 0.82 -0.04 - n/a -
Dissect - logs.firewall llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.firewall parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.firewall pattern_quality_score 1 0.95 0.95 0.00 - n/a -
Dissect - logs.haproxy llm_extraction_quality 1 0.60 0.80 -0.20 - n/a -
Dissect - logs.haproxy parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.haproxy pattern_quality_score 1 0.87 0.91 -0.04 - n/a -
Dissect - logs.health_check llm_extraction_quality 1 0.60 0.40 +0.20 - n/a -
Dissect - logs.health_check parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.health_check pattern_quality_score 1 0.75 0.80 -0.06 - n/a -
Dissect - logs.keyvalue llm_extraction_quality 1 0.80 1.00 -0.20 - n/a -
Dissect - logs.keyvalue parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.keyvalue pattern_quality_score 1 0.83 0.67 +0.15 - n/a -
Dissect - logs.pipeline llm_extraction_quality 1 0.80 0.60 +0.20 - n/a -
Dissect - logs.pipeline parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.pipeline pattern_quality_score 1 0.75 0.70 +0.05 - n/a -
Dissect - logs.spring_boot llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.spring_boot parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.spring_boot pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Dissect - logs.syslog_bsd llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_bsd parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_bsd pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Dissect - logs.syslog_rfc5424 llm_extraction_quality 1 0.60 1.00 -0.40 - n/a -
Dissect - logs.syslog_rfc5424 parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.syslog_rfc5424 pattern_quality_score 1 0.94 0.97 -0.03 - n/a -
Dissect - logs.system_metrics llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.system_metrics parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.system_metrics pattern_quality_score 1 0.95 0.96 -0.02 - n/a -
Dissect - logs.tsv llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Dissect - logs.tsv parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.tsv pattern_quality_score 1 0.76 0.76 0.00 - n/a -
Dissect - logs.vpn llm_extraction_quality 1 0.80 0.80 0.00 - n/a -
Dissect - logs.vpn parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.vpn pattern_quality_score 1 1.00 1.00 0.00 - n/a -
Dissect - logs.web_access llm_extraction_quality 1 1.00 0.80 +0.20 - n/a -
Dissect - logs.web_access parse_rate 1 1.00 1.00 0.00 - n/a -
Dissect - logs.web_access pattern_quality_score 1 0.93 0.93 0.00 - n/a -
Grok - logs.apache llm_extraction_quality 1 0.20 0.20 0.00 - n/a -
Grok - logs.apache parse_rate 1 0.70 0.70 0.00 - n/a -
Grok - logs.apache pattern_quality_score 1 0.79 0.77 +0.03 - n/a -
Grok - logs.auth llm_extraction_quality 1 1.00 0.80 +0.20 - n/a -
Grok - logs.auth parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.auth pattern_quality_score 1 0.94 0.94 0.00 - n/a -
Grok - logs.docker llm_extraction_quality 1 0.40 0.80 -0.40 - n/a -
Grok - logs.docker parse_rate 1 0.00 0.00 0.00 - n/a -
Grok - logs.docker pattern_quality_score 1 0.00 0.00 0.00 - n/a -
Grok - logs.hdfs llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.hdfs parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.hdfs pattern_quality_score 1 0.80 0.80 0.00 - n/a -
Grok - logs.kafka llm_extraction_quality 1 0.60 0.60 0.00 - n/a -
Grok - logs.kafka parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.kafka pattern_quality_score 1 0.72 0.72 0.00 - n/a -
Grok - logs.mongodb llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.mongodb parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.mongodb pattern_quality_score 1 0.69 0.69 0.00 - n/a -
Grok - logs.mysql_slowlog llm_extraction_quality 1 0.40 0.40 0.00 - n/a -
Grok - logs.mysql_slowlog parse_rate 1 0.60 0.60 0.00 - n/a -
Grok - logs.mysql_slowlog pattern_quality_score 1 0.50 0.51 -0.01 - n/a -
Grok - logs.nginx llm_extraction_quality 1 0.40 0.20 +0.20 - n/a -
Grok - logs.nginx parse_rate 1 0.22 0.22 0.00 - n/a -
Grok - logs.nginx pattern_quality_score 1 0.70 0.68 +0.02 - n/a -
Grok - logs.nginx_error llm_extraction_quality 1 0.60 0.80 -0.20 - n/a -
Grok - logs.nginx_error parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.nginx_error pattern_quality_score 1 0.68 0.66 +0.02 - n/a -
Grok - logs.openssh llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.openssh parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.openssh pattern_quality_score 1 0.97 0.97 0.00 - n/a -
Grok - logs.postgresql llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.postgresql parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.postgresql pattern_quality_score 1 0.82 0.82 0.00 - n/a -
Grok - logs.rabbitmq llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.rabbitmq parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.rabbitmq pattern_quality_score 1 0.67 0.67 0.00 - n/a -
Grok - logs.redis llm_extraction_quality 1 0.40 0.60 -0.20 - n/a -
Grok - logs.redis parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.redis pattern_quality_score 1 0.59 0.58 +0.01 - n/a -
Grok - logs.spark llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.spark parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.spark pattern_quality_score 1 0.83 0.83 0.00 - n/a -
Grok - logs.syslog llm_extraction_quality 1 0.60 0.80 -0.20 - n/a -
Grok - logs.syslog parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.syslog pattern_quality_score 1 0.91 0.94 -0.02 - n/a -
Grok - logs.tomcat llm_extraction_quality 1 0.20 0.20 0.00 - n/a -
Grok - logs.tomcat parse_rate 1 0.90 1.00 -0.10 - n/a -
Grok - logs.tomcat pattern_quality_score 1 0.57 0.66 -0.10 - n/a -
Grok - logs.zookeeper llm_extraction_quality 1 1.00 1.00 0.00 - n/a -
Grok - logs.zookeeper parse_rate 1 1.00 1.00 0.00 - n/a -
Grok - logs.zookeeper pattern_quality_score 1 0.75 0.72 +0.04 - n/a -
Homogeneous - no existing llm_partition_quality 1 0.83 0.33 +0.50 - n/a -
Homogeneous - no existing overlap_score 1 1.00 0.04 +0.96 - n/a -
Homogeneous - no existing partition_coverage 1 0.00 1.00 -1.00 - n/a -
Multi-System - no existing llm_partition_quality 1 0.50 0.83 -0.33 - n/a -
Multi-System - no existing overlap_score 1 1.00 1.00 0.00 - n/a -
Multi-System - no existing partition_coverage 1 0.00 1.00 -1.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" llm_partition_quality 1 0.50 0.67 -0.17 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" overlap_score 1 1.00 0.00 +1.00 - n/a -
Overlapping Metadata - prompt: "Group logs by processing layer: batch pr…" partition_coverage 1 0.00 1.00 -1.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" llm_partition_quality 1 0.50 0.83 -0.33 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" overlap_score 1 1.00 1.00 0.00 - n/a -
Overlapping Metadata - prompt: "Split logs into data platform services a…" partition_coverage 1 0.00 1.00 -1.00 - n/a -
Pipeline Suggestion - structured llm_pipeline_quality 1 1.00 1.00 0.00 - n/a -
Pipeline Suggestion - structured pipeline_quality_score 1 1.00 1.00 0.00 - n/a -
Refinement - 1 existing llm_partition_quality 1 0.50 0.67 -0.17 - n/a -
Refinement - 1 existing overlap_score 1 1.00 1.00 0.00 - n/a -
Refinement - 1 existing partition_coverage 1 0.00 0.75 -0.75 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" llm_partition_quality 1 0.50 0.83 -0.33 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Create two partitions: cloud infrastruct…" partition_coverage 1 0.00 1.00 -1.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" llm_partition_quality 1 0.50 0.67 -0.17 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Group logs by their source system or ser…" partition_coverage 1 0.00 0.75 -0.75 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" llm_partition_quality 1 0.50 0.83 -0.33 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" overlap_score 1 1.00 1.00 0.00 - n/a -
User-Guided - prompt: "Partition logs by their service.name fie…" partition_coverage 1 0.00 1.00 -1.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" llm_partition_quality 1 0.50 0.33 +0.17 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" overlap_score 1 1.00 0.00 +1.00 - n/a -
User-Guided Refinement - prompt: "Partition the remaining logs by service …" partition_coverage 1 0.00 1.00 -1.00 - n/a -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

🧪 LLM eval (PR) — Streams (streams) failed.

Failing models:

  • eis-openai-gpt-oss-120b

View build

Triage summary:

ERROR Error: Playwright exited with code 1

x-pack/platform/packages/shared/kbn-evals-suite-streams/evals/pipeline_suggestion/pipeline_suggestion.spec.ts:52:17 — eis-openai-gpt-oss-120b
Root cause: Five pipeline suggestion quality evaluation tests failed during pipeline simulation. Inspect error-context.md and model suggestions to address quality evaluation regressions.

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-persona-matrix-attack-discovery::anthropic-claude-4.6-opus | Baseline (main): bk-019fdc33-33ac-4468-813f-ec408a156454::security-persona-matrix-attack-discovery::anthropic-claude-4.6-opus
Baseline: commit a9de85a, 35 days ago

Warning: Baseline is 35 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (6 evaluator comparisons).

Note: 4 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (6 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Attack Discovery Chrysalis kill-chains (generate API) AttackDiscoveryBasic 1 0.00 0.00 0.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) AttackDiscoveryRubric 1 0.00 0.00 0.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Input Tokens 1 105962.00 106173.00 -211.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Latency 1 291.31 287.71 +3.60 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Output Tokens 1 18083.00 18463.00 -380.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Tool Calls 1 0.00 0.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-persona-matrix-attack-discovery::anthropic-claude-4.6-sonnet | Baseline (main): bk-019fdc33-33ac-4468-813f-ec408a156454::security-persona-matrix-attack-discovery::anthropic-claude-4.6-sonnet
Baseline: commit a9de85a, 35 days ago

Warning: Baseline is 35 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (6 evaluator comparisons).

Note: 4 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (6 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Attack Discovery Chrysalis kill-chains (generate API) AttackDiscoveryBasic 1 0.00 0.00 0.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) AttackDiscoveryRubric 1 0.00 0.00 0.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Input Tokens 1 95733.00 95469.00 +264.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Latency 1 212.42 293.55 -81.13 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Output Tokens 1 10885.00 10719.00 +166.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Tool Calls 1 0.00 0.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-persona-matrix-attack-discovery::google-gemini-3.0-flash | Baseline (main): bk-019fdc33-33ac-4468-813f-ec408a156454::security-persona-matrix-attack-discovery::google-gemini-3.0-flash
Baseline: commit a9de85a, 35 days ago

Warning: Baseline is 35 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (6 evaluator comparisons).

Note: 4 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (6 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Attack Discovery Chrysalis kill-chains (generate API) AttackDiscoveryBasic 1 0.00 0.00 0.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) AttackDiscoveryRubric 1 0.00 0.00 0.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Input Tokens 1 108101.00 108209.00 -108.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Latency 1 26.43 22.12 +4.31 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Output Tokens 1 3803.00 3246.00 +557.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Tool Calls 1 0.00 0.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-persona-matrix-attack-discovery::google-gemini-3.1-pro | Baseline (main): bk-019fdc33-33ac-4468-813f-ec408a156454::security-persona-matrix-attack-discovery::google-gemini-3.1-pro
Baseline: commit a9de85a, 35 days ago

Warning: Baseline is 35 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (6 evaluator comparisons).

Note: 4 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (6 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Attack Discovery Chrysalis kill-chains (generate API) AttackDiscoveryBasic 1 0.00 0.00 0.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) AttackDiscoveryRubric 1 0.00 0.00 0.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Input Tokens 1 108148.00 107882.00 +266.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Latency 1 306.92 338.79 -31.88 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Output Tokens 1 34751.00 32941.00 +1810.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Tool Calls 1 0.00 0.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-persona-matrix-attack-discovery::openai-gpt-5.4 | Baseline (main): bk-019fdc33-33ac-4468-813f-ec408a156454::security-persona-matrix-attack-discovery::openai-gpt-5.4
Baseline: commit a9de85a, 35 days ago

Warning: Baseline is 35 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (6 evaluator comparisons).

Note: 4 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (6 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Attack Discovery Chrysalis kill-chains (generate API) AttackDiscoveryBasic 1 0.00 0.00 0.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) AttackDiscoveryRubric 1 0.00 0.00 0.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Input Tokens 1 80806.00 80879.00 -73.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Latency 1 48.43 46.83 +1.60 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Output Tokens 1 6250.00 5237.00 +1013.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Tool Calls 1 0.00 0.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-persona-matrix-attack-discovery::openai-gpt-oss-120b | Baseline (main): bk-019fdc33-33ac-4468-813f-ec408a156454::security-persona-matrix-attack-discovery::openai-gpt-oss-120b
Baseline: commit a9de85a, 35 days ago

Warning: Baseline is 35 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (6 evaluator comparisons).

Note: 1 unpaired (ran in only one experiment), 3 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (6 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Attack Discovery Chrysalis kill-chains (generate API) AttackDiscoveryBasic 1 0.00 0.00 0.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) AttackDiscoveryRubric 1 0.00 0.00 0.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Input Tokens 1 80683.00 80727.00 -44.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Latency 1 49.06 130.54 -81.48 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Output Tokens 1 5314.00 7024.00 -1710.00 - n/a -
Attack Discovery Chrysalis kill-chains (generate API) Tool Calls 1 0.00 0.00 0.00 - n/a -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::pci-compliance::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::pci-compliance::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (16 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (16 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
pci-compliance: field mapping PCI Criteria 1 0.64 0.64 0.00 - n/a -
pci-compliance: field mapping Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: full report PCI Criteria 1 0.73 0.73 0.00 - n/a -
pci-compliance: full report Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: no matching data PCI Criteria 1 1.00 1.00 0.00 - n/a -
pci-compliance: no matching data Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 2.2.4 default accounts PCI Criteria 1 1.00 1.00 0.00 - n/a -
pci-compliance: requirement 2.2.4 default accounts Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 4.1 weak TLS PCI Criteria 1 0.89 0.89 0.00 - n/a -
pci-compliance: requirement 4.1 weak TLS Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 8.3.4 brute force PCI Criteria 1 1.00 1.00 0.00 - n/a -
pci-compliance: requirement 8.3.4 brute force Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: scope discovery PCI Criteria 1 0.67 0.78 -0.11 - n/a -
pci-compliance: scope discovery Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: scoped to auth index PCI Criteria 1 1.00 1.00 0.00 - n/a -
pci-compliance: scoped to auth index Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::pci-compliance::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::pci-compliance::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (16 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (16 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
pci-compliance: field mapping PCI Criteria 1 0.73 0.73 0.00 - n/a -
pci-compliance: field mapping Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: full report PCI Criteria 1 0.91 0.73 +0.18 - n/a -
pci-compliance: full report Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: no matching data PCI Criteria 1 1.00 1.00 0.00 - n/a -
pci-compliance: no matching data Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 2.2.4 default accounts PCI Criteria 1 1.00 1.00 0.00 - n/a -
pci-compliance: requirement 2.2.4 default accounts Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 4.1 weak TLS PCI Criteria 1 0.89 0.89 0.00 - n/a -
pci-compliance: requirement 4.1 weak TLS Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 8.3.4 brute force PCI Criteria 1 1.00 1.00 0.00 - n/a -
pci-compliance: requirement 8.3.4 brute force Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: scope discovery PCI Criteria 1 0.89 0.78 +0.11 - n/a -
pci-compliance: scope discovery Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: scoped to auth index PCI Criteria 1 1.00 1.00 0.00 - n/a -
pci-compliance: scoped to auth index Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::pci-compliance::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::pci-compliance::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (16 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (16 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
pci-compliance: field mapping PCI Criteria 1 0.82 0.73 +0.09 - n/a -
pci-compliance: field mapping Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: full report PCI Criteria 1 0.64 0.55 +0.09 - n/a -
pci-compliance: full report Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: no matching data PCI Criteria 1 1.00 1.00 0.00 - n/a -
pci-compliance: no matching data Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 2.2.4 default accounts PCI Criteria 1 0.86 1.00 -0.14 - n/a -
pci-compliance: requirement 2.2.4 default accounts Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 4.1 weak TLS PCI Criteria 1 0.89 0.89 0.00 - n/a -
pci-compliance: requirement 4.1 weak TLS Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 8.3.4 brute force PCI Criteria 1 1.00 1.00 0.00 - n/a -
pci-compliance: requirement 8.3.4 brute force Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: scope discovery PCI Criteria 1 0.89 0.89 0.00 - n/a -
pci-compliance: scope discovery Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: scoped to auth index PCI Criteria 1 1.00 0.88 +0.13 - n/a -
pci-compliance: scoped to auth index Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::pci-compliance::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::pci-compliance::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (16 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (16 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
pci-compliance: field mapping PCI Criteria 1 0.73 0.64 +0.09 - n/a -
pci-compliance: field mapping Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: full report PCI Criteria 1 0.64 0.64 0.00 - n/a -
pci-compliance: full report Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: no matching data PCI Criteria 1 1.00 1.00 0.00 - n/a -
pci-compliance: no matching data Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 2.2.4 default accounts PCI Criteria 1 1.00 1.00 0.00 - n/a -
pci-compliance: requirement 2.2.4 default accounts Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 4.1 weak TLS PCI Criteria 1 0.89 1.00 -0.11 - n/a -
pci-compliance: requirement 4.1 weak TLS Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 8.3.4 brute force PCI Criteria 1 1.00 1.00 0.00 - n/a -
pci-compliance: requirement 8.3.4 brute force Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: scope discovery PCI Criteria 1 0.78 0.67 +0.11 - n/a -
pci-compliance: scope discovery Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: scoped to auth index PCI Criteria 1 1.00 1.00 0.00 - n/a -
pci-compliance: scoped to auth index Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::pci-compliance::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::pci-compliance::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (16 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (16 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
pci-compliance: field mapping PCI Criteria 1 0.73 0.64 +0.09 - n/a -
pci-compliance: field mapping Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: full report PCI Criteria 1 0.73 0.64 +0.09 - n/a -
pci-compliance: full report Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: no matching data PCI Criteria 1 1.00 0.88 +0.13 - n/a -
pci-compliance: no matching data Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 2.2.4 default accounts PCI Criteria 1 1.00 0.86 +0.14 - n/a -
pci-compliance: requirement 2.2.4 default accounts Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 4.1 weak TLS PCI Criteria 1 0.78 0.67 +0.11 - n/a -
pci-compliance: requirement 4.1 weak TLS Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 8.3.4 brute force PCI Criteria 1 0.89 0.89 0.00 - n/a -
pci-compliance: requirement 8.3.4 brute force Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: scope discovery PCI Criteria 1 0.89 0.67 +0.22 - n/a -
pci-compliance: scope discovery Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: scoped to auth index PCI Criteria 1 0.88 0.88 0.00 - n/a -
pci-compliance: scoped to auth index Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::pci-compliance::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::pci-compliance::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (16 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (16 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
pci-compliance: field mapping PCI Criteria 1 0.18 0.64 -0.45 - n/a -
pci-compliance: field mapping Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: full report PCI Criteria 1 0.18 0.73 -0.55 - n/a -
pci-compliance: full report Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: no matching data PCI Criteria 1 0.38 0.38 0.00 - n/a -
pci-compliance: no matching data Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 2.2.4 default accounts PCI Criteria 1 0.43 0.29 +0.14 - n/a -
pci-compliance: requirement 2.2.4 default accounts Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 4.1 weak TLS PCI Criteria 1 0.22 0.22 0.00 - n/a -
pci-compliance: requirement 4.1 weak TLS Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: requirement 8.3.4 brute force PCI Criteria 1 0.33 0.33 0.00 - n/a -
pci-compliance: requirement 8.3.4 brute force Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: scope discovery PCI Criteria 1 0.22 0.22 0.00 - n/a -
pci-compliance: scope discovery Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -
pci-compliance: scoped to auth index PCI Criteria 1 0.25 0.25 0.00 - n/a -
pci-compliance: scoped to auth index Skill Invoked (pci-compliance) 1 0.00 0.00 0.00 - n/a -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alert-triage::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-triage::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (34 evaluator comparisons).

Note: 1 unpaired (ran in only one experiment), 3 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (34 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder: security-alert-triage-e2e-summary-mode AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Cached Tokens 1 17464.00 17464.00 0.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Input Tokens 1 107234.00 107233.00 +1.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Latency 1 6.89 5.92 +0.97 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Output Tokens 1 670.00 663.00 +7.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Tool Calls 1 12.00 12.00 0.00 - n/a -
agent builder: security-alert-triage-entity-correlation AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-entity-correlation criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-entity-correlation Input Tokens 1 70927.00 83267.00 -12340.00 - n/a -
agent builder: security-alert-triage-entity-correlation Latency 1 5.17 5.89 -0.72 - n/a -
agent builder: security-alert-triage-entity-correlation Output Tokens 1 390.00 474.00 -84.00 - n/a -
agent builder: security-alert-triage-entity-correlation Tool Calls 1 6.00 8.00 -2.00 - n/a -
agent builder: security-alert-triage-priority AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-priority criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-priority Input Tokens 1 83150.00 83146.00 +4.00 - n/a -
agent builder: security-alert-triage-priority Latency 1 6.69 6.03 +0.66 - n/a -
agent builder: security-alert-triage-priority Output Tokens 1 574.00 580.00 -6.00 - n/a -
agent builder: security-alert-triage-priority Tool Calls 1 10.00 10.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert Cached Tokens 1 47022.00 32913.00 +14109.00 - n/a -
agent builder: security-alert-triage-single-alert criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert Input Tokens 1 183499.00 209681.00 -26182.00 - n/a -
agent builder: security-alert-triage-single-alert Latency 1 20.04 14.50 +5.54 - n/a -
agent builder: security-alert-triage-single-alert Output Tokens 1 3359.00 2791.00 +568.00 - n/a -
agent builder: security-alert-triage-single-alert Skill Invoked (alert-analysis) 1 0.00 0.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert Tool Calls 1 7.00 9.00 -2.00 - n/a -
agent builder: security-bulk-alerts-attachment-read AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Cached Tokens 1 10221.00 28304.00 -18083.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Input Tokens 1 115945.00 75595.00 +40350.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Latency 1 6.83 12.97 -6.14 - n/a -
agent builder: security-bulk-alerts-attachment-read Output Tokens 1 622.00 1726.00 -1104.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Tool Calls 1 9.00 9.00 0.00 - n/a -
ease: alert-summary-json-contract EaseJsonCompliance 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alert-triage::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-triage::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (31 evaluator comparisons).

Note: 1 unpaired (ran in only one experiment), 9 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (31 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder: security-alert-triage-e2e-summary-mode AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Input Tokens 1 17609.00 17608.00 +1.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Latency 1 4.19 3.81 +0.39 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Output Tokens 1 493.00 493.00 0.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Tool Calls 1 10.00 10.00 0.00 - n/a -
agent builder: security-alert-triage-entity-correlation AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-entity-correlation criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-entity-correlation Input Tokens 1 72736.00 91972.00 -19236.00 - n/a -
agent builder: security-alert-triage-entity-correlation Latency 1 3.92 3.61 +0.30 - n/a -
agent builder: security-alert-triage-entity-correlation Output Tokens 1 485.00 507.00 -22.00 - n/a -
agent builder: security-alert-triage-entity-correlation Tool Calls 1 8.00 8.00 0.00 - n/a -
agent builder: security-alert-triage-priority AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-priority criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-priority Input Tokens 1 73366.00 73370.00 -4.00 - n/a -
agent builder: security-alert-triage-priority Latency 1 8.01 7.67 +0.34 - n/a -
agent builder: security-alert-triage-priority Output Tokens 1 1307.00 1322.00 -15.00 - n/a -
agent builder: security-alert-triage-priority Tool Calls 1 7.00 7.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert Input Tokens 1 171572.00 110365.00 +61207.00 - n/a -
agent builder: security-alert-triage-single-alert Latency 1 5.98 5.01 +0.97 - n/a -
agent builder: security-alert-triage-single-alert Output Tokens 1 819.00 656.00 +163.00 - n/a -
agent builder: security-alert-triage-single-alert Skill Invoked (alert-analysis) 1 0.00 0.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert Tool Calls 1 8.00 6.00 +2.00 - n/a -
agent builder: security-bulk-alerts-attachment-read AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Input Tokens 1 45622.00 45636.00 -14.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Latency 1 9.53 9.29 +0.24 - n/a -
agent builder: security-bulk-alerts-attachment-read Output Tokens 1 1587.00 1603.00 -16.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Tool Calls 1 8.00 8.00 0.00 - n/a -
ease: alert-summary-json-contract EaseJsonCompliance 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alert-triage::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-triage::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (32 evaluator comparisons).

Note: 8 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (32 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder: security-alert-triage-e2e-summary-mode AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode criteria 1 1.00 0.75 +0.25 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Input Tokens 1 259.00 13998.00 -13739.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Latency 1 0.89 3.32 -2.43 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Output Tokens 1 21.00 278.00 -257.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Tool Calls 1 0.00 10.00 -10.00 - n/a -
agent builder: security-alert-triage-entity-correlation AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-entity-correlation criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-entity-correlation Input Tokens 1 13786.00 13785.00 +1.00 - n/a -
agent builder: security-alert-triage-entity-correlation Latency 1 2.46 2.62 -0.16 - n/a -
agent builder: security-alert-triage-entity-correlation Output Tokens 1 152.00 156.00 -4.00 - n/a -
agent builder: security-alert-triage-entity-correlation Tool Calls 1 0.00 5.00 -5.00 - n/a -
agent builder: security-alert-triage-priority AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-priority criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-priority Input Tokens 1 13790.00 13790.00 0.00 - n/a -
agent builder: security-alert-triage-priority Latency 1 2.91 3.08 -0.17 - n/a -
agent builder: security-alert-triage-priority Output Tokens 1 171.00 243.00 -72.00 - n/a -
agent builder: security-alert-triage-priority Tool Calls 1 5.00 8.00 -3.00 - n/a -
agent builder: security-alert-triage-single-alert AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert Cached Tokens 1 28582.00 32681.00 -4099.00 - n/a -
agent builder: security-alert-triage-single-alert criteria 1 0.80 0.80 0.00 - n/a -
agent builder: security-alert-triage-single-alert Input Tokens 1 152470.00 187104.00 -34634.00 - n/a -
agent builder: security-alert-triage-single-alert Latency 1 7.45 5.54 +1.92 - n/a -
agent builder: security-alert-triage-single-alert Output Tokens 1 2440.00 1410.00 +1030.00 - n/a -
agent builder: security-alert-triage-single-alert Skill Invoked (alert-analysis) 1 0.00 0.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert Tool Calls 1 5.00 9.00 -4.00 - n/a -
agent builder: security-bulk-alerts-attachment-read AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Input Tokens 1 13836.00 44551.00 -30715.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Latency 1 2.89 3.04 -0.15 - n/a -
agent builder: security-bulk-alerts-attachment-read Output Tokens 1 204.00 355.00 -151.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Tool Calls 1 6.00 9.00 -3.00 - n/a -
ease: alert-summary-json-contract EaseJsonCompliance 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alert-triage::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-triage::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (32 evaluator comparisons).

Note: 8 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (32 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder: security-alert-triage-e2e-summary-mode AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Input Tokens 1 13998.00 14002.00 -4.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Latency 1 5.07 7.46 -2.39 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Output Tokens 1 378.00 704.00 -326.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Tool Calls 1 10.00 11.00 -1.00 - n/a -
agent builder: security-alert-triage-entity-correlation AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-entity-correlation criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-entity-correlation Input Tokens 1 13786.00 13784.00 +2.00 - n/a -
agent builder: security-alert-triage-entity-correlation Latency 1 5.18 4.51 +0.66 - n/a -
agent builder: security-alert-triage-entity-correlation Output Tokens 1 383.00 323.00 +60.00 - n/a -
agent builder: security-alert-triage-entity-correlation Tool Calls 1 5.00 5.00 0.00 - n/a -
agent builder: security-alert-triage-priority AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-priority criteria 1 0.80 0.80 0.00 - n/a -
agent builder: security-alert-triage-priority Input Tokens 1 92572.00 92573.00 -1.00 - n/a -
agent builder: security-alert-triage-priority Latency 1 9.69 18.13 -8.45 - n/a -
agent builder: security-alert-triage-priority Output Tokens 1 2216.00 2531.00 -315.00 - n/a -
agent builder: security-alert-triage-priority Tool Calls 1 9.00 9.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert Cached Tokens 1 13977.00 54538.00 -40561.00 - n/a -
agent builder: security-alert-triage-single-alert criteria 1 0.20 0.80 -0.60 - n/a -
agent builder: security-alert-triage-single-alert Input Tokens 1 154896.00 212076.00 -57180.00 - n/a -
agent builder: security-alert-triage-single-alert Latency 1 16.12 5.37 +10.74 - n/a -
agent builder: security-alert-triage-single-alert Output Tokens 1 3425.00 1002.00 +2423.00 - n/a -
agent builder: security-alert-triage-single-alert Skill Invoked (alert-analysis) 1 0.00 0.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert Tool Calls 1 8.00 8.00 0.00 - n/a -
agent builder: security-bulk-alerts-attachment-read AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Input Tokens 1 13833.00 258.00 +13575.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Latency 1 4.68 0.80 +3.88 - n/a -
agent builder: security-bulk-alerts-attachment-read Output Tokens 1 371.00 21.00 +350.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Tool Calls 1 6.00 0.00 +6.00 - n/a -
ease: alert-summary-json-contract EaseJsonCompliance 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alert-triage::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-triage::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (34 evaluator comparisons).

Note: 1 unpaired (ran in only one experiment), 3 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (34 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder: security-alert-triage-e2e-summary-mode AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Cached Tokens 1 2432.00 5504.00 -3072.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Input Tokens 1 13593.00 13589.00 +4.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Latency 1 3.92 4.66 -0.74 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Output Tokens 1 305.00 306.00 -1.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Tool Calls 1 10.00 10.00 0.00 - n/a -
agent builder: security-alert-triage-entity-correlation AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-entity-correlation criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-entity-correlation Input Tokens 1 13402.00 13406.00 -4.00 - n/a -
agent builder: security-alert-triage-entity-correlation Latency 1 5.57 3.99 +1.58 - n/a -
agent builder: security-alert-triage-entity-correlation Output Tokens 1 194.00 188.00 +6.00 - n/a -
agent builder: security-alert-triage-entity-correlation Tool Calls 1 5.00 5.00 0.00 - n/a -
agent builder: security-alert-triage-priority AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-priority criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-priority Input Tokens 1 13409.00 13411.00 -2.00 - n/a -
agent builder: security-alert-triage-priority Latency 1 3.61 4.23 -0.63 - n/a -
agent builder: security-alert-triage-priority Output Tokens 1 192.00 201.00 -9.00 - n/a -
agent builder: security-alert-triage-priority Tool Calls 1 5.00 5.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert Cached Tokens 1 26496.00 26496.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert criteria 1 0.80 1.00 -0.20 - n/a -
agent builder: security-alert-triage-single-alert Input Tokens 1 43623.00 43617.00 +6.00 - n/a -
agent builder: security-alert-triage-single-alert Latency 1 3.26 10.11 -6.85 - n/a -
agent builder: security-alert-triage-single-alert Output Tokens 1 292.00 293.00 -1.00 - n/a -
agent builder: security-alert-triage-single-alert Skill Invoked (alert-analysis) 1 0.00 0.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert Tool Calls 1 4.00 4.00 0.00 - n/a -
agent builder: security-bulk-alerts-attachment-read AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Cached Tokens 1 18688.00 8576.00 +10112.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Input Tokens 1 29177.00 39606.00 -10429.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Latency 1 0.29 4.08 -3.78 - n/a -
agent builder: security-bulk-alerts-attachment-read Output Tokens 1 268.00 323.00 -55.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Tool Calls 1 0.00 9.00 -9.00 - n/a -
ease: alert-summary-json-contract EaseJsonCompliance 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alert-triage::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-triage::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (32 evaluator comparisons).

Note: 1 unpaired (ran in only one experiment), 7 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (32 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
agent builder: security-alert-triage-e2e-summary-mode AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Cached Tokens 1 87.00 5855.00 -5768.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode criteria 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Input Tokens 1 13645.00 13640.00 +5.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Latency 1 7.35 7.25 +0.10 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Output Tokens 1 845.00 759.00 +86.00 - n/a -
agent builder: security-alert-triage-e2e-summary-mode Tool Calls 1 10.00 10.00 0.00 - n/a -
agent builder: security-alert-triage-entity-correlation AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-entity-correlation criteria 1 0.25 1.00 -0.75 - n/a -
agent builder: security-alert-triage-entity-correlation Input Tokens 1 16544.00 32350.00 -15806.00 - n/a -
agent builder: security-alert-triage-entity-correlation Latency 1 2.24 1.65 +0.59 - n/a -
agent builder: security-alert-triage-entity-correlation Output Tokens 1 74.00 145.00 -71.00 - n/a -
agent builder: security-alert-triage-entity-correlation Tool Calls 1 1.00 2.00 -1.00 - n/a -
agent builder: security-alert-triage-priority AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-priority criteria 1 0.20 0.80 -0.60 - n/a -
agent builder: security-alert-triage-priority Input Tokens 1 16549.00 57132.00 -40583.00 - n/a -
agent builder: security-alert-triage-priority Latency 1 4.99 7.35 -2.36 - n/a -
agent builder: security-alert-triage-priority Output Tokens 1 285.00 589.00 -304.00 - n/a -
agent builder: security-alert-triage-priority Tool Calls 1 1.00 3.00 -2.00 - n/a -
agent builder: security-alert-triage-single-alert AttachmentReadCompliance 1 1.00 1.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert criteria 1 0.40 0.40 0.00 - n/a -
agent builder: security-alert-triage-single-alert Input Tokens 1 13508.00 99860.00 -86352.00 - n/a -
agent builder: security-alert-triage-single-alert Latency 1 12.96 4.28 +8.69 - n/a -
agent builder: security-alert-triage-single-alert Output Tokens 1 1046.00 1221.00 -175.00 - n/a -
agent builder: security-alert-triage-single-alert Skill Invoked (alert-analysis) 1 0.00 0.00 0.00 - n/a -
agent builder: security-alert-triage-single-alert Tool Calls 1 11.00 4.00 +7.00 - n/a -
agent builder: security-bulk-alerts-attachment-read AttachmentReadCompliance 1 0.17 0.00 +0.17 - n/a -
agent builder: security-bulk-alerts-attachment-read Input Tokens 1 16590.00 13445.00 +3145.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Latency 1 4.52 8.68 -4.17 - n/a -
agent builder: security-bulk-alerts-attachment-read Output Tokens 1 568.00 904.00 -336.00 - n/a -
agent builder: security-bulk-alerts-attachment-read Tool Calls 1 1.00 1.00 0.00 - n/a -
ease: alert-summary-json-contract EaseJsonCompliance 1 0.00 1.00 -1.00 - n/a -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-esql-generation-regression::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-esql-generation-regression::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (9 evaluator comparisons).

Note: 12 unpaired (ran in only one experiment), 19 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (9 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security-esql-generation: regression suite Cached Tokens 15 28713.73 31784.20 -3070.47 0.65 No -
security-esql-generation: regression suite ES|QL Execution Validity 31 0.98 0.95 +0.03 0.15 No -
security-esql-generation: regression suite ES|QL Functional Equivalence 31 0.15 0.16 -0.02 0.76 No -
security-esql-generation: regression suite ES|QL Result Equivalence 31 0.48 0.48 0.00 1.00 No -
security-esql-generation: regression suite ES|QL Validity 31 1.00 0.97 +0.03 0.32 No -
security-esql-generation: regression suite Input Tokens 31 48212.52 44436.45 +3776.06 0.59 No -
security-esql-generation: regression suite Latency 31 6.86 5.84 +1.02 0.05 No -
security-esql-generation: regression suite Output Tokens 31 627.06 540.81 +86.26 0.18 No -
security-esql-generation: regression suite Tool Calls 31 1.65 1.55 +0.10 0.75 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-esql-generation-regression::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-esql-generation-regression::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
2 significant difference(s) detected out of 9 comparisons.

Note: 7 unpaired (ran in only one experiment), 24 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security-esql-generation: regression suite Input Tokens 31 99990.81 126196.81 -26206.00 0.01 Yes Improvement
security-esql-generation: regression suite Tool Calls 31 4.65 6.10 -1.45 0.01 Yes -
No significant changes (7 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security-esql-generation: regression suite Cached Tokens 14 25568.29 26746.79 -1178.50 0.80 No -
security-esql-generation: regression suite ES|QL Execution Validity 31 0.87 0.79 +0.08 0.37 No -
security-esql-generation: regression suite ES|QL Functional Equivalence 31 0.16 0.16 0.00 1.00 No -
security-esql-generation: regression suite ES|QL Result Equivalence 31 0.39 0.39 0.00 1.00 No -
security-esql-generation: regression suite ES|QL Validity 30 0.97 0.87 +0.10 0.18 No -
security-esql-generation: regression suite Latency 31 5.19 5.21 -0.02 0.96 No -
security-esql-generation: regression suite Output Tokens 31 1136.29 1394.23 -257.94 0.06 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-esql-generation-regression::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-esql-generation-regression::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (9 evaluator comparisons).

Note: 4 unpaired (ran in only one experiment), 16 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (9 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security-esql-generation: regression suite Cached Tokens 20 72211.25 61605.10 +10606.15 0.60 No -
security-esql-generation: regression suite ES|QL Execution Validity 31 0.97 1.00 -0.03 0.15 No -
security-esql-generation: regression suite ES|QL Functional Equivalence 31 0.24 0.19 +0.05 0.32 No -
security-esql-generation: regression suite ES|QL Result Equivalence 31 0.45 0.48 -0.03 0.32 No -
security-esql-generation: regression suite ES|QL Validity 31 1.00 1.00 0.00 1.00 No -
security-esql-generation: regression suite Input Tokens 31 92929.90 75328.55 +17601.35 0.39 No -
security-esql-generation: regression suite Latency 31 3.54 3.92 -0.38 0.65 No -
security-esql-generation: regression suite Output Tokens 31 591.48 607.03 -15.55 0.91 No -
security-esql-generation: regression suite Tool Calls 31 5.61 3.90 +1.71 0.18 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-esql-generation-regression::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-esql-generation-regression::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
2 significant difference(s) detected out of 9 comparisons.

Note: 9 unpaired (ran in only one experiment), 34 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security-esql-generation: regression suite Input Tokens 31 25749.19 16586.77 +9162.42 0.05 Yes Regression
security-esql-generation: regression suite Tool Calls 31 0.71 0.29 +0.42 0.03 Yes -
No significant changes (7 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security-esql-generation: regression suite Cached Tokens 7 26062.57 15722.57 +10340.00 0.24 No -
security-esql-generation: regression suite ES|QL Execution Validity 31 1.00 0.98 +0.02 0.32 No -
security-esql-generation: regression suite ES|QL Functional Equivalence 31 0.15 0.18 -0.03 0.48 No -
security-esql-generation: regression suite ES|QL Result Equivalence 31 0.48 0.45 +0.03 0.32 No -
security-esql-generation: regression suite ES|QL Validity 31 1.00 1.00 0.00 1.00 No -
security-esql-generation: regression suite Latency 31 4.24 2.95 +1.29 0.10 No -
security-esql-generation: regression suite Output Tokens 31 577.00 381.55 +195.45 0.06 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-esql-generation-regression::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-esql-generation-regression::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (9 evaluator comparisons).

Note: 6 unpaired (ran in only one experiment), 21 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (9 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security-esql-generation: regression suite Cached Tokens 16 12416.31 12435.00 -18.69 0.99 No -
security-esql-generation: regression suite ES|QL Execution Validity 31 0.97 0.92 +0.05 0.08 No -
security-esql-generation: regression suite ES|QL Functional Equivalence 31 0.11 0.16 -0.05 0.08 No -
security-esql-generation: regression suite ES|QL Result Equivalence 31 0.48 0.45 +0.03 0.32 No -
security-esql-generation: regression suite ES|QL Validity 31 0.97 0.97 0.00 1.00 No -
security-esql-generation: regression suite Input Tokens 31 13844.39 18286.35 -4441.97 0.34 No -
security-esql-generation: regression suite Latency 31 2.53 2.88 -0.36 0.57 No -
security-esql-generation: regression suite Output Tokens 31 223.71 284.19 -60.48 0.36 No -
security-esql-generation: regression suite Tool Calls 31 0.16 0.42 -0.26 0.11 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-esql-generation-regression::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-esql-generation-regression::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 9 comparisons.

Note: 9 unpaired (ran in only one experiment), 38 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security-esql-generation: regression suite ES|QL Functional Equivalence 31 0.18 0.05 +0.13 0.03 Yes Improvement
No significant changes (8 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security-esql-generation: regression suite Cached Tokens 5 11341.00 13508.20 -2167.20 0.45 No -
security-esql-generation: regression suite ES|QL Execution Validity 31 0.76 0.84 -0.08 0.41 No -
security-esql-generation: regression suite ES|QL Result Equivalence 31 0.39 0.39 0.00 1.00 No -
security-esql-generation: regression suite ES|QL Validity 31 0.94 0.97 -0.03 0.57 No -
security-esql-generation: regression suite Input Tokens 31 13469.94 19888.61 -6418.68 0.19 No -
security-esql-generation: regression suite Latency 31 2.06 2.08 -0.02 0.96 No -
security-esql-generation: regression suite Output Tokens 31 287.03 418.61 -131.58 0.09 No -
security-esql-generation: regression suite Tool Calls 31 0.10 0.32 -0.23 0.10 No -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alerts-rag-regression::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alerts-rag-regression::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 55 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Security Alerts RAG Regression › single_alert_query Latency 2 62.54 52.71 +9.83 0.05 Yes Regression
No significant changes (54 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Security Alerts RAG Regression › field_specific_lookup Cached Tokens 2 44584.50 39446.50 +5138.00 0.39 No -
Security Alerts RAG Regression › field_specific_lookup Factuality 2 0.31 0.31 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Groundedness 2 0.49 0.49 +0.00 0.39 No -
Security Alerts RAG Regression › field_specific_lookup Input Tokens 2 166936.00 166413.50 +522.50 0.14 No -
Security Alerts RAG Regression › field_specific_lookup Latency 2 36.05 35.60 +0.45 0.74 No -
Security Alerts RAG Regression › field_specific_lookup Output Tokens 2 2536.00 2488.00 +48.00 0.13 No -
Security Alerts RAG Regression › field_specific_lookup Relevance 2 0.29 0.29 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Tool Calls 2 3.00 3.00 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Trajectory 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Cached Tokens 2 17307.50 36701.00 -19393.50 0.39 No -
Security Alerts RAG Regression › multi_alert_correlation Factuality 2 0.24 0.23 +0.01 0.89 No -
Security Alerts RAG Regression › multi_alert_correlation Groundedness 2 0.42 0.49 -0.07 0.39 No -
Security Alerts RAG Regression › multi_alert_correlation Input Tokens 2 105248.50 104599.50 +649.00 0.38 No -
Security Alerts RAG Regression › multi_alert_correlation Latency 2 29.90 21.90 +8.00 0.31 No -
Security Alerts RAG Regression › multi_alert_correlation Output Tokens 2 967.00 743.50 +223.50 0.34 No -
Security Alerts RAG Regression › multi_alert_correlation Relevance 2 0.47 0.47 -0.01 0.97 No -
Security Alerts RAG Regression › multi_alert_correlation Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Tool Calls 2 2.50 2.50 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Trajectory 2 0.50 0.50 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Cached Tokens 2 61176.50 67877.00 -6700.50 0.68 No -
Security Alerts RAG Regression › single_alert_query Factuality 2 0.29 0.27 +0.02 0.66 No -
Security Alerts RAG Regression › single_alert_query Groundedness 2 0.83 0.99 -0.15 0.42 No -
Security Alerts RAG Regression › single_alert_query Input Tokens 2 283367.50 243080.50 +40287.00 0.38 No -
Security Alerts RAG Regression › single_alert_query Output Tokens 2 4770.00 4090.00 +680.00 0.29 No -
Security Alerts RAG Regression › single_alert_query Relevance 2 0.58 0.65 -0.07 0.76 No -
Security Alerts RAG Regression › single_alert_query Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Tool Calls 2 5.00 5.00 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Trajectory 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › space_isolation Cached Tokens 1 16981.00 16981.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Factuality 1 0.62 0.69 -0.07 - n/a -
Security Alerts RAG Regression › space_isolation Groundedness 1 0.83 0.86 -0.03 - n/a -
Security Alerts RAG Regression › space_isolation Input Tokens 1 49203.00 49200.00 +3.00 - n/a -
Security Alerts RAG Regression › space_isolation Latency 1 13.76 13.77 -0.01 - n/a -
Security Alerts RAG Regression › space_isolation Output Tokens 1 329.00 339.00 -10.00 - n/a -
Security Alerts RAG Regression › space_isolation Relevance 1 0.33 0.50 -0.17 - n/a -
Security Alerts RAG Regression › space_isolation Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Skill Invoked (alert-analysis) 1 0.00 0.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Tool Calls 1 2.00 2.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Trajectory 1 1.00 1.00 0.00 - n/a -
Security Alerts RAG Regression › temporal_query Cached Tokens 2 27199.00 38424.00 -11225.00 0.59 No -
Security Alerts RAG Regression › temporal_query Factuality 2 0.43 0.60 -0.17 0.06 No -
Security Alerts RAG Regression › temporal_query Groundedness 2 0.99 0.99 +0.00 0.39 No -
Security Alerts RAG Regression › temporal_query Input Tokens 2 126143.00 144806.50 -18663.50 0.39 No -
Security Alerts RAG Regression › temporal_query Latency 2 30.63 33.62 -2.99 0.15 No -
Security Alerts RAG Regression › temporal_query Output Tokens 2 2360.50 2805.50 -445.00 0.26 No -
Security Alerts RAG Regression › temporal_query Relevance 2 0.42 0.37 +0.05 0.39 No -
Security Alerts RAG Regression › temporal_query Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Tool Calls 2 3.00 3.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Trajectory 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alerts-rag-regression::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alerts-rag-regression::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
4 significant difference(s) detected out of 54 comparisons.

Note: 4 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Security Alerts RAG Regression › field_specific_lookup Factuality 2 0.30 0.19 +0.11 0.00 Yes Improvement
Security Alerts RAG Regression › temporal_query Groundedness 2 0.86 0.91 -0.05 0.04 Yes Regression
Security Alerts RAG Regression › temporal_query Latency 2 20.60 20.66 -0.07 0.01 Yes Improvement
Security Alerts RAG Regression › temporal_query Output Tokens 2 2154.00 2196.50 -42.50 0.03 Yes Improvement
No significant changes (50 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Security Alerts RAG Regression › field_specific_lookup Cached Tokens 2 10222.00 23483.50 -13261.50 0.39 No -
Security Alerts RAG Regression › field_specific_lookup Groundedness 2 0.50 0.89 -0.40 0.50 No -
Security Alerts RAG Regression › field_specific_lookup Input Tokens 2 149629.00 151432.50 -1803.50 0.95 No -
Security Alerts RAG Regression › field_specific_lookup Latency 2 22.82 25.01 -2.19 0.37 No -
Security Alerts RAG Regression › field_specific_lookup Output Tokens 2 1693.50 2491.00 -797.50 0.30 No -
Security Alerts RAG Regression › field_specific_lookup Relevance 2 0.33 0.63 -0.30 0.07 No -
Security Alerts RAG Regression › field_specific_lookup Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Tool Calls 2 3.00 3.00 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Trajectory 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Cached Tokens 1 6136.00 10216.00 -4080.00 - n/a -
Security Alerts RAG Regression › multi_alert_correlation Factuality 2 0.27 0.30 -0.03 0.69 No -
Security Alerts RAG Regression › multi_alert_correlation Groundedness 2 0.92 0.49 +0.42 0.48 No -
Security Alerts RAG Regression › multi_alert_correlation Input Tokens 2 88206.50 107299.50 -19093.00 0.39 No -
Security Alerts RAG Regression › multi_alert_correlation Latency 2 17.40 23.82 -6.42 0.44 No -
Security Alerts RAG Regression › multi_alert_correlation Output Tokens 2 1431.50 1897.00 -465.50 0.40 No -
Security Alerts RAG Regression › multi_alert_correlation Relevance 2 0.39 0.50 -0.11 0.74 No -
Security Alerts RAG Regression › multi_alert_correlation Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Tool Calls 2 2.50 2.50 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Trajectory 2 0.50 0.50 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Cached Tokens 2 30666.50 13278.50 +17388.00 0.08 No -
Security Alerts RAG Regression › single_alert_query Factuality 2 0.26 0.22 +0.04 0.73 No -
Security Alerts RAG Regression › single_alert_query Groundedness 2 0.48 0.47 +0.01 0.99 No -
Security Alerts RAG Regression › single_alert_query Input Tokens 2 274014.00 427518.50 -153504.50 0.12 No -
Security Alerts RAG Regression › single_alert_query Latency 2 43.14 58.17 -15.02 0.59 No -
Security Alerts RAG Regression › single_alert_query Output Tokens 2 5011.00 5661.50 -650.50 0.84 No -
Security Alerts RAG Regression › single_alert_query Relevance 2 0.65 0.50 +0.15 0.39 No -
Security Alerts RAG Regression › single_alert_query Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Tool Calls 2 6.00 8.50 -2.50 0.39 No -
Security Alerts RAG Regression › single_alert_query Trajectory 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › space_isolation Factuality 1 0.69 0.77 -0.08 - n/a -
Security Alerts RAG Regression › space_isolation Groundedness 1 0.79 0.79 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Input Tokens 1 48862.00 48886.00 -24.00 - n/a -
Security Alerts RAG Regression › space_isolation Latency 1 6.73 8.70 -1.96 - n/a -
Security Alerts RAG Regression › space_isolation Output Tokens 1 288.00 282.00 +6.00 - n/a -
Security Alerts RAG Regression › space_isolation Relevance 1 0.50 0.67 -0.17 - n/a -
Security Alerts RAG Regression › space_isolation Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Skill Invoked (alert-analysis) 1 0.00 0.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Tool Calls 1 2.00 2.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Trajectory 1 1.00 1.00 0.00 - n/a -
Security Alerts RAG Regression › temporal_query Cached Tokens 2 10221.50 5107.00 +5114.50 0.09 No -
Security Alerts RAG Regression › temporal_query Factuality 2 0.51 0.51 +0.00 0.39 No -
Security Alerts RAG Regression › temporal_query Input Tokens 2 125166.00 125309.50 -143.50 0.67 No -
Security Alerts RAG Regression › temporal_query Relevance 2 0.38 0.63 -0.25 0.39 No -
Security Alerts RAG Regression › temporal_query Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Tool Calls 2 3.00 3.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Trajectory 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alerts-rag-regression::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alerts-rag-regression::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 55 comparisons.

Note: 1 unpaired (ran in only one experiment), 1 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Security Alerts RAG Regression › multi_alert_correlation Relevance 2 0.52 0.68 -0.15 0.03 Yes Regression
No significant changes (54 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Security Alerts RAG Regression › field_specific_lookup Cached Tokens 2 108926.50 81549.50 +27377.00 0.81 No -
Security Alerts RAG Regression › field_specific_lookup Factuality 2 0.32 0.26 +0.06 0.71 No -
Security Alerts RAG Regression › field_specific_lookup Groundedness 2 1.00 0.66 +0.34 0.39 No -
Security Alerts RAG Regression › field_specific_lookup Input Tokens 2 279037.00 237286.50 +41750.50 0.84 No -
Security Alerts RAG Regression › field_specific_lookup Latency 2 29.27 38.09 -8.82 0.25 No -
Security Alerts RAG Regression › field_specific_lookup Output Tokens 2 2172.50 1597.00 +575.50 0.67 No -
Security Alerts RAG Regression › field_specific_lookup Relevance 2 0.31 0.42 -0.12 0.73 No -
Security Alerts RAG Regression › field_specific_lookup Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Tool Calls 2 5.00 4.50 +0.50 0.82 No -
Security Alerts RAG Regression › field_specific_lookup Trajectory 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Cached Tokens 2 40710.50 23968.50 +16742.00 0.51 No -
Security Alerts RAG Regression › multi_alert_correlation Factuality 2 0.23 0.17 +0.05 0.16 No -
Security Alerts RAG Regression › multi_alert_correlation Groundedness 2 0.39 0.99 -0.60 0.27 No -
Security Alerts RAG Regression › multi_alert_correlation Input Tokens 2 123451.50 114284.50 +9167.00 0.75 No -
Security Alerts RAG Regression › multi_alert_correlation Latency 2 18.04 16.57 +1.47 0.77 No -
Security Alerts RAG Regression › multi_alert_correlation Output Tokens 2 1720.00 1297.00 +423.00 0.52 No -
Security Alerts RAG Regression › multi_alert_correlation Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Tool Calls 2 3.00 4.00 -1.00 0.39 No -
Security Alerts RAG Regression › multi_alert_correlation Trajectory 2 0.50 0.50 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Cached Tokens 2 23705.50 71463.00 -47757.50 0.29 No -
Security Alerts RAG Regression › single_alert_query Factuality 2 0.23 0.22 +0.01 0.91 No -
Security Alerts RAG Regression › single_alert_query Groundedness 2 0.80 0.46 +0.34 0.44 No -
Security Alerts RAG Regression › single_alert_query Input Tokens 2 133960.50 216402.50 -82442.00 0.37 No -
Security Alerts RAG Regression › single_alert_query Latency 2 24.56 42.02 -17.46 0.16 No -
Security Alerts RAG Regression › single_alert_query Output Tokens 2 2362.00 3290.00 -928.00 0.14 No -
Security Alerts RAG Regression › single_alert_query Relevance 2 0.65 0.51 +0.14 0.56 No -
Security Alerts RAG Regression › single_alert_query Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Tool Calls 2 3.00 4.00 -1.00 0.39 No -
Security Alerts RAG Regression › single_alert_query Trajectory 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › space_isolation Cached Tokens 1 11135.00 11109.00 +26.00 - n/a -
Security Alerts RAG Regression › space_isolation Factuality 1 1.00 0.79 +0.21 - n/a -
Security Alerts RAG Regression › space_isolation Groundedness 1 1.00 0.79 +0.21 - n/a -
Security Alerts RAG Regression › space_isolation Input Tokens 1 25814.00 39644.00 -13830.00 - n/a -
Security Alerts RAG Regression › space_isolation Latency 1 2.07 6.69 -4.62 - n/a -
Security Alerts RAG Regression › space_isolation Output Tokens 1 100.00 152.00 -52.00 - n/a -
Security Alerts RAG Regression › space_isolation Relevance 1 1.00 0.33 +0.67 - n/a -
Security Alerts RAG Regression › space_isolation Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Skill Invoked (alert-analysis) 1 0.00 0.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Tool Calls 1 2.00 2.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Trajectory 1 1.00 1.00 0.00 - n/a -
Security Alerts RAG Regression › temporal_query Cached Tokens 1 32687.00 34711.00 -2024.00 - n/a -
Security Alerts RAG Regression › temporal_query Factuality 2 0.61 0.67 -0.06 0.39 No -
Security Alerts RAG Regression › temporal_query Groundedness 2 1.00 0.99 +0.00 0.39 No -
Security Alerts RAG Regression › temporal_query Input Tokens 2 115618.00 107781.50 +7836.50 0.85 No -
Security Alerts RAG Regression › temporal_query Latency 2 22.10 20.00 +2.10 0.83 No -
Security Alerts RAG Regression › temporal_query Output Tokens 2 1952.50 2166.00 -213.50 0.79 No -
Security Alerts RAG Regression › temporal_query Relevance 2 0.75 0.64 +0.11 0.39 No -
Security Alerts RAG Regression › temporal_query Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Tool Calls 2 3.00 2.00 +1.00 0.39 No -
Security Alerts RAG Regression › temporal_query Trajectory 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alerts-rag-regression::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alerts-rag-regression::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (55 evaluator comparisons).

Note: 1 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (55 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Security Alerts RAG Regression › field_specific_lookup Cached Tokens 2 17391.00 40837.50 -23446.50 0.15 No -
Security Alerts RAG Regression › field_specific_lookup Factuality 2 0.26 0.22 +0.03 0.39 No -
Security Alerts RAG Regression › field_specific_lookup Groundedness 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Input Tokens 2 180810.50 200428.50 -19618.00 0.39 No -
Security Alerts RAG Regression › field_specific_lookup Latency 2 32.85 32.59 +0.26 0.93 No -
Security Alerts RAG Regression › field_specific_lookup Output Tokens 2 2967.50 3828.50 -861.00 0.44 No -
Security Alerts RAG Regression › field_specific_lookup Relevance 2 0.42 0.50 -0.08 0.39 No -
Security Alerts RAG Regression › field_specific_lookup Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Tool Calls 2 3.50 3.50 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Trajectory 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Cached Tokens 1 3719.00 18865.00 -15146.00 - n/a -
Security Alerts RAG Regression › multi_alert_correlation Factuality 2 0.22 0.19 +0.04 0.39 No -
Security Alerts RAG Regression › multi_alert_correlation Groundedness 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Input Tokens 2 42515.00 77603.50 -35088.50 0.39 No -
Security Alerts RAG Regression › multi_alert_correlation Latency 2 12.47 18.86 -6.39 0.46 No -
Security Alerts RAG Regression › multi_alert_correlation Output Tokens 2 467.00 1613.50 -1146.50 0.39 No -
Security Alerts RAG Regression › multi_alert_correlation Relevance 2 0.50 0.63 -0.13 0.39 No -
Security Alerts RAG Regression › multi_alert_correlation Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Tool Calls 2 2.50 2.50 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Trajectory 2 0.00 0.50 -0.50 0.39 No -
Security Alerts RAG Regression › single_alert_query Cached Tokens 2 224781.00 92013.00 +132768.00 0.05 No -
Security Alerts RAG Regression › single_alert_query Factuality 2 0.26 0.16 +0.09 0.35 No -
Security Alerts RAG Regression › single_alert_query Groundedness 2 0.96 0.99 -0.03 0.47 No -
Security Alerts RAG Regression › single_alert_query Input Tokens 2 601247.00 279754.00 +321493.00 0.19 No -
Security Alerts RAG Regression › single_alert_query Latency 2 74.55 45.68 +28.87 0.41 No -
Security Alerts RAG Regression › single_alert_query Output Tokens 2 6337.00 4557.00 +1780.00 0.64 No -
Security Alerts RAG Regression › single_alert_query Relevance 2 0.47 0.70 -0.22 0.32 No -
Security Alerts RAG Regression › single_alert_query Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Tool Calls 2 11.00 5.50 +5.50 0.20 No -
Security Alerts RAG Regression › single_alert_query Trajectory 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › space_isolation Cached Tokens 1 22127.00 11056.00 +11071.00 - n/a -
Security Alerts RAG Regression › space_isolation Factuality 1 1.00 1.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Groundedness 1 1.00 1.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Input Tokens 1 39591.00 39580.00 +11.00 - n/a -
Security Alerts RAG Regression › space_isolation Latency 1 12.65 11.77 +0.88 - n/a -
Security Alerts RAG Regression › space_isolation Output Tokens 1 626.00 660.00 -34.00 - n/a -
Security Alerts RAG Regression › space_isolation Relevance 1 1.00 1.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Skill Invoked (alert-analysis) 1 0.00 0.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Tool Calls 1 2.00 2.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Trajectory 1 1.00 1.00 0.00 - n/a -
Security Alerts RAG Regression › temporal_query Cached Tokens 2 90387.50 49233.50 +41154.00 0.47 No -
Security Alerts RAG Regression › temporal_query Factuality 2 0.51 0.22 +0.29 0.29 No -
Security Alerts RAG Regression › temporal_query Groundedness 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Input Tokens 2 253670.50 161013.00 +92657.50 0.52 No -
Security Alerts RAG Regression › temporal_query Latency 2 42.02 33.70 +8.32 0.72 No -
Security Alerts RAG Regression › temporal_query Output Tokens 2 3345.50 3897.00 -551.50 0.80 No -
Security Alerts RAG Regression › temporal_query Relevance 2 0.39 0.50 -0.11 0.39 No -
Security Alerts RAG Regression › temporal_query Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Tool Calls 2 5.00 4.50 +0.50 0.87 No -
Security Alerts RAG Regression › temporal_query Trajectory 2 0.50 1.00 -0.50 0.39 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alerts-rag-regression::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alerts-rag-regression::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
5 significant difference(s) detected out of 55 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Security Alerts RAG Regression › field_specific_lookup Input Tokens 2 192851.00 133377.50 +59473.50 0.02 Yes Regression
Security Alerts RAG Regression › field_specific_lookup Latency 2 28.54 21.59 +6.96 0.00 Yes Regression
Security Alerts RAG Regression › field_specific_lookup Output Tokens 2 2930.00 1271.50 +1658.50 0.03 Yes Regression
Security Alerts RAG Regression › multi_alert_correlation Latency 2 21.22 23.26 -2.04 0.01 Yes Improvement
Security Alerts RAG Regression › single_alert_query Cached Tokens 2 46120.50 34418.50 +11702.00 0.02 Yes -
No significant changes (50 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Security Alerts RAG Regression › field_specific_lookup Cached Tokens 2 70191.50 52233.00 +17958.50 0.47 No -
Security Alerts RAG Regression › field_specific_lookup Factuality 2 0.23 0.23 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Groundedness 2 0.98 1.00 -0.02 0.39 No -
Security Alerts RAG Regression › field_specific_lookup Relevance 2 0.50 0.50 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Tool Calls 2 4.00 3.00 +1.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Trajectory 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Cached Tokens 2 55575.00 24064.00 +31511.00 0.27 No -
Security Alerts RAG Regression › multi_alert_correlation Factuality 2 0.24 0.07 +0.17 0.31 No -
Security Alerts RAG Regression › multi_alert_correlation Groundedness 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Input Tokens 2 137897.00 93481.00 +44416.00 0.39 No -
Security Alerts RAG Regression › multi_alert_correlation Output Tokens 2 1586.50 1140.00 +446.50 0.42 No -
Security Alerts RAG Regression › multi_alert_correlation Relevance 2 0.47 0.71 -0.24 0.17 No -
Security Alerts RAG Regression › multi_alert_correlation Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Tool Calls 2 3.00 2.50 +0.50 0.39 No -
Security Alerts RAG Regression › multi_alert_correlation Trajectory 2 0.50 0.50 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Factuality 2 0.30 0.18 +0.12 0.42 No -
Security Alerts RAG Regression › single_alert_query Groundedness 2 0.98 0.99 -0.00 0.72 No -
Security Alerts RAG Regression › single_alert_query Input Tokens 2 141033.00 112412.50 +28620.50 0.41 No -
Security Alerts RAG Regression › single_alert_query Latency 2 22.68 23.16 -0.47 0.91 No -
Security Alerts RAG Regression › single_alert_query Output Tokens 2 1973.00 1214.00 +759.00 0.59 No -
Security Alerts RAG Regression › single_alert_query Relevance 2 0.70 0.78 -0.08 0.55 No -
Security Alerts RAG Regression › single_alert_query Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Tool Calls 2 3.00 3.00 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Trajectory 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › space_isolation Cached Tokens 1 29312.00 36352.00 -7040.00 - n/a -
Security Alerts RAG Regression › space_isolation Factuality 1 0.95 0.79 +0.15 - n/a -
Security Alerts RAG Regression › space_isolation Groundedness 1 0.98 1.00 -0.02 - n/a -
Security Alerts RAG Regression › space_isolation Input Tokens 1 38239.00 38248.00 -9.00 - n/a -
Security Alerts RAG Regression › space_isolation Latency 1 10.74 19.74 -8.99 - n/a -
Security Alerts RAG Regression › space_isolation Output Tokens 1 172.00 193.00 -21.00 - n/a -
Security Alerts RAG Regression › space_isolation Relevance 1 1.00 0.33 +0.67 - n/a -
Security Alerts RAG Regression › space_isolation Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Skill Invoked (alert-analysis) 1 0.00 0.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Tool Calls 1 2.00 2.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Trajectory 1 1.00 1.00 0.00 - n/a -
Security Alerts RAG Regression › temporal_query Cached Tokens 2 64972.50 22962.00 +42010.50 0.20 No -
Security Alerts RAG Regression › temporal_query Factuality 2 0.45 0.57 -0.12 0.50 No -
Security Alerts RAG Regression › temporal_query Groundedness 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Input Tokens 2 132275.00 111473.00 +20802.00 0.40 No -
Security Alerts RAG Regression › temporal_query Latency 2 30.07 23.15 +6.92 0.44 No -
Security Alerts RAG Regression › temporal_query Output Tokens 2 2209.50 1069.50 +1140.00 0.43 No -
Security Alerts RAG Regression › temporal_query Relevance 2 0.55 0.90 -0.35 0.19 No -
Security Alerts RAG Regression › temporal_query Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Tool Calls 2 3.00 3.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Trajectory 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-alerts-rag-regression::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alerts-rag-regression::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 55 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Security Alerts RAG Regression › single_alert_query Groundedness 2 0.74 0.83 -0.09 0.01 Yes Regression
No significant changes (54 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
Security Alerts RAG Regression › field_specific_lookup Cached Tokens 2 38579.50 28220.50 +10359.00 0.78 No -
Security Alerts RAG Regression › field_specific_lookup Factuality 2 0.55 0.47 +0.08 0.66 No -
Security Alerts RAG Regression › field_specific_lookup Groundedness 2 1.00 0.90 +0.10 0.39 No -
Security Alerts RAG Regression › field_specific_lookup Input Tokens 2 115603.00 141635.00 -26032.00 0.83 No -
Security Alerts RAG Regression › field_specific_lookup Latency 2 18.99 39.53 -20.54 0.55 No -
Security Alerts RAG Regression › field_specific_lookup Output Tokens 2 1151.50 2365.50 -1214.00 0.58 No -
Security Alerts RAG Regression › field_specific_lookup Relevance 2 0.13 0.50 -0.38 0.15 No -
Security Alerts RAG Regression › field_specific_lookup Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › field_specific_lookup Tool Calls 2 6.00 16.50 -10.50 0.60 No -
Security Alerts RAG Regression › field_specific_lookup Trajectory 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Cached Tokens 2 16550.00 12249.50 +4300.50 0.14 No -
Security Alerts RAG Regression › multi_alert_correlation Factuality 2 0.60 0.60 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Groundedness 2 0.78 0.92 -0.14 0.39 No -
Security Alerts RAG Regression › multi_alert_correlation Input Tokens 2 26449.00 30713.00 -4264.00 0.56 No -
Security Alerts RAG Regression › multi_alert_correlation Latency 2 13.06 25.83 -12.77 0.52 No -
Security Alerts RAG Regression › multi_alert_correlation Output Tokens 2 1044.00 2007.00 -963.00 0.51 No -
Security Alerts RAG Regression › multi_alert_correlation Relevance 2 0.25 0.25 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › multi_alert_correlation Tool Calls 2 4.50 11.50 -7.00 0.61 No -
Security Alerts RAG Regression › multi_alert_correlation Trajectory 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Cached Tokens 2 17780.00 26186.00 -8406.00 0.16 No -
Security Alerts RAG Regression › single_alert_query Factuality 2 0.73 0.70 +0.03 0.70 No -
Security Alerts RAG Regression › single_alert_query Input Tokens 2 27007.50 133914.00 -106906.50 0.11 No -
Security Alerts RAG Regression › single_alert_query Latency 2 46.05 42.68 +3.38 0.67 No -
Security Alerts RAG Regression › single_alert_query Output Tokens 2 2241.50 2719.00 -477.50 0.49 No -
Security Alerts RAG Regression › single_alert_query Relevance 2 0.42 0.25 +0.17 0.61 No -
Security Alerts RAG Regression › single_alert_query Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › single_alert_query Tool Calls 2 8.50 10.50 -2.00 0.22 No -
Security Alerts RAG Regression › single_alert_query Trajectory 2 0.00 0.50 -0.50 0.39 No -
Security Alerts RAG Regression › space_isolation Cached Tokens 1 8271.00 11422.00 -3151.00 - n/a -
Security Alerts RAG Regression › space_isolation Factuality 1 0.50 0.50 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Groundedness 1 0.79 0.66 +0.13 - n/a -
Security Alerts RAG Regression › space_isolation Input Tokens 1 27054.00 24575.00 +2479.00 - n/a -
Security Alerts RAG Regression › space_isolation Latency 1 13.64 4.73 +8.91 - n/a -
Security Alerts RAG Regression › space_isolation Output Tokens 1 399.00 497.00 -98.00 - n/a -
Security Alerts RAG Regression › space_isolation Relevance 1 0.00 0.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Skill Invoked (alert-analysis) 1 0.00 0.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Tool Calls 1 1.00 1.00 0.00 - n/a -
Security Alerts RAG Regression › space_isolation Trajectory 1 0.00 0.00 0.00 - n/a -
Security Alerts RAG Regression › temporal_query Cached Tokens 2 17521.50 23204.00 -5682.50 0.59 No -
Security Alerts RAG Regression › temporal_query Factuality 2 0.79 0.45 +0.34 0.13 No -
Security Alerts RAG Regression › temporal_query Groundedness 2 0.82 0.35 +0.47 0.21 No -
Security Alerts RAG Regression › temporal_query Input Tokens 2 98135.50 71447.00 +26688.50 0.50 No -
Security Alerts RAG Regression › temporal_query Latency 2 25.59 36.03 -10.43 0.70 No -
Security Alerts RAG Regression › temporal_query Output Tokens 2 1401.00 2562.00 -1161.00 0.55 No -
Security Alerts RAG Regression › temporal_query Relevance 2 0.67 0.55 +0.12 0.74 No -
Security Alerts RAG Regression › temporal_query Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Skill Invoked (alert-analysis) 2 0.00 0.00 0.00 1.00 No -
Security Alerts RAG Regression › temporal_query Tool Calls 2 8.50 2.00 +6.50 0.39 No -
Security Alerts RAG Regression › temporal_query Trajectory 2 0.00 0.00 0.00 1.00 No -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::alerting-v2::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::alerting-v2::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (28 evaluator comparisons).

Note: 50 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (28 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
alerting-v2: no-data strategy updates Criteria 6 1.00 1.00 0.00 1.00 No -
alerting-v2: no-data strategy updates ExpectedAttachmentData 6 0.67 0.83 -0.17 0.32 No -
alerting-v2: no-data strategy updates ExpectedRenderAttachment 6 1.00 1.00 0.00 1.00 No -
alerting-v2: no-data strategy updates ExpectedSkill 6 1.00 1.00 0.00 1.00 No -
alerting-v2: no-data strategy updates ExpectedToolCalled 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates Criteria 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedAttachmentData 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedRenderAttachment 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedSkill 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedToolCalled 6 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition Criteria 3 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedAnyOfToolIds 1 0.00 0.00 0.00 - n/a -
alerting-v2: rule composition ExpectedAttachmentData 3 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedRenderAttachment 3 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedSkill 3 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedToolCalled 3 1.00 1.00 0.00 1.00 No -
alerting-v2: rule-management notification setup Criteria 1 1.00 1.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedAttachmentData 1 1.00 1.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedRenderAttachment 1 1.00 1.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedSkill 1 1.00 1.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedToolCalled 1 1.00 1.00 0.00 - n/a -
alerting-v2: runbook composition Criteria 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedAttachmentData 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedRenderAttachment 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedSkill 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedToolCalled 2 1.00 1.00 0.00 1.00 No -
alerting-v2: skill routing Criteria 2 1.00 1.00 0.00 1.00 No -
alerting-v2: skill routing ExpectedSkill 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::alerting-v2::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::alerting-v2::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (28 evaluator comparisons).

Note: 50 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (28 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
alerting-v2: no-data strategy updates Criteria 6 1.00 1.00 0.00 1.00 No -
alerting-v2: no-data strategy updates ExpectedAttachmentData 6 0.83 0.67 +0.17 0.32 No -
alerting-v2: no-data strategy updates ExpectedRenderAttachment 6 1.00 1.00 0.00 1.00 No -
alerting-v2: no-data strategy updates ExpectedSkill 6 1.00 1.00 0.00 1.00 No -
alerting-v2: no-data strategy updates ExpectedToolCalled 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates Criteria 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedAttachmentData 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedRenderAttachment 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedSkill 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedToolCalled 6 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition Criteria 3 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedAnyOfToolIds 1 1.00 1.00 0.00 - n/a -
alerting-v2: rule composition ExpectedAttachmentData 3 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedRenderAttachment 3 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedSkill 3 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedToolCalled 3 1.00 1.00 0.00 1.00 No -
alerting-v2: rule-management notification setup Criteria 1 1.00 1.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedAttachmentData 1 1.00 1.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedRenderAttachment 1 1.00 1.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedSkill 1 1.00 1.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedToolCalled 1 1.00 1.00 0.00 - n/a -
alerting-v2: runbook composition Criteria 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedAttachmentData 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedRenderAttachment 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedSkill 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedToolCalled 2 1.00 1.00 0.00 1.00 No -
alerting-v2: skill routing Criteria 2 1.00 1.00 0.00 1.00 No -
alerting-v2: skill routing ExpectedSkill 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::alerting-v2::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::alerting-v2::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (28 evaluator comparisons).

Note: 52 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (28 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
alerting-v2: no-data strategy updates Criteria 6 0.78 0.79 -0.01 0.85 No -
alerting-v2: no-data strategy updates ExpectedAttachmentData 6 1.00 0.83 +0.17 0.32 No -
alerting-v2: no-data strategy updates ExpectedRenderAttachment 6 1.00 1.00 0.00 1.00 No -
alerting-v2: no-data strategy updates ExpectedSkill 6 1.00 1.00 0.00 1.00 No -
alerting-v2: no-data strategy updates ExpectedToolCalled 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates Criteria 6 0.85 0.90 -0.06 0.58 No -
alerting-v2: recovery strategy updates ExpectedAttachmentData 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedRenderAttachment 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedSkill 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedToolCalled 6 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition Criteria 4 0.95 0.95 0.00 1.00 No -
alerting-v2: rule composition ExpectedAnyOfToolIds 1 1.00 1.00 0.00 - n/a -
alerting-v2: rule composition ExpectedAttachmentData 4 1.00 0.75 +0.25 0.33 No -
alerting-v2: rule composition ExpectedRenderAttachment 4 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedSkill 4 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedToolCalled 4 1.00 1.00 0.00 1.00 No -
alerting-v2: rule-management notification setup Criteria 1 1.00 1.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedAttachmentData 1 0.00 0.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedRenderAttachment 1 1.00 1.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedSkill 1 1.00 1.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedToolCalled 1 1.00 1.00 0.00 - n/a -
alerting-v2: runbook composition Criteria 2 0.88 0.88 0.00 1.00 No -
alerting-v2: runbook composition ExpectedAttachmentData 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedRenderAttachment 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedSkill 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedToolCalled 2 1.00 1.00 0.00 1.00 No -
alerting-v2: skill routing Criteria 2 1.00 1.00 0.00 1.00 No -
alerting-v2: skill routing ExpectedSkill 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::alerting-v2::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::alerting-v2::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 28 comparisons.

Note: 52 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
alerting-v2: rule composition Criteria 4 1.00 0.84 +0.16 0.04 Yes Improvement
No significant changes (27 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
alerting-v2: no-data strategy updates Criteria 6 0.88 0.82 +0.06 0.51 No -
alerting-v2: no-data strategy updates ExpectedAttachmentData 6 0.67 0.83 -0.17 0.32 No -
alerting-v2: no-data strategy updates ExpectedRenderAttachment 6 1.00 1.00 0.00 1.00 No -
alerting-v2: no-data strategy updates ExpectedSkill 6 1.00 1.00 0.00 1.00 No -
alerting-v2: no-data strategy updates ExpectedToolCalled 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates Criteria 6 0.75 0.90 -0.15 0.19 No -
alerting-v2: recovery strategy updates ExpectedAttachmentData 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedRenderAttachment 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedSkill 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedToolCalled 6 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedAnyOfToolIds 1 0.00 1.00 -1.00 - n/a -
alerting-v2: rule composition ExpectedAttachmentData 4 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedRenderAttachment 4 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedSkill 4 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedToolCalled 4 0.75 1.00 -0.25 0.33 No -
alerting-v2: rule-management notification setup Criteria 1 0.67 0.33 +0.33 - n/a -
alerting-v2: rule-management notification setup ExpectedAttachmentData 1 0.00 0.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedRenderAttachment 1 1.00 0.00 +1.00 - n/a -
alerting-v2: rule-management notification setup ExpectedSkill 1 1.00 0.00 +1.00 - n/a -
alerting-v2: rule-management notification setup ExpectedToolCalled 1 1.00 0.00 +1.00 - n/a -
alerting-v2: runbook composition Criteria 2 0.88 1.00 -0.13 0.39 No -
alerting-v2: runbook composition ExpectedAttachmentData 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedRenderAttachment 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedSkill 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedToolCalled 2 1.00 1.00 0.00 1.00 No -
alerting-v2: skill routing Criteria 2 1.00 1.00 0.00 1.00 No -
alerting-v2: skill routing ExpectedSkill 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::alerting-v2::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::alerting-v2::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (28 evaluator comparisons).

Note: 5 unpaired (ran in only one experiment), 51 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (28 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
alerting-v2: no-data strategy updates Criteria 6 1.00 0.96 +0.04 0.32 No -
alerting-v2: no-data strategy updates ExpectedAttachmentData 6 0.83 1.00 -0.17 0.32 No -
alerting-v2: no-data strategy updates ExpectedRenderAttachment 6 1.00 1.00 0.00 1.00 No -
alerting-v2: no-data strategy updates ExpectedSkill 6 1.00 1.00 0.00 1.00 No -
alerting-v2: no-data strategy updates ExpectedToolCalled 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates Criteria 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedAttachmentData 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedRenderAttachment 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedSkill 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedToolCalled 6 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition Criteria 3 0.92 1.00 -0.08 0.35 No -
alerting-v2: rule composition ExpectedAnyOfToolIds 1 1.00 1.00 0.00 - n/a -
alerting-v2: rule composition ExpectedAttachmentData 3 0.67 1.00 -0.33 0.35 No -
alerting-v2: rule composition ExpectedRenderAttachment 3 0.67 1.00 -0.33 0.35 No -
alerting-v2: rule composition ExpectedSkill 3 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedToolCalled 3 1.00 1.00 0.00 1.00 No -
alerting-v2: rule-management notification setup Criteria 1 1.00 0.33 +0.67 - n/a -
alerting-v2: rule-management notification setup ExpectedAttachmentData 1 0.00 0.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedRenderAttachment 1 1.00 0.00 +1.00 - n/a -
alerting-v2: rule-management notification setup ExpectedSkill 1 1.00 0.00 +1.00 - n/a -
alerting-v2: rule-management notification setup ExpectedToolCalled 1 1.00 0.00 +1.00 - n/a -
alerting-v2: runbook composition Criteria 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedAttachmentData 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedRenderAttachment 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedSkill 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedToolCalled 2 1.00 1.00 0.00 1.00 No -
alerting-v2: skill routing Criteria 2 1.00 1.00 0.00 1.00 No -
alerting-v2: skill routing ExpectedSkill 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::alerting-v2::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::alerting-v2::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (28 evaluator comparisons).

Note: 52 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (28 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
alerting-v2: no-data strategy updates Criteria 6 0.31 0.51 -0.21 0.11 No -
alerting-v2: no-data strategy updates ExpectedAttachmentData 6 0.00 0.33 -0.33 0.14 No -
alerting-v2: no-data strategy updates ExpectedRenderAttachment 6 0.00 0.50 -0.50 0.06 No -
alerting-v2: no-data strategy updates ExpectedSkill 6 1.00 1.00 0.00 1.00 No -
alerting-v2: no-data strategy updates ExpectedToolCalled 6 0.67 0.67 0.00 1.00 No -
alerting-v2: recovery strategy updates Criteria 6 0.31 0.35 -0.04 0.84 No -
alerting-v2: recovery strategy updates ExpectedAttachmentData 6 0.00 0.33 -0.33 0.14 No -
alerting-v2: recovery strategy updates ExpectedRenderAttachment 6 0.00 0.33 -0.33 0.14 No -
alerting-v2: recovery strategy updates ExpectedSkill 6 1.00 1.00 0.00 1.00 No -
alerting-v2: recovery strategy updates ExpectedToolCalled 6 0.67 0.67 0.00 1.00 No -
alerting-v2: rule composition Criteria 4 0.53 0.68 -0.15 0.62 No -
alerting-v2: rule composition ExpectedAnyOfToolIds 1 0.00 0.00 0.00 - n/a -
alerting-v2: rule composition ExpectedAttachmentData 4 0.25 0.50 -0.25 0.59 No -
alerting-v2: rule composition ExpectedRenderAttachment 4 0.25 0.50 -0.25 0.59 No -
alerting-v2: rule composition ExpectedSkill 4 1.00 1.00 0.00 1.00 No -
alerting-v2: rule composition ExpectedToolCalled 4 0.25 0.50 -0.25 0.59 No -
alerting-v2: rule-management notification setup Criteria 1 0.33 0.50 -0.17 - n/a -
alerting-v2: rule-management notification setup ExpectedAttachmentData 1 0.00 0.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedRenderAttachment 1 0.00 0.00 0.00 - n/a -
alerting-v2: rule-management notification setup ExpectedSkill 1 0.00 1.00 -1.00 - n/a -
alerting-v2: rule-management notification setup ExpectedToolCalled 1 0.00 0.00 0.00 - n/a -
alerting-v2: runbook composition Criteria 2 0.50 0.75 -0.25 0.39 No -
alerting-v2: runbook composition ExpectedAttachmentData 2 0.00 1.00 -1.00 1.00 No -
alerting-v2: runbook composition ExpectedRenderAttachment 2 0.00 1.00 -1.00 1.00 No -
alerting-v2: runbook composition ExpectedSkill 2 1.00 1.00 0.00 1.00 No -
alerting-v2: runbook composition ExpectedToolCalled 2 0.00 1.00 -1.00 1.00 No -
alerting-v2: skill routing Criteria 2 1.00 1.00 0.00 1.00 No -
alerting-v2: skill routing ExpectedSkill 2 1.00 1.00 0.00 1.00 No -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-persona-matrix::anthropic-claude-4.6-opus | Baseline (main): sweep-1788679167-rja-s1of3::security-persona-matrix::anthropic-claude-4.6-opus
Baseline: commit f94843e, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (7 evaluator comparisons).

Note: 217 unpaired (ran in only one experiment), 43 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (7 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security: security-persona-matrix criteria 7 0.93 0.93 0.00 1.00 No -
security: security-persona-matrix ExpectedToolCalled 7 0.57 0.43 +0.14 0.67 No -
security: security-persona-matrix Factuality 7 0.25 0.47 -0.22 0.22 No -
security: security-persona-matrix Groundedness 7 0.64 0.95 -0.32 0.07 No -
security: security-persona-matrix Relevance 7 0.53 0.66 -0.13 0.23 No -
security: security-persona-matrix Sequence Accuracy 7 1.00 1.00 0.00 1.00 No -
security: security-persona-matrix Trajectory 3 0.50 0.83 -0.33 0.35 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-persona-matrix::anthropic-claude-4.6-sonnet | Baseline (main): sweep-1788679167-rja-s2of3::security-persona-matrix::anthropic-claude-4.6-sonnet
Baseline: commit f94843e, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 7 comparisons.

Note: 217 unpaired (ran in only one experiment), 43 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security: security-persona-matrix Factuality 7 0.27 0.64 -0.36 0.04 Yes Regression
No significant changes (6 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security: security-persona-matrix criteria 7 0.93 1.00 -0.07 0.15 No -
security: security-persona-matrix ExpectedToolCalled 7 0.57 0.71 -0.14 0.58 No -
security: security-persona-matrix Groundedness 7 0.78 0.96 -0.18 0.21 No -
security: security-persona-matrix Relevance 7 0.51 0.64 -0.14 0.43 No -
security: security-persona-matrix Sequence Accuracy 7 1.00 1.00 0.00 1.00 No -
security: security-persona-matrix Trajectory 3 0.50 0.83 -0.33 0.35 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-persona-matrix::google-gemini-3.0-flash | Baseline (main): sweep-1788685888-rj3-s1of3::security-persona-matrix::google-gemini-3.0-flash
Baseline: commit f94843e, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (7 evaluator comparisons).

Note: 217 unpaired (ran in only one experiment), 43 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (7 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security: security-persona-matrix criteria 7 0.86 0.93 -0.07 0.60 No -
security: security-persona-matrix ExpectedToolCalled 7 0.57 0.57 0.00 1.00 No -
security: security-persona-matrix Factuality 7 0.18 0.46 -0.29 0.16 No -
security: security-persona-matrix Groundedness 7 0.80 0.97 -0.17 0.25 No -
security: security-persona-matrix Relevance 7 0.58 0.82 -0.24 0.21 No -
security: security-persona-matrix Sequence Accuracy 7 1.00 1.00 0.00 1.00 No -
security: security-persona-matrix Trajectory 3 0.50 0.83 -0.33 0.35 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-persona-matrix::google-gemini-3.1-pro | Baseline (main): sweep-1788685888-rj3-s2of3::security-persona-matrix::google-gemini-3.1-pro
Baseline: commit f94843e, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 7 comparisons.

Note: 217 unpaired (ran in only one experiment), 43 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security: security-persona-matrix Factuality 7 0.06 0.45 -0.39 0.02 Yes Regression
No significant changes (6 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security: security-persona-matrix criteria 7 0.89 0.93 -0.04 0.58 No -
security: security-persona-matrix ExpectedToolCalled 7 0.43 0.43 0.00 1.00 No -
security: security-persona-matrix Groundedness 7 0.84 0.98 -0.13 0.35 No -
security: security-persona-matrix Relevance 7 0.63 0.67 -0.04 0.73 No -
security: security-persona-matrix Sequence Accuracy 7 1.00 0.95 +0.05 0.32 No -
security: security-persona-matrix Trajectory 3 0.17 0.50 -0.33 0.35 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-persona-matrix::openai-gpt-5.4 | Baseline (main): sweep-1788679167-rja-s2of3::security-persona-matrix::openai-gpt-5.4
Baseline: commit f94843e, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (7 evaluator comparisons).

Note: 217 unpaired (ran in only one experiment), 43 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (7 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security: security-persona-matrix criteria 7 0.89 0.86 +0.04 0.79 No -
security: security-persona-matrix ExpectedToolCalled 7 0.57 0.29 +0.29 0.42 No -
security: security-persona-matrix Factuality 7 0.29 0.59 -0.30 0.13 No -
security: security-persona-matrix Groundedness 7 0.78 0.99 -0.21 0.18 No -
security: security-persona-matrix Relevance 7 0.59 0.59 -0.01 0.97 No -
security: security-persona-matrix Sequence Accuracy 7 1.00 1.00 0.00 1.00 No -
security: security-persona-matrix Trajectory 3 0.50 0.50 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-persona-matrix::openai-gpt-oss-120b | Baseline (main): bk-019fdc33-33ac-4468-813f-ec408a156454::security-persona-matrix::openai-gpt-oss-120b
Baseline: commit a9de85a, 35 days ago

Warning: Baseline is 35 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 12 comparisons.

Note: 1 unpaired (ran in only one experiment), 12 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security: security-persona-matrix Tool Calls 21 1.48 5.95 -4.48 0.01 Yes -
No significant changes (11 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
security: security-persona-matrix criteria 21 0.54 0.52 +0.01 0.89 No -
security: security-persona-matrix ExpectedToolCalled 21 0.05 0.19 -0.14 0.08 No -
security: security-persona-matrix Factuality 21 0.52 0.47 +0.04 0.65 No -
security: security-persona-matrix Groundedness 21 0.85 0.77 +0.08 0.35 No -
security: security-persona-matrix Input Tokens 21 34984.52 43132.48 -8147.95 0.30 No -
security: security-persona-matrix Latency 21 33.44 25.92 +7.52 0.23 No -
security: security-persona-matrix Output Tokens 21 2335.48 2443.38 -107.90 0.74 No -
security: security-persona-matrix Relevance 21 0.59 0.72 -0.14 0.09 No -
security: security-persona-matrix Sequence Accuracy 21 1.00 1.00 0.00 1.00 No -
security: security-persona-matrix SkillInvoked 13 0.00 0.00 0.00 1.00 No -
security: security-persona-matrix Trajectory 21 0.02 0.09 -0.06 0.08 No -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::siem-readiness::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::siem-readiness::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (47 evaluator comparisons).

Note: 13 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (47 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
siem-readiness: blast radius accuracy Cached Tokens 1 109982.00 216826.00 -106844.00 - n/a -
siem-readiness: blast radius accuracy Input Tokens 1 638031.00 320578.00 +317453.00 - n/a -
siem-readiness: blast radius accuracy Latency 1 19.93 7.34 +12.59 - n/a -
siem-readiness: blast radius accuracy Output Tokens 1 8649.00 3396.00 +5253.00 - n/a -
siem-readiness: blast radius accuracy SIEM Readiness Criteria 1 0.18 0.27 -0.09 - n/a -
siem-readiness: blast radius accuracy Tool Calls 1 23.00 33.00 -10.00 - n/a -
siem-readiness: continuity critical pipeline failure Input Tokens 1 20610.00 48086.00 -27476.00 - n/a -
siem-readiness: continuity critical pipeline failure Latency 1 2.45 4.21 -1.76 - n/a -
siem-readiness: continuity critical pipeline failure Output Tokens 1 148.00 253.00 -105.00 - n/a -
siem-readiness: continuity critical pipeline failure SIEM Readiness Criteria 1 0.18 0.18 0.00 - n/a -
siem-readiness: continuity critical pipeline failure Tool Calls 1 1.00 2.00 -1.00 - n/a -
siem-readiness: coverage missing Application/SaaS Input Tokens 1 46381.00 81863.00 -35482.00 - n/a -
siem-readiness: coverage missing Application/SaaS Latency 1 4.42 6.00 -1.58 - n/a -
siem-readiness: coverage missing Application/SaaS Output Tokens 1 265.00 462.00 -197.00 - n/a -
siem-readiness: coverage missing Application/SaaS SIEM Readiness Criteria 1 0.40 0.30 +0.10 - n/a -
siem-readiness: coverage missing Application/SaaS Tool Calls 1 3.00 6.00 -3.00 - n/a -
siem-readiness: dimension-scoped continuity Input Tokens 1 48296.00 48114.00 +182.00 - n/a -
siem-readiness: dimension-scoped continuity Latency 1 3.69 3.76 -0.07 - n/a -
siem-readiness: dimension-scoped continuity Output Tokens 1 242.00 247.00 -5.00 - n/a -
siem-readiness: dimension-scoped continuity SIEM Readiness Criteria 1 0.55 0.45 +0.09 - n/a -
siem-readiness: dimension-scoped continuity Tool Calls 1 2.00 2.00 0.00 - n/a -
siem-readiness: four-section format Input Tokens 1 81011.00 290441.00 -209430.00 - n/a -
siem-readiness: four-section format Latency 1 4.73 36.31 -31.58 - n/a -
siem-readiness: four-section format Output Tokens 1 539.00 5297.00 -4758.00 - n/a -
siem-readiness: four-section format SIEM Readiness Criteria 1 0.25 0.08 +0.17 - n/a -
siem-readiness: four-section format Tool Calls 1 7.00 29.00 -22.00 - n/a -
siem-readiness: full report Cached Tokens 1 232426.00 16269.00 +216157.00 - n/a -
siem-readiness: full report Input Tokens 1 285170.00 210846.00 +74324.00 - n/a -
siem-readiness: full report Latency 1 6.36 6.28 +0.08 - n/a -
siem-readiness: full report Output Tokens 1 2788.00 1191.00 +1597.00 - n/a -
siem-readiness: full report SIEM Readiness Criteria 1 0.25 0.17 +0.08 - n/a -
siem-readiness: full report Tool Calls 1 32.00 14.00 +18.00 - n/a -
siem-readiness: noData for non-existent index Input Tokens 1 20654.00 48145.00 -27491.00 - n/a -
siem-readiness: noData for non-existent index Latency 1 3.23 4.09 -0.86 - n/a -
siem-readiness: noData for non-existent index Output Tokens 1 146.00 261.00 -115.00 - n/a -
siem-readiness: noData for non-existent index SIEM Readiness Criteria 1 0.60 0.60 0.00 - n/a -
siem-readiness: noData for non-existent index Tool Calls 1 1.00 2.00 -1.00 - n/a -
siem-readiness: quality ECS incompatibility Input Tokens 1 48295.00 78372.00 -30077.00 - n/a -
siem-readiness: quality ECS incompatibility Latency 1 3.81 6.28 -2.47 - n/a -
siem-readiness: quality ECS incompatibility Output Tokens 1 268.00 511.00 -243.00 - n/a -
siem-readiness: quality ECS incompatibility SIEM Readiness Criteria 1 0.36 0.18 +0.18 - n/a -
siem-readiness: quality ECS incompatibility Tool Calls 1 2.00 4.00 -2.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Input Tokens 1 48587.00 77817.00 -29230.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Latency 1 3.98 4.54 -0.56 - n/a -
siem-readiness: retention FedRAMP non-compliance Output Tokens 1 337.00 346.00 -9.00 - n/a -
siem-readiness: retention FedRAMP non-compliance SIEM Readiness Criteria 1 0.36 0.36 0.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Tool Calls 1 3.00 3.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::siem-readiness::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::siem-readiness::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (45 evaluator comparisons).

Note: 1 unpaired (ran in only one experiment), 15 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (45 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
siem-readiness: blast radius accuracy Input Tokens 1 41067.00 677814.00 -636747.00 - n/a -
siem-readiness: blast radius accuracy Latency 1 2.50 10.06 -7.56 - n/a -
siem-readiness: blast radius accuracy Output Tokens 1 346.00 6498.00 -6152.00 - n/a -
siem-readiness: blast radius accuracy SIEM Readiness Criteria 1 0.64 0.27 +0.36 - n/a -
siem-readiness: blast radius accuracy Tool Calls 1 4.00 26.00 -22.00 - n/a -
siem-readiness: continuity critical pipeline failure Input Tokens 1 47752.00 47557.00 +195.00 - n/a -
siem-readiness: continuity critical pipeline failure Latency 1 2.30 2.36 -0.06 - n/a -
siem-readiness: continuity critical pipeline failure Output Tokens 1 232.00 238.00 -6.00 - n/a -
siem-readiness: continuity critical pipeline failure SIEM Readiness Criteria 1 0.27 0.18 +0.09 - n/a -
siem-readiness: continuity critical pipeline failure Tool Calls 1 2.00 2.00 0.00 - n/a -
siem-readiness: coverage missing Application/SaaS Input Tokens 1 45985.00 81159.00 -35174.00 - n/a -
siem-readiness: coverage missing Application/SaaS Latency 1 2.48 2.99 -0.51 - n/a -
siem-readiness: coverage missing Application/SaaS Output Tokens 1 267.00 424.00 -157.00 - n/a -
siem-readiness: coverage missing Application/SaaS SIEM Readiness Criteria 1 0.30 0.40 -0.10 - n/a -
siem-readiness: coverage missing Application/SaaS Tool Calls 1 4.00 6.00 -2.00 - n/a -
siem-readiness: dimension-scoped continuity Input Tokens 1 20461.00 47587.00 -27126.00 - n/a -
siem-readiness: dimension-scoped continuity Latency 1 1.45 2.54 -1.09 - n/a -
siem-readiness: dimension-scoped continuity Output Tokens 1 133.00 235.00 -102.00 - n/a -
siem-readiness: dimension-scoped continuity SIEM Readiness Criteria 1 0.55 0.55 0.00 - n/a -
siem-readiness: dimension-scoped continuity Tool Calls 1 1.00 2.00 -1.00 - n/a -
siem-readiness: four-section format Input Tokens 1 377136.00 589273.00 -212137.00 - n/a -
siem-readiness: four-section format Latency 1 10.58 6.52 +4.07 - n/a -
siem-readiness: four-section format Output Tokens 1 3815.00 2391.00 +1424.00 - n/a -
siem-readiness: four-section format SIEM Readiness Criteria 1 0.33 0.33 0.00 - n/a -
siem-readiness: four-section format Tool Calls 1 17.00 22.00 -5.00 - n/a -
siem-readiness: full report Input Tokens 1 294480.00 241560.00 +52920.00 - n/a -
siem-readiness: full report Latency 1 5.58 3.42 +2.17 - n/a -
siem-readiness: full report Output Tokens 1 2101.00 2193.00 -92.00 - n/a -
siem-readiness: full report SIEM Readiness Criteria 1 0.33 0.25 +0.08 - n/a -
siem-readiness: full report Tool Calls 1 13.00 25.00 -12.00 - n/a -
siem-readiness: noData for non-existent index Input Tokens 1 47825.00 20257.00 +27568.00 - n/a -
siem-readiness: noData for non-existent index Latency 1 2.65 1.46 +1.19 - n/a -
siem-readiness: noData for non-existent index Output Tokens 1 265.00 136.00 +129.00 - n/a -
siem-readiness: noData for non-existent index SIEM Readiness Criteria 1 0.60 0.60 0.00 - n/a -
siem-readiness: noData for non-existent index Tool Calls 1 2.00 1.00 +1.00 - n/a -
siem-readiness: quality ECS incompatibility Input Tokens 1 47798.00 47588.00 +210.00 - n/a -
siem-readiness: quality ECS incompatibility Latency 1 4.71 2.39 +2.32 - n/a -
siem-readiness: quality ECS incompatibility Output Tokens 1 231.00 245.00 -14.00 - n/a -
siem-readiness: quality ECS incompatibility SIEM Readiness Criteria 1 0.27 0.27 0.00 - n/a -
siem-readiness: quality ECS incompatibility Tool Calls 1 2.00 2.00 0.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Input Tokens 1 76830.00 77223.00 -393.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Latency 1 2.62 2.52 +0.10 - n/a -
siem-readiness: retention FedRAMP non-compliance Output Tokens 1 306.00 378.00 -72.00 - n/a -
siem-readiness: retention FedRAMP non-compliance SIEM Readiness Criteria 1 0.45 0.45 0.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Tool Calls 1 3.00 3.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::siem-readiness::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::siem-readiness::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (49 evaluator comparisons).

Note: 2 unpaired (ran in only one experiment), 6 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (49 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
siem-readiness: blast radius accuracy Cached Tokens 1 614478.00 717400.00 -102922.00 - n/a -
siem-readiness: blast radius accuracy Input Tokens 1 959730.00 858766.00 +100964.00 - n/a -
siem-readiness: blast radius accuracy Latency 1 14.74 14.66 +0.07 - n/a -
siem-readiness: blast radius accuracy Output Tokens 1 6735.00 2309.00 +4426.00 - n/a -
siem-readiness: blast radius accuracy SIEM Readiness Criteria 1 0.27 0.18 +0.09 - n/a -
siem-readiness: blast radius accuracy Tool Calls 1 56.00 60.00 -4.00 - n/a -
siem-readiness: continuity critical pipeline failure Input Tokens 1 38536.00 3943.00 +34593.00 - n/a -
siem-readiness: continuity critical pipeline failure Latency 1 2.04 1.55 +0.49 - n/a -
siem-readiness: continuity critical pipeline failure Output Tokens 1 155.00 66.00 +89.00 - n/a -
siem-readiness: continuity critical pipeline failure SIEM Readiness Criteria 1 0.45 0.27 +0.18 - n/a -
siem-readiness: continuity critical pipeline failure Tool Calls 1 2.00 0.00 +2.00 - n/a -
siem-readiness: coverage missing Application/SaaS Input Tokens 1 37462.00 291585.00 -254123.00 - n/a -
siem-readiness: coverage missing Application/SaaS Latency 1 2.10 7.85 -5.75 - n/a -
siem-readiness: coverage missing Application/SaaS Output Tokens 1 129.00 1460.00 -1331.00 - n/a -
siem-readiness: coverage missing Application/SaaS SIEM Readiness Criteria 1 0.30 0.50 -0.20 - n/a -
siem-readiness: coverage missing Application/SaaS Tool Calls 1 3.00 18.00 -15.00 - n/a -
siem-readiness: dimension-scoped continuity Input Tokens 1 4222.00 38434.00 -34212.00 - n/a -
siem-readiness: dimension-scoped continuity Latency 1 1.60 2.35 -0.75 - n/a -
siem-readiness: dimension-scoped continuity Output Tokens 1 66.00 273.00 -207.00 - n/a -
siem-readiness: dimension-scoped continuity SIEM Readiness Criteria 1 0.45 0.55 -0.09 - n/a -
siem-readiness: dimension-scoped continuity Tool Calls 1 0.00 4.00 -4.00 - n/a -
siem-readiness: four-section format Cached Tokens 1 10222.00 10682.00 -460.00 - n/a -
siem-readiness: four-section format Input Tokens 1 117990.00 158124.00 -40134.00 - n/a -
siem-readiness: four-section format Latency 1 14.52 8.54 +5.98 - n/a -
siem-readiness: four-section format Output Tokens 1 2122.00 968.00 +1154.00 - n/a -
siem-readiness: four-section format SIEM Readiness Criteria 1 0.50 0.17 +0.33 - n/a -
siem-readiness: four-section format Tool Calls 1 9.00 19.00 -10.00 - n/a -
siem-readiness: full report Cached Tokens 1 354242.00 423682.00 -69440.00 - n/a -
siem-readiness: full report Input Tokens 1 528862.00 639216.00 -110354.00 - n/a -
siem-readiness: full report Latency 1 11.84 9.44 +2.40 - n/a -
siem-readiness: full report Output Tokens 1 1549.00 2578.00 -1029.00 - n/a -
siem-readiness: full report SIEM Readiness Criteria 1 0.25 0.17 +0.08 - n/a -
siem-readiness: full report Tool Calls 1 32.00 41.00 -9.00 - n/a -
siem-readiness: noData for non-existent index Input Tokens 1 38620.00 38515.00 +105.00 - n/a -
siem-readiness: noData for non-existent index Latency 1 2.27 2.57 -0.29 - n/a -
siem-readiness: noData for non-existent index Output Tokens 1 223.00 348.00 -125.00 - n/a -
siem-readiness: noData for non-existent index SIEM Readiness Criteria 1 0.70 0.60 +0.10 - n/a -
siem-readiness: noData for non-existent index Tool Calls 1 2.00 4.00 -2.00 - n/a -
siem-readiness: quality ECS incompatibility Cached Tokens 1 14999.00 7331.00 +7668.00 - n/a -
siem-readiness: quality ECS incompatibility Input Tokens 1 75221.00 75267.00 -46.00 - n/a -
siem-readiness: quality ECS incompatibility Latency 1 2.50 3.27 -0.78 - n/a -
siem-readiness: quality ECS incompatibility Output Tokens 1 188.00 417.00 -229.00 - n/a -
siem-readiness: quality ECS incompatibility SIEM Readiness Criteria 1 0.27 0.27 0.00 - n/a -
siem-readiness: quality ECS incompatibility Tool Calls 1 4.00 6.00 -2.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Input Tokens 1 4248.00 38729.00 -34481.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Latency 1 1.69 2.51 -0.81 - n/a -
siem-readiness: retention FedRAMP non-compliance Output Tokens 1 94.00 390.00 -296.00 - n/a -
siem-readiness: retention FedRAMP non-compliance SIEM Readiness Criteria 1 0.45 0.55 -0.09 - n/a -
siem-readiness: retention FedRAMP non-compliance Tool Calls 1 0.00 4.00 -4.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::siem-readiness::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::siem-readiness::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (48 evaluator comparisons).

Note: 4 unpaired (ran in only one experiment), 6 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (48 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
siem-readiness: blast radius accuracy Cached Tokens 1 472112.00 289715.00 +182397.00 - n/a -
siem-readiness: blast radius accuracy Input Tokens 1 745357.00 487106.00 +258251.00 - n/a -
siem-readiness: blast radius accuracy Latency 1 13.69 13.02 +0.67 - n/a -
siem-readiness: blast radius accuracy Output Tokens 1 14030.00 9700.00 +4330.00 - n/a -
siem-readiness: blast radius accuracy SIEM Readiness Criteria 1 0.27 0.36 -0.09 - n/a -
siem-readiness: blast radius accuracy Tool Calls 1 47.00 29.00 +18.00 - n/a -
siem-readiness: continuity critical pipeline failure Input Tokens 1 170396.00 38353.00 +132043.00 - n/a -
siem-readiness: continuity critical pipeline failure Latency 1 3.87 5.09 -1.22 - n/a -
siem-readiness: continuity critical pipeline failure Output Tokens 1 822.00 497.00 +325.00 - n/a -
siem-readiness: continuity critical pipeline failure SIEM Readiness Criteria 1 0.45 0.18 +0.27 - n/a -
siem-readiness: continuity critical pipeline failure Tool Calls 1 6.00 2.00 +4.00 - n/a -
siem-readiness: coverage missing Application/SaaS Input Tokens 1 98868.00 41087.00 +57781.00 - n/a -
siem-readiness: coverage missing Application/SaaS Latency 1 6.17 5.21 +0.96 - n/a -
siem-readiness: coverage missing Application/SaaS Output Tokens 1 1255.00 484.00 +771.00 - n/a -
siem-readiness: coverage missing Application/SaaS SIEM Readiness Criteria 1 0.30 0.30 0.00 - n/a -
siem-readiness: coverage missing Application/SaaS Tool Calls 1 5.00 3.00 +2.00 - n/a -
siem-readiness: dimension-scoped continuity Input Tokens 1 126823.00 3955.00 +122868.00 - n/a -
siem-readiness: dimension-scoped continuity Latency 1 3.99 1.68 +2.31 - n/a -
siem-readiness: dimension-scoped continuity Output Tokens 1 417.00 67.00 +350.00 - n/a -
siem-readiness: dimension-scoped continuity SIEM Readiness Criteria 1 0.45 0.45 0.00 - n/a -
siem-readiness: dimension-scoped continuity Tool Calls 1 4.00 0.00 +4.00 - n/a -
siem-readiness: four-section format Cached Tokens 1 10618.00 315226.00 -304608.00 - n/a -
siem-readiness: four-section format Input Tokens 1 20643.00 522302.00 -501659.00 - n/a -
siem-readiness: four-section format Latency 1 3.56 6.68 -3.12 - n/a -
siem-readiness: four-section format Output Tokens 1 269.00 5129.00 -4860.00 - n/a -
siem-readiness: four-section format SIEM Readiness Criteria 1 0.33 0.17 +0.17 - n/a -
siem-readiness: four-section format Tool Calls 1 1.00 33.00 -32.00 - n/a -
siem-readiness: full report Cached Tokens 1 336943.00 74908.00 +262035.00 - n/a -
siem-readiness: full report Input Tokens 1 471522.00 193049.00 +278473.00 - n/a -
siem-readiness: full report Latency 1 9.10 10.55 -1.44 - n/a -
siem-readiness: full report Output Tokens 1 4748.00 2752.00 +1996.00 - n/a -
siem-readiness: full report SIEM Readiness Criteria 1 0.25 0.17 +0.08 - n/a -
siem-readiness: full report Tool Calls 1 24.00 12.00 +12.00 - n/a -
siem-readiness: noData for non-existent index Input Tokens 1 4242.00 16636.00 -12394.00 - n/a -
siem-readiness: noData for non-existent index Latency 1 1.19 3.35 -2.15 - n/a -
siem-readiness: noData for non-existent index Output Tokens 1 79.00 261.00 -182.00 - n/a -
siem-readiness: noData for non-existent index SIEM Readiness Criteria 1 0.60 0.60 0.00 - n/a -
siem-readiness: noData for non-existent index Tool Calls 1 0.00 1.00 -1.00 - n/a -
siem-readiness: quality ECS incompatibility Input Tokens 1 4224.00 70128.00 -65904.00 - n/a -
siem-readiness: quality ECS incompatibility Latency 1 1.32 5.89 -4.57 - n/a -
siem-readiness: quality ECS incompatibility Output Tokens 1 69.00 437.00 -368.00 - n/a -
siem-readiness: quality ECS incompatibility SIEM Readiness Criteria 1 0.18 0.18 0.00 - n/a -
siem-readiness: quality ECS incompatibility Tool Calls 1 0.00 3.00 -3.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Input Tokens 1 38796.00 38400.00 +396.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Latency 1 3.85 4.60 -0.76 - n/a -
siem-readiness: retention FedRAMP non-compliance Output Tokens 1 408.00 585.00 -177.00 - n/a -
siem-readiness: retention FedRAMP non-compliance SIEM Readiness Criteria 1 0.36 0.36 0.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Tool Calls 1 2.00 2.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::siem-readiness::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::siem-readiness::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (53 evaluator comparisons).

Note: 1 unpaired (ran in only one experiment), 1 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (53 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
siem-readiness: blast radius accuracy Cached Tokens 1 163501.00 138548.00 +24953.00 - n/a -
siem-readiness: blast radius accuracy Input Tokens 1 309050.00 235645.00 +73405.00 - n/a -
siem-readiness: blast radius accuracy Latency 1 14.99 4.99 +10.00 - n/a -
siem-readiness: blast radius accuracy Output Tokens 1 4136.00 2305.00 +1831.00 - n/a -
siem-readiness: blast radius accuracy SIEM Readiness Criteria 1 0.36 0.27 +0.09 - n/a -
siem-readiness: blast radius accuracy Tool Calls 1 11.00 19.00 -8.00 - n/a -
siem-readiness: continuity critical pipeline failure Input Tokens 1 37787.00 16202.00 +21585.00 - n/a -
siem-readiness: continuity critical pipeline failure Latency 1 7.71 2.37 +5.35 - n/a -
siem-readiness: continuity critical pipeline failure Output Tokens 1 184.00 124.00 +60.00 - n/a -
siem-readiness: continuity critical pipeline failure SIEM Readiness Criteria 1 0.55 0.18 +0.36 - n/a -
siem-readiness: continuity critical pipeline failure Tool Calls 1 2.00 1.00 +1.00 - n/a -
siem-readiness: coverage missing Application/SaaS Cached Tokens 1 11904.00 20864.00 -8960.00 - n/a -
siem-readiness: coverage missing Application/SaaS Input Tokens 1 16427.00 36506.00 -20079.00 - n/a -
siem-readiness: coverage missing Application/SaaS Latency 1 2.45 3.03 -0.59 - n/a -
siem-readiness: coverage missing Application/SaaS Output Tokens 1 127.00 193.00 -66.00 - n/a -
siem-readiness: coverage missing Application/SaaS SIEM Readiness Criteria 1 0.30 0.30 0.00 - n/a -
siem-readiness: coverage missing Application/SaaS Tool Calls 1 1.00 3.00 -2.00 - n/a -
siem-readiness: dimension-scoped continuity Cached Tokens 1 28416.00 20864.00 +7552.00 - n/a -
siem-readiness: dimension-scoped continuity Input Tokens 1 37810.00 37566.00 +244.00 - n/a -
siem-readiness: dimension-scoped continuity Latency 1 3.54 3.19 +0.35 - n/a -
siem-readiness: dimension-scoped continuity Output Tokens 1 187.00 136.00 +51.00 - n/a -
siem-readiness: dimension-scoped continuity SIEM Readiness Criteria 1 0.36 0.55 -0.18 - n/a -
siem-readiness: dimension-scoped continuity Tool Calls 1 2.00 2.00 0.00 - n/a -
siem-readiness: four-section format Cached Tokens 1 166657.00 107835.00 +58822.00 - n/a -
siem-readiness: four-section format Input Tokens 1 293173.00 316663.00 -23490.00 - n/a -
siem-readiness: four-section format Latency 1 8.53 14.20 -5.67 - n/a -
siem-readiness: four-section format Output Tokens 1 2344.00 2666.00 -322.00 - n/a -
siem-readiness: four-section format SIEM Readiness Criteria 1 0.25 0.25 0.00 - n/a -
siem-readiness: four-section format Tool Calls 1 14.00 30.00 -16.00 - n/a -
siem-readiness: full report Cached Tokens 1 181440.00 68309.00 +113131.00 - n/a -
siem-readiness: full report Input Tokens 1 347551.00 246357.00 +101194.00 - n/a -
siem-readiness: full report Latency 1 6.19 5.54 +0.65 - n/a -
siem-readiness: full report Output Tokens 1 1784.00 1539.00 +245.00 - n/a -
siem-readiness: full report SIEM Readiness Criteria 1 0.17 0.17 0.00 - n/a -
siem-readiness: full report Tool Calls 1 21.00 13.00 +8.00 - n/a -
siem-readiness: noData for non-existent index Cached Tokens 1 11904.00 12032.00 -128.00 - n/a -
siem-readiness: noData for non-existent index Input Tokens 1 16486.00 16251.00 +235.00 - n/a -
siem-readiness: noData for non-existent index Latency 1 2.29 1.83 +0.46 - n/a -
siem-readiness: noData for non-existent index Output Tokens 1 121.00 101.00 +20.00 - n/a -
siem-readiness: noData for non-existent index SIEM Readiness Criteria 1 0.60 0.60 0.00 - n/a -
siem-readiness: noData for non-existent index Tool Calls 1 1.00 1.00 0.00 - n/a -
siem-readiness: quality ECS incompatibility Cached Tokens 1 11904.00 20864.00 -8960.00 - n/a -
siem-readiness: quality ECS incompatibility Input Tokens 1 16468.00 37585.00 -21117.00 - n/a -
siem-readiness: quality ECS incompatibility Latency 1 2.03 3.04 -1.00 - n/a -
siem-readiness: quality ECS incompatibility Output Tokens 1 130.00 209.00 -79.00 - n/a -
siem-readiness: quality ECS incompatibility SIEM Readiness Criteria 1 0.27 0.27 0.00 - n/a -
siem-readiness: quality ECS incompatibility Tool Calls 1 1.00 3.00 -2.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Cached Tokens 1 20608.00 29696.00 -9088.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Input Tokens 1 60595.00 60660.00 -65.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Latency 1 3.31 2.71 +0.60 - n/a -
siem-readiness: retention FedRAMP non-compliance Output Tokens 1 254.00 316.00 -62.00 - n/a -
siem-readiness: retention FedRAMP non-compliance SIEM Readiness Criteria 1 0.36 0.36 0.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Tool Calls 1 3.00 3.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::siem-readiness::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::siem-readiness::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (47 evaluator comparisons).

Note: 4 unpaired (ran in only one experiment), 8 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (47 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
siem-readiness: blast radius accuracy Input Tokens 1 16912.00 33860.00 -16948.00 - n/a -
siem-readiness: blast radius accuracy Latency 1 1.82 4.21 -2.39 - n/a -
siem-readiness: blast radius accuracy Output Tokens 1 308.00 709.00 -401.00 - n/a -
siem-readiness: blast radius accuracy SIEM Readiness Criteria 1 0.45 0.45 0.00 - n/a -
siem-readiness: blast radius accuracy Tool Calls 1 1.00 2.00 -1.00 - n/a -
siem-readiness: continuity critical pipeline failure Input Tokens 1 4210.00 16578.00 -12368.00 - n/a -
siem-readiness: continuity critical pipeline failure Latency 1 1.12 2.85 -1.73 - n/a -
siem-readiness: continuity critical pipeline failure Output Tokens 1 66.00 208.00 -142.00 - n/a -
siem-readiness: continuity critical pipeline failure SIEM Readiness Criteria 1 0.45 0.55 -0.09 - n/a -
siem-readiness: continuity critical pipeline failure Tool Calls 1 0.00 1.00 -1.00 - n/a -
siem-readiness: coverage missing Application/SaaS Cached Tokens 1 8279.00 12480.00 -4201.00 - n/a -
siem-readiness: coverage missing Application/SaaS Input Tokens 1 16872.00 16642.00 +230.00 - n/a -
siem-readiness: coverage missing Application/SaaS Latency 1 24.20 5.96 +18.24 - n/a -
siem-readiness: coverage missing Application/SaaS Output Tokens 1 1678.00 465.00 +1213.00 - n/a -
siem-readiness: coverage missing Application/SaaS SIEM Readiness Criteria 1 0.40 0.40 0.00 - n/a -
siem-readiness: coverage missing Application/SaaS Tool Calls 1 0.00 0.00 0.00 - n/a -
siem-readiness: dimension-scoped continuity Input Tokens 1 243.00 3955.00 -3712.00 - n/a -
siem-readiness: dimension-scoped continuity Latency 1 0.94 1.56 -0.62 - n/a -
siem-readiness: dimension-scoped continuity Output Tokens 1 20.00 64.00 -44.00 - n/a -
siem-readiness: dimension-scoped continuity SIEM Readiness Criteria 1 0.55 0.45 +0.09 - n/a -
siem-readiness: dimension-scoped continuity Tool Calls 1 0.00 0.00 0.00 - n/a -
siem-readiness: four-section format Cached Tokens 1 12058.00 12480.00 -422.00 - n/a -
siem-readiness: four-section format Input Tokens 1 16963.00 27558.00 -10595.00 - n/a -
siem-readiness: four-section format Latency 1 6.39 3.46 +2.93 - n/a -
siem-readiness: four-section format Output Tokens 1 1053.00 547.00 +506.00 - n/a -
siem-readiness: four-section format SIEM Readiness Criteria 1 0.33 0.25 +0.08 - n/a -
siem-readiness: four-section format Tool Calls 1 3.00 3.00 0.00 - n/a -
siem-readiness: full report Input Tokens 1 19180.00 16557.00 +2623.00 - n/a -
siem-readiness: full report Latency 1 1.27 6.09 -4.81 - n/a -
siem-readiness: full report Output Tokens 1 92.00 340.00 -248.00 - n/a -
siem-readiness: full report SIEM Readiness Criteria 1 0.17 0.17 0.00 - n/a -
siem-readiness: full report Tool Calls 1 1.00 1.00 0.00 - n/a -
siem-readiness: noData for non-existent index Input Tokens 1 16864.00 3975.00 +12889.00 - n/a -
siem-readiness: noData for non-existent index Latency 1 30.95 1.27 +29.67 - n/a -
siem-readiness: noData for non-existent index Output Tokens 1 2272.00 65.00 +2207.00 - n/a -
siem-readiness: noData for non-existent index SIEM Readiness Criteria 1 0.60 0.60 0.00 - n/a -
siem-readiness: noData for non-existent index Tool Calls 1 21.00 0.00 +21.00 - n/a -
siem-readiness: quality ECS incompatibility Input Tokens 1 16839.00 3957.00 +12882.00 - n/a -
siem-readiness: quality ECS incompatibility Latency 1 6.63 1.53 +5.10 - n/a -
siem-readiness: quality ECS incompatibility Output Tokens 1 260.00 72.00 +188.00 - n/a -
siem-readiness: quality ECS incompatibility SIEM Readiness Criteria 1 0.45 0.45 0.00 - n/a -
siem-readiness: quality ECS incompatibility Tool Calls 1 1.00 0.00 +1.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Input Tokens 1 16882.00 38705.00 -21823.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Latency 1 3.53 3.17 +0.36 - n/a -
siem-readiness: retention FedRAMP non-compliance Output Tokens 1 232.00 538.00 -306.00 - n/a -
siem-readiness: retention FedRAMP non-compliance SIEM Readiness Criteria 1 0.45 0.45 0.00 - n/a -
siem-readiness: retention FedRAMP non-compliance Tool Calls 1 1.00 2.00 -1.00 - n/a -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::security-automatic-migrations::openai-gpt-5.4 | Baseline (main): cf2eccfa9472f4cd::security-automatic-migrations::openai-gpt-5.4
Baseline: commit f94843e, 10 days ago

Warning: Baseline is 10 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (18 evaluator comparisons).

Note: 64 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (18 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
rule-migration: qradar Hallucination Detection 2 0.50 0.50 0.00 1.00 No -
rule-migration: qradar LOOKUP JOIN Preservation 2 1.00 1.00 0.00 1.00 No -
rule-migration: qradar NL Description Faithfulness 3 0.67 1.00 -0.33 0.35 No -
rule-migration: qradar Prebuilt Rule Match 3 1.00 1.00 0.00 1.00 No -
rule-migration: qradar Translated ESQL Validity 3 0.33 0.33 0.00 1.00 No -
rule-migration: qradar Translation Result 3 0.67 0.67 0.00 1.00 No -
rule-migration: qradar Unsupported Pattern Detection 1 1.00 1.00 0.00 - n/a -
rule-migration: splunk-spl Hallucination Detection 2 0.50 0.00 +0.50 0.39 No -
rule-migration: splunk-spl LOOKUP JOIN Preservation 2 1.00 1.00 0.00 1.00 No -
rule-migration: splunk-spl Prebuilt Rule Match 3 1.00 1.00 0.00 1.00 No -
rule-migration: splunk-spl Translated ESQL Validity 3 0.33 0.33 0.00 1.00 No -
rule-migration: splunk-spl Translation Result 3 0.67 0.67 0.00 1.00 No -
rule-migration: splunk-spl Unsupported Pattern Detection 1 1.00 1.00 0.00 - n/a -
security-automatic-migrations: standard-dashboards ES|QL Completeness 4 0.70 0.71 -0.01 0.91 No -
security-automatic-migrations: standard-dashboards Markdown Error Detection 5 1.00 1.00 0.00 1.00 No -
security-automatic-migrations: standard-dashboards Panel Count Preservation 5 0.00 0.00 0.00 1.00 No -
security-automatic-migrations: standard-dashboards Translation Completeness 5 0.50 0.50 0.00 1.00 No -
security-automatic-migrations: standard-dashboards translation_fidelity 4 0.00 0.00 0.00 1.00 No -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::attack-discovery-agent-builder::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::attack-discovery-agent-builder::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (54 evaluator comparisons).

Note: 22 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (54 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
attack-discovery-agent-builder: golden-path (live-retrieval) AdToolResult 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) CostPerAlert 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Criteria 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) ForbiddenTools 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Input Tokens 1 545746.00 500087.00 +45659.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Latency 1 48.06 49.02 -0.95 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Output Tokens 1 6715.00 6584.00 +131.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Rubric 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) StrictTrajectory 1 0.21 0.25 -0.04 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Tool Calls 1 18.00 16.00 +2.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) ForbiddenTools 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Input Tokens 1 320601.00 233203.00 +87398.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Latency 1 22.50 19.53 +2.97 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Output Tokens 1 8239.00 5100.00 +3139.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) StrictTrajectory 1 0.25 0.50 -0.25 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Tool Calls 1 10.00 6.00 +4.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) WorkflowEvidence 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) AdToolResult 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) CostPerAlert 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Criteria 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Input Tokens 1 285270.00 313860.00 -28590.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Latency 1 46.05 50.08 -4.03 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Output Tokens 1 2574.00 5118.00 -2544.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Rubric 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) StrictTrajectory 1 0.11 0.11 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Tool Calls 1 11.00 11.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) AdToolResult 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) CostPerAlert 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Criteria 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Input Tokens 1 124796.00 165160.00 -40364.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Latency 1 44.74 44.36 +0.38 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Output Tokens 1 1953.00 2054.00 -101.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Rubric 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) StrictTrajectory 1 0.25 0.20 +0.05 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Tool Calls 1 5.00 7.00 -2.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Input Tokens 1 49906.00 49743.00 +163.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Latency 1 3.93 4.89 -0.95 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Output Tokens 1 240.00 261.00 -21.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) StrictTrajectory 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Tool Calls 1 2.00 2.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::attack-discovery-agent-builder::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::attack-discovery-agent-builder::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (54 evaluator comparisons).

Note: 22 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (54 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
attack-discovery-agent-builder: golden-path (live-retrieval) AdToolResult 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) CostPerAlert 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Criteria 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) ForbiddenTools 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Input Tokens 1 476995.00 441158.00 +35837.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Latency 1 38.63 127.69 -89.06 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Output Tokens 1 8054.00 7404.00 +650.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Rubric 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) StrictTrajectory 1 0.21 0.25 -0.04 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Tool Calls 1 16.00 14.00 +2.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Input Tokens 1 154490.00 118982.00 +35508.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Latency 1 7.96 9.09 -1.13 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Output Tokens 1 2410.00 2246.00 +164.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) StrictTrajectory 1 0.50 0.67 -0.17 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Tool Calls 1 6.00 5.00 +1.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) WorkflowEvidence 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) AdToolResult 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) CostPerAlert 1 1.00 0.00 +1.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Criteria 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Input Tokens 1 156124.00 553776.00 -397652.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Latency 1 38.67 39.12 -0.45 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Output Tokens 1 4327.00 3849.00 +478.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Rubric 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) StrictTrajectory 1 0.20 0.07 +0.13 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Tool Calls 1 6.00 21.00 -15.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) AdToolResult 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) CostPerAlert 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Criteria 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Input Tokens 1 163604.00 191477.00 -27873.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Latency 1 36.36 94.39 -58.03 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Output Tokens 1 1975.00 4250.00 -2275.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Rubric 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) StrictTrajectory 1 0.25 0.20 +0.05 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Tool Calls 1 6.00 6.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Input Tokens 1 78731.00 49391.00 +29340.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Latency 1 11.02 2.81 +8.20 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Output Tokens 1 486.00 265.00 +221.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) StrictTrajectory 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Tool Calls 1 2.00 2.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::attack-discovery-agent-builder::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::attack-discovery-agent-builder::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (54 evaluator comparisons).

Note: 22 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (54 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
attack-discovery-agent-builder: golden-path (live-retrieval) AdToolResult 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) CostPerAlert 1 0.00 1.00 -1.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Criteria 1 0.80 1.00 -0.20 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) ForbiddenTools 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Input Tokens 1 322561.00 266190.00 +56371.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Latency 1 14.98 17.63 -2.64 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Output Tokens 1 1915.00 2579.00 -664.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Rubric 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) StrictTrajectory 1 0.00 0.60 -0.60 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Tool Calls 1 11.00 7.00 +4.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) ForbiddenTools 1 0.00 1.00 -1.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Input Tokens 1 239800.00 726340.00 -486540.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Latency 1 7.07 9.78 -2.71 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Output Tokens 1 2073.00 1834.00 +239.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) StrictTrajectory 1 0.50 0.00 +0.50 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Tool Calls 1 5.00 20.00 -15.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) AdToolResult 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) CostPerAlert 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Criteria 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Input Tokens 1 70011.00 95339.00 -25328.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Latency 1 19.59 14.43 +5.16 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Output Tokens 1 657.00 357.00 +300.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Rubric 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) StrictTrajectory 1 0.50 0.50 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Tool Calls 1 3.00 3.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) AdToolResult 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) CostPerAlert 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Criteria 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Input Tokens 1 44354.00 94710.00 -50356.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Latency 1 17.58 16.31 +1.27 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Output Tokens 1 375.00 410.00 -35.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Rubric 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) StrictTrajectory 1 0.50 0.50 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Tool Calls 1 3.00 3.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Input Tokens 1 40892.00 40753.00 +139.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Latency 1 2.03 1.98 +0.05 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Output Tokens 1 150.00 241.00 -91.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) StrictTrajectory 1 1.00 0.25 +0.75 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Tool Calls 1 1.00 4.00 -3.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::attack-discovery-agent-builder::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::attack-discovery-agent-builder::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (55 evaluator comparisons).

Note: 19 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (55 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
attack-discovery-agent-builder: golden-path (live-retrieval) AdToolResult 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) CostPerAlert 1 1.00 0.00 +1.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Criteria 1 0.80 0.80 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) ForbiddenTools 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Input Tokens 1 272878.00 386523.00 -113645.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Latency 1 52.22 55.50 -3.27 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Output Tokens 1 2601.00 4693.00 -2092.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Rubric 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) StrictTrajectory 1 0.50 0.38 +0.13 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Tool Calls 1 9.00 12.00 -3.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) ForbiddenTools 1 0.00 1.00 -1.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Input Tokens 1 199150.00 69687.00 +129463.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Latency 1 14.61 48.98 -34.37 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Output Tokens 1 5005.00 1260.00 +3745.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) StrictTrajectory 1 0.50 0.00 +0.50 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Tool Calls 1 6.00 4.00 +2.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) WorkflowEvidence 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) AdToolResult 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) CostPerAlert 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Criteria 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Input Tokens 1 71591.00 44472.00 +27119.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Latency 1 41.66 42.75 -1.09 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Output Tokens 1 1391.00 1759.00 -368.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Rubric 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) StrictTrajectory 1 0.33 0.50 -0.17 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Tool Calls 1 4.00 3.00 +1.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) AdToolResult 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) CostPerAlert 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Criteria 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Input Tokens 1 152406.00 44146.00 +108260.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Latency 1 47.62 38.55 +9.07 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Output Tokens 1 2408.00 1640.00 +768.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Rubric 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) StrictTrajectory 1 0.17 0.50 -0.33 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Tool Calls 1 7.00 3.00 +4.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Input Tokens 1 17050.00 40718.00 -23668.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Latency 1 3.71 4.62 -0.91 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Output Tokens 1 294.00 512.00 -218.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) StrictTrajectory 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Tool Calls 1 1.00 2.00 -1.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::attack-discovery-agent-builder::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::attack-discovery-agent-builder::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (53 evaluator comparisons).

Note: 23 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (53 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
attack-discovery-agent-builder: golden-path (live-retrieval) AdToolResult 1 0.00 1.00 -1.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) AttackDiscoveryBasic 1 0.00 1.00 -1.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Criteria 1 0.00 0.80 -0.80 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) ForbiddenTools 1 1.00 0.00 +1.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Input Tokens 1 194510.00 226229.00 -31719.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Latency 1 8.81 18.68 -9.87 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Output Tokens 1 1829.00 1840.00 -11.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Rubric 1 0.00 1.00 -1.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) StrictTrajectory 1 0.75 0.43 +0.32 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Tool Calls 1 6.00 10.00 -4.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) ForbiddenTools 1 0.00 1.00 -1.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Input Tokens 1 198613.00 124199.00 +74414.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Latency 1 7.73 7.37 +0.36 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Output Tokens 1 1467.00 1358.00 +109.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) StrictTrajectory 1 0.00 0.33 -0.33 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Tool Calls 1 3.00 8.00 -5.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) WorkflowEvidence 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) AdToolResult 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) CostPerAlert 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Criteria 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Input Tokens 1 69348.00 69297.00 +51.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Latency 1 18.20 20.90 -2.70 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Output Tokens 1 461.00 558.00 -97.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Rubric 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) StrictTrajectory 1 0.50 0.50 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Tool Calls 1 3.00 3.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) AdToolResult 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) AttackDiscoveryBasic 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) CostPerAlert 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Criteria 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Input Tokens 1 69030.00 68889.00 +141.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Latency 1 21.76 17.85 +3.91 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Output Tokens 1 523.00 592.00 -69.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Rubric 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) StrictTrajectory 1 0.50 0.50 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Tool Calls 1 3.00 3.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Input Tokens 1 16660.00 39971.00 -23311.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Latency 1 2.11 6.27 -4.16 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Output Tokens 1 132.00 190.00 -58.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) StrictTrajectory 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Tool Calls 1 1.00 2.00 -1.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::attack-discovery-agent-builder::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::attack-discovery-agent-builder::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (49 evaluator comparisons).

Note: 32 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (49 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
attack-discovery-agent-builder: golden-path (live-retrieval) AdToolResult 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) AttackDiscoveryBasic 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Criteria 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Input Tokens 1 17103.00 53817.00 -36714.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Latency 1 3.11 3.22 -0.10 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Output Tokens 1 386.00 1009.00 -623.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Rubric 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) StrictTrajectory 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (live-retrieval) Tool Calls 1 1.00 2.00 -1.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) ForbiddenTools 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Input Tokens 1 17095.00 16861.00 +234.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Latency 1 11.05 18.68 -7.63 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Output Tokens 1 907.00 1590.00 -683.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) StrictTrajectory 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) Tool Calls 1 7.00 19.00 -12.00 - n/a -
attack-discovery-agent-builder: golden-path (missing-alert-retrieval) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) AdToolResult 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) AttackDiscoveryBasic 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Criteria 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Input Tokens 1 21713.00 43151.00 -21438.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Latency 1 7.26 2.73 +4.54 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Output Tokens 1 863.00 511.00 +352.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Rubric 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) StrictTrajectory 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (multiple-alert-sets) Tool Calls 1 1.00 2.00 -1.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) AdToolResult 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) AttackDiscoveryBasic 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Criteria 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Input Tokens 1 18403.00 121107.00 -102704.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Latency 1 4.61 9.79 -5.18 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Output Tokens 1 543.00 2184.00 -1641.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Rubric 1 0.00 0.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Skill Invoked 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) StrictTrajectory 1 0.00 0.50 -0.50 - n/a -
attack-discovery-agent-builder: golden-path (provided-alerts) Tool Calls 1 3.00 2.00 +1.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) ForbiddenTools 1 1.00 1.00 0.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Input Tokens 1 4280.00 16794.00 -12514.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Latency 1 1.60 2.84 -1.23 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Output Tokens 1 75.00 396.00 -321.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Skill Invoked 1 0.00 1.00 -1.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) StrictTrajectory 1 0.00 1.00 -1.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) Tool Calls 1 0.00 1.00 -1.00 - n/a -
attack-discovery-agent-builder: golden-path (status-only) WorkflowEvidence 1 1.00 1.00 0.00 - n/a -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::workflows::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::workflows::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 283 comparisons.

Note: 1 unpaired (ran in only one experiment), 247 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
workflow-alert-rule: rule-field-routing Latency 6 0.82 0.95 -0.13 0.03 Yes Regression
No significant changes (282 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
workflow-alert-dedup: search-before-create Criteria 3 0.85 0.85 0.00 1.00 No -
workflow-alert-dedup: search-before-create EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create Latency 3 0.95 0.97 -0.02 0.49 No -
workflow-alert-dedup: search-before-create NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create StructuralCorrectness 3 0.96 1.00 -0.04 0.35 No -
workflow-alert-dedup: search-before-create trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing Criteria 6 0.92 0.83 +0.09 0.32 No -
workflow-alert-rule: rule-field-routing EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing Efficiency 6 0.99 1.00 -0.01 0.32 No -
workflow-alert-rule: rule-field-routing NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing StructuralCorrectness 6 1.00 0.90 +0.10 0.32 No -
workflow-alert-rule: rule-field-routing trajectory 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core Criteria 2 0.76 0.81 -0.06 0.39 No -
workflow-creation-alert: core EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core Efficiency 2 0.95 0.92 +0.03 0.39 No -
workflow-creation-alert: core Latency 2 0.88 0.97 -0.09 0.39 No -
workflow-creation-alert: core LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core LiquidPresence 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core StructuralCorrectness 2 0.72 0.57 +0.15 0.39 No -
workflow-creation-alert: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin Criteria 2 0.84 0.84 0.00 1.00 No -
workflow-creation-builtin EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin Latency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation-builtin NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases Criteria 2 0.62 0.70 -0.07 0.39 No -
workflow-creation-cases EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases Latency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases StructuralCorrectness 2 0.71 0.54 +0.17 0.39 No -
workflow-creation-cases trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation-connector Criteria 5 0.94 0.94 0.00 1.00 No -
workflow-creation-connector EditToolSuccess 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector Efficiency 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector Latency 5 0.94 0.95 -0.01 0.32 No -
workflow-creation-connector LiquidCorrectness 4 1.00 1.00 0.00 1.00 No -
workflow-creation-connector NoErrors 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector StructuralCorrectness 5 0.73 0.68 +0.05 0.17 No -
workflow-creation-connector trajectory 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector ValidationPass 5 1.00 1.00 0.00 1.00 No -
workflow-creation-detection-rule: notify Criteria 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify Latency 1 1.00 0.97 +0.03 - n/a -
workflow-creation-detection-rule: notify LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify StructuralCorrectness 1 0.50 0.50 0.00 - n/a -
workflow-creation-detection-rule: notify trajectory 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-creation-es Criteria 3 0.88 1.00 -0.12 0.35 No -
workflow-creation-es EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es Latency 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-es NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es StructuralCorrectness 3 0.89 0.94 -0.06 0.35 No -
workflow-creation-es trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es: index-cleanup Criteria 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup Latency 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup StructuralCorrectness 1 0.88 0.88 0.00 - n/a -
workflow-creation-es: index-cleanup trajectory 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-creation-hard Criteria 5 0.55 0.58 -0.03 0.69 No -
workflow-creation-hard EditToolSuccess 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard Efficiency 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard Latency 5 0.79 0.89 -0.10 0.26 No -
workflow-creation-hard LiquidCorrectness 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard LiquidPresence 2 0.50 0.50 0.00 1.00 No -
workflow-creation-hard NoErrors 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard StructuralCorrectness 5 0.73 0.65 +0.09 0.21 No -
workflow-creation-hard trajectory 5 0.90 0.90 0.00 1.00 No -
workflow-creation-hard ValidationPass 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 Criteria 8 0.91 0.90 +0.01 0.91 No -
workflow-creation-hard-wave2 EditToolSuccess 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 Efficiency 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 Latency 8 0.73 0.93 -0.20 0.09 No -
workflow-creation-hard-wave2 LiquidCorrectness 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 LiquidPresence 5 0.60 0.80 -0.20 0.33 No -
workflow-creation-hard-wave2 NoErrors 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 StructuralCorrectness 8 0.82 0.86 -0.03 0.19 No -
workflow-creation-hard-wave2 trajectory 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 ValidationPass 8 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core Criteria 2 0.76 0.81 -0.06 0.39 No -
workflow-creation-long: core EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core Efficiency 2 1.00 0.95 +0.05 0.39 No -
workflow-creation-long: core Latency 2 0.25 0.28 -0.03 0.39 No -
workflow-creation-long: core LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core LiquidPresence 1 0.67 0.67 0.00 - n/a -
workflow-creation-long: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core StructuralCorrectness 2 0.77 0.85 -0.08 0.39 No -
workflow-creation-long: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation: core Criteria 6 0.94 0.89 +0.05 0.32 No -
workflow-creation: core EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core Latency 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-creation: core NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core StructuralCorrectness 6 0.91 0.88 +0.03 0.32 No -
workflow-creation: core trajectory 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core Latency 3 0.90 1.00 -0.10 0.35 No -
workflow-editing-cases: core LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core StructuralCorrectness 3 0.61 0.61 0.00 1.00 No -
workflow-editing-cases: core trajectory 2 0.85 0.85 0.00 1.00 No -
workflow-editing-cases: core UsedExpectedTools 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core UsedExpectedTools 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage Criteria 2 0.80 0.80 0.00 1.00 No -
workflow-editing-diverse: alert-triage EditPreservation 2 0.50 0.50 0.00 1.00 No -
workflow-editing-diverse: alert-triage EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage Latency 2 0.86 0.84 +0.02 0.66 No -
workflow-editing-diverse: alert-triage LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: alert-triage UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill Criteria 2 0.80 0.80 0.00 1.00 No -
workflow-editing-diverse: backfill EditPreservation 2 0.00 0.00 0.00 1.00 No -
workflow-editing-diverse: backfill EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill Latency 2 0.95 1.00 -0.05 0.07 No -
workflow-editing-diverse: backfill LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: backfill NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill StructuralCorrectness 2 0.50 0.50 0.00 1.00 No -
workflow-editing-diverse: backfill trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: backfill UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: enrichment-add-email Criteria 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email EditPreservation 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email Efficiency 1 0.78 0.78 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email Latency 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email StructuralCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email UsedExpectedTools 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: triaging Criteria 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging EditPreservation 2 0.76 0.76 0.00 1.00 No -
workflow-editing-diverse: triaging EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging Latency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: triaging UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-es: core UsedExpectedTools 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Criteria 2 0.86 0.86 0.00 1.00 No -
workflow-editing-multiturn: core EditPreservation 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Latency 2 0.34 0.50 -0.15 0.20 No -
workflow-editing-multiturn: core LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-multiturn: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing: conditional-indexing-on-fetch-success Criteria 1 0.60 1.00 -0.40 - n/a -
workflow-editing: conditional-indexing-on-fetch-success EditPreservation 1 0.00 0.50 -0.50 - n/a -
workflow-editing: conditional-indexing-on-fetch-success EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success Latency 1 0.98 1.00 -0.02 - n/a -
workflow-editing: conditional-indexing-on-fetch-success LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success StructuralCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success UsedExpectedTools 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-editing: core Criteria 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core EditPreservation 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core Latency 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core StructuralCorrectness 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core trajectory 5 0.88 0.88 0.00 1.00 No -
workflow-editing: core UsedExpectedTools 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure BulkOperationsShape 3 1.00 0.67 +0.33 0.35 No -
workflow-es-ops: bulk-insert-structure Criteria 3 0.88 0.89 -0.01 0.95 No -
workflow-es-ops: bulk-insert-structure EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure Latency 3 0.93 1.00 -0.07 0.35 No -
workflow-es-ops: bulk-insert-structure NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure trajectory 3 0.50 0.50 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path Latency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Latency 3 0.95 0.99 -0.04 0.46 No -
workflow-es-ops: index-with-id-in-path NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness Criteria 6 1.00 0.89 +0.11 0.32 No -
workflow-es-ops: search-correctness EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness Latency 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness StructuralCorrectness 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness trajectory 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create Criteria 2 0.63 1.00 -0.38 0.39 No -
workflow-es-ops: store-with-index-create EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create Efficiency 2 0.96 1.00 -0.04 0.39 No -
workflow-es-ops: store-with-index-create Latency 2 0.84 1.00 -0.15 0.25 No -
workflow-es-ops: store-with-index-create NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create StructuralCorrectness 2 0.72 0.97 -0.25 0.39 No -
workflow-es-ops: store-with-index-create trajectory 2 0.50 0.50 0.00 1.00 No -
workflow-es-ops: store-with-index-create ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-negative: impossible-trigger Criteria 2 0.33 0.33 0.00 1.00 No -
workflow-negative: impossible-trigger Rejection 2 0.00 0.00 0.00 1.00 No -
workflow-negative: out-of-scope Criteria 2 0.42 0.42 0.00 1.00 No -
workflow-negative: out-of-scope Rejection 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsafe Criteria 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsafe Rejection 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsupported-features Criteria 3 0.33 0.50 -0.17 0.35 No -
workflow-negative: unsupported-features Rejection 3 0.33 0.00 +0.33 0.35 No -
workflow-self-correction: indent-mismatch Criteria 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: indent-mismatch NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: indent-mismatch SelfCorrection 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: semantic Criteria 2 0.72 0.72 0.00 1.00 No -
workflow-self-correction: semantic NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-self-correction: semantic SelfCorrection 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::workflows::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::workflows::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (283 evaluator comparisons).

Note: 4 unpaired (ran in only one experiment), 249 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (283 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
workflow-alert-dedup: search-before-create Criteria 3 0.76 0.85 -0.10 0.35 No -
workflow-alert-dedup: search-before-create EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create Latency 3 1.00 0.99 +0.01 0.35 No -
workflow-alert-dedup: search-before-create NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create StructuralCorrectness 3 0.96 1.00 -0.04 0.35 No -
workflow-alert-dedup: search-before-create trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing Criteria 6 0.84 0.74 +0.10 0.51 No -
workflow-alert-rule: rule-field-routing EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing Efficiency 6 0.98 0.96 +0.02 0.60 No -
workflow-alert-rule: rule-field-routing Latency 6 0.99 0.94 +0.05 0.40 No -
workflow-alert-rule: rule-field-routing NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing StructuralCorrectness 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing trajectory 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core Criteria 2 0.48 0.54 -0.06 0.39 No -
workflow-creation-alert: core EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core Efficiency 2 1.00 0.89 +0.11 0.39 No -
workflow-creation-alert: core Latency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core LiquidPresence 2 0.75 0.75 0.00 1.00 No -
workflow-creation-alert: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core StructuralCorrectness 2 0.56 0.58 -0.02 0.90 No -
workflow-creation-alert: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin Criteria 2 0.84 0.84 0.00 1.00 No -
workflow-creation-builtin EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin Latency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation-builtin NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases Criteria 2 0.61 0.61 0.00 1.00 No -
workflow-creation-cases EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases Latency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases StructuralCorrectness 2 0.58 0.51 +0.07 0.76 No -
workflow-creation-cases trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation-connector Criteria 5 0.94 0.60 +0.34 0.15 No -
workflow-creation-connector EditToolSuccess 5 1.00 0.60 +0.40 0.14 No -
workflow-creation-connector Efficiency 5 1.00 0.96 +0.04 0.05 No -
workflow-creation-connector Latency 5 0.97 1.00 -0.03 0.33 No -
workflow-creation-connector LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-connector NoErrors 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector StructuralCorrectness 5 0.72 0.48 +0.23 0.25 No -
workflow-creation-connector trajectory 5 1.00 0.60 +0.40 0.14 No -
workflow-creation-connector ValidationPass 5 1.00 0.60 +0.40 0.14 No -
workflow-creation-detection-rule: notify Criteria 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify Efficiency 1 1.00 0.80 +0.20 - n/a -
workflow-creation-detection-rule: notify Latency 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify StructuralCorrectness 1 0.50 0.50 0.00 - n/a -
workflow-creation-detection-rule: notify trajectory 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-creation-es Criteria 3 0.89 0.89 0.00 1.00 No -
workflow-creation-es EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es Latency 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-es NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es StructuralCorrectness 3 0.92 0.92 0.00 1.00 No -
workflow-creation-es trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es: index-cleanup Criteria 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup Latency 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup StructuralCorrectness 1 1.00 0.88 +0.13 - n/a -
workflow-creation-es: index-cleanup trajectory 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-creation-hard Criteria 5 0.62 0.68 -0.06 0.33 No -
workflow-creation-hard EditToolSuccess 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard Efficiency 5 0.87 0.91 -0.04 0.60 No -
workflow-creation-hard Latency 5 0.99 1.00 -0.01 0.52 No -
workflow-creation-hard LiquidCorrectness 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard LiquidPresence 2 1.00 1.00 0.00 1.00 No -
workflow-creation-hard NoErrors 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard StructuralCorrectness 5 0.66 0.63 +0.03 0.33 No -
workflow-creation-hard trajectory 5 0.90 0.90 0.00 1.00 No -
workflow-creation-hard ValidationPass 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 Criteria 8 0.90 0.90 0.00 1.00 No -
workflow-creation-hard-wave2 EditToolSuccess 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 Efficiency 8 0.94 0.93 +0.01 0.86 No -
workflow-creation-hard-wave2 Latency 8 0.96 0.91 +0.05 0.43 No -
workflow-creation-hard-wave2 LiquidCorrectness 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 LiquidPresence 5 0.80 0.60 +0.20 0.33 No -
workflow-creation-hard-wave2 NoErrors 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 StructuralCorrectness 8 0.92 0.83 +0.09 0.16 No -
workflow-creation-hard-wave2 trajectory 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 ValidationPass 8 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core Criteria 2 0.64 0.81 -0.17 0.16 No -
workflow-creation-long: core EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core Efficiency 2 0.83 1.00 -0.17 1.00 No -
workflow-creation-long: core Latency 2 0.08 0.39 -0.32 0.21 No -
workflow-creation-long: core LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core LiquidPresence 1 0.67 0.67 0.00 - n/a -
workflow-creation-long: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core StructuralCorrectness 2 0.81 0.85 -0.05 0.49 No -
workflow-creation-long: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation: core Criteria 6 0.83 0.94 -0.11 0.15 No -
workflow-creation: core EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core Latency 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-creation: core NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core StructuralCorrectness 6 0.98 0.99 -0.01 0.32 No -
workflow-creation: core trajectory 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core Criteria 3 0.78 1.00 -0.22 0.35 No -
workflow-editing-cases: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core EditToolSuccess 3 0.67 1.00 -0.33 0.35 No -
workflow-editing-cases: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core StructuralCorrectness 3 0.61 0.61 0.00 1.00 No -
workflow-editing-cases: core trajectory 2 0.85 0.85 0.00 1.00 No -
workflow-editing-cases: core UsedExpectedTools 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core ValidationPass 3 0.67 1.00 -0.33 0.35 No -
workflow-editing-connector: core Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core UsedExpectedTools 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage Criteria 2 0.80 0.80 0.00 1.00 No -
workflow-editing-diverse: alert-triage EditPreservation 2 0.50 0.50 0.00 1.00 No -
workflow-editing-diverse: alert-triage EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage Latency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: alert-triage UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill Criteria 2 0.60 0.80 -0.20 0.39 No -
workflow-editing-diverse: backfill EditPreservation 2 0.00 0.00 0.00 1.00 No -
workflow-editing-diverse: backfill EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill Latency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: backfill NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill StructuralCorrectness 2 0.50 0.50 0.00 1.00 No -
workflow-editing-diverse: backfill trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: backfill UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: enrichment-add-email Criteria 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email EditPreservation 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email Efficiency 1 0.78 0.78 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email Latency 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email StructuralCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email UsedExpectedTools 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: triaging Criteria 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging EditPreservation 2 0.76 0.76 0.00 1.00 No -
workflow-editing-diverse: triaging EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging Latency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: triaging UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-es: core UsedExpectedTools 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Criteria 2 0.86 0.59 +0.28 0.39 No -
workflow-editing-multiturn: core EditPreservation 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Latency 2 0.92 0.97 -0.05 0.39 No -
workflow-editing-multiturn: core LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-multiturn: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing: conditional-indexing-on-fetch-success Criteria 1 0.60 1.00 -0.40 - n/a -
workflow-editing: conditional-indexing-on-fetch-success EditPreservation 1 0.00 0.50 -0.50 - n/a -
workflow-editing: conditional-indexing-on-fetch-success EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success Efficiency 1 1.00 0.90 +0.10 - n/a -
workflow-editing: conditional-indexing-on-fetch-success Latency 1 1.00 0.44 +0.56 - n/a -
workflow-editing: conditional-indexing-on-fetch-success LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success StructuralCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success UsedExpectedTools 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-editing: core Criteria 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core EditPreservation 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core Latency 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core StructuralCorrectness 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core trajectory 5 0.88 0.88 0.00 1.00 No -
workflow-editing: core UsedExpectedTools 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure BulkOperationsShape 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure Criteria 3 1.00 0.88 +0.12 0.35 No -
workflow-es-ops: bulk-insert-structure EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure Latency 3 0.99 0.97 +0.02 0.55 No -
workflow-es-ops: bulk-insert-structure NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure StructuralCorrectness 3 0.88 1.00 -0.13 0.35 No -
workflow-es-ops: bulk-insert-structure trajectory 3 0.50 0.50 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path Latency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Latency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness Criteria 6 0.92 0.92 0.00 1.00 No -
workflow-es-ops: search-correctness EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness Latency 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness StructuralCorrectness 6 0.94 0.93 +0.02 0.32 No -
workflow-es-ops: search-correctness trajectory 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create Criteria 2 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create Efficiency 2 0.94 0.96 -0.03 0.39 No -
workflow-es-ops: store-with-index-create Latency 2 1.00 0.91 +0.09 0.39 No -
workflow-es-ops: store-with-index-create NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create StructuralCorrectness 2 0.97 0.82 +0.15 0.27 No -
workflow-es-ops: store-with-index-create trajectory 2 0.50 0.50 0.00 1.00 No -
workflow-es-ops: store-with-index-create ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-negative: impossible-trigger Criteria 2 0.50 0.17 +0.33 0.39 No -
workflow-negative: impossible-trigger Rejection 2 0.50 0.50 0.00 1.00 No -
workflow-negative: out-of-scope Criteria 2 0.42 0.42 0.00 1.00 No -
workflow-negative: out-of-scope Rejection 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsafe Criteria 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsafe Rejection 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsupported-features Criteria 3 0.83 0.83 0.00 1.00 No -
workflow-negative: unsupported-features Rejection 3 0.67 0.33 +0.33 0.35 No -
workflow-self-correction: indent-mismatch Criteria 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: indent-mismatch NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: indent-mismatch SelfCorrection 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: semantic Criteria 2 0.63 0.53 +0.10 0.39 No -
workflow-self-correction: semantic NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-self-correction: semantic SelfCorrection 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::workflows::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::workflows::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 282 comparisons.

Note: 248 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
workflow-creation-hard-wave2 Efficiency 8 0.96 0.85 +0.11 0.01 Yes Improvement
No significant changes (281 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
workflow-alert-dedup: search-before-create Criteria 3 0.85 0.52 +0.33 0.35 No -
workflow-alert-dedup: search-before-create EditToolSuccess 3 1.00 0.67 +0.33 0.35 No -
workflow-alert-dedup: search-before-create Efficiency 3 0.98 0.98 0.00 1.00 No -
workflow-alert-dedup: search-before-create Latency 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create StructuralCorrectness 3 1.00 0.67 +0.33 0.35 No -
workflow-alert-dedup: search-before-create trajectory 3 1.00 0.67 +0.33 0.35 No -
workflow-alert-dedup: search-before-create ValidationPass 3 1.00 0.67 +0.33 0.35 No -
workflow-alert-rule: rule-field-routing Criteria 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing Efficiency 6 0.99 0.91 +0.08 0.10 No -
workflow-alert-rule: rule-field-routing Latency 6 0.98 1.00 -0.02 0.32 No -
workflow-alert-rule: rule-field-routing NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing StructuralCorrectness 6 1.00 0.92 +0.08 0.32 No -
workflow-alert-rule: rule-field-routing trajectory 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core Criteria 2 0.44 0.44 0.00 1.00 No -
workflow-creation-alert: core EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core Efficiency 2 0.88 0.95 -0.07 0.39 No -
workflow-creation-alert: core Latency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core LiquidPresence 2 0.75 0.75 0.00 1.00 No -
workflow-creation-alert: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core StructuralCorrectness 2 0.46 0.46 0.00 1.00 No -
workflow-creation-alert: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin Criteria 2 0.84 0.84 0.00 1.00 No -
workflow-creation-builtin EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin Efficiency 2 0.97 1.00 -0.03 0.39 No -
workflow-creation-builtin Latency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation-builtin NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin StructuralCorrectness 2 1.00 0.69 +0.31 0.39 No -
workflow-creation-builtin trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases Criteria 2 0.35 0.84 -0.49 0.18 No -
workflow-creation-cases EditToolSuccess 2 0.50 1.00 -0.50 0.39 No -
workflow-creation-cases Efficiency 2 1.00 0.82 +0.18 0.39 No -
workflow-creation-cases Latency 2 1.00 0.78 +0.22 0.39 No -
workflow-creation-cases LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation-cases NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases StructuralCorrectness 2 0.33 0.71 -0.37 0.39 No -
workflow-creation-cases trajectory 2 0.50 1.00 -0.50 0.39 No -
workflow-creation-cases ValidationPass 2 0.50 1.00 -0.50 0.39 No -
workflow-creation-connector Criteria 5 1.00 0.94 +0.06 0.33 No -
workflow-creation-connector EditToolSuccess 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector Efficiency 5 1.00 0.90 +0.10 0.09 No -
workflow-creation-connector Latency 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector LiquidCorrectness 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector NoErrors 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector StructuralCorrectness 5 0.75 0.68 +0.07 0.33 No -
workflow-creation-connector trajectory 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector ValidationPass 5 1.00 1.00 0.00 1.00 No -
workflow-creation-detection-rule: notify Criteria 1 0.68 1.00 -0.32 - n/a -
workflow-creation-detection-rule: notify EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify Efficiency 1 0.93 0.77 +0.16 - n/a -
workflow-creation-detection-rule: notify Latency 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify StructuralCorrectness 1 0.50 0.50 0.00 - n/a -
workflow-creation-detection-rule: notify trajectory 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-creation-es Criteria 3 0.88 1.00 -0.12 0.35 No -
workflow-creation-es EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es Efficiency 3 0.93 0.95 -0.02 0.81 No -
workflow-creation-es Latency 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es: index-cleanup Criteria 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup Latency 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup StructuralCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup trajectory 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-creation-hard Criteria 5 0.61 0.68 -0.07 0.33 No -
workflow-creation-hard EditToolSuccess 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard Efficiency 5 0.88 0.79 +0.09 0.18 No -
workflow-creation-hard Latency 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard LiquidCorrectness 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard LiquidPresence 2 1.00 1.00 0.00 1.00 No -
workflow-creation-hard NoErrors 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard StructuralCorrectness 5 0.73 0.73 0.00 1.00 No -
workflow-creation-hard trajectory 5 0.90 0.90 0.00 1.00 No -
workflow-creation-hard ValidationPass 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 Criteria 8 0.84 0.79 +0.06 0.64 No -
workflow-creation-hard-wave2 EditToolSuccess 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 Latency 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 LiquidCorrectness 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 LiquidPresence 5 0.80 0.60 +0.20 0.33 No -
workflow-creation-hard-wave2 NoErrors 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 StructuralCorrectness 8 0.82 0.86 -0.03 0.22 No -
workflow-creation-hard-wave2 trajectory 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 ValidationPass 8 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core Criteria 2 0.20 0.37 -0.17 0.39 No -
workflow-creation-long: core EditToolSuccess 2 0.50 0.50 0.00 1.00 No -
workflow-creation-long: core Efficiency 2 1.00 0.75 +0.25 0.11 No -
workflow-creation-long: core Latency 2 0.50 0.50 0.00 1.00 No -
workflow-creation-long: core LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation-long: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core StructuralCorrectness 2 0.23 0.33 -0.10 0.39 No -
workflow-creation-long: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core ValidationPass 2 0.50 0.50 0.00 1.00 No -
workflow-creation: core Criteria 6 0.83 0.83 0.00 1.00 No -
workflow-creation: core EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core Efficiency 6 0.99 0.93 +0.06 0.22 No -
workflow-creation: core Latency 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-creation: core NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core StructuralCorrectness 6 0.99 0.96 +0.03 0.32 No -
workflow-creation: core trajectory 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core Criteria 3 0.12 0.12 0.00 1.00 No -
workflow-editing-cases: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core EditToolSuccess 3 0.00 0.00 0.00 1.00 No -
workflow-editing-cases: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core StructuralCorrectness 3 0.49 0.49 0.00 1.00 No -
workflow-editing-cases: core trajectory 2 0.00 0.00 0.00 1.00 No -
workflow-editing-cases: core UsedExpectedTools 3 0.67 0.67 0.00 1.00 No -
workflow-editing-cases: core ValidationPass 3 0.00 0.00 0.00 1.00 No -
workflow-editing-connector: core Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core UsedExpectedTools 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage Criteria 2 0.80 0.80 0.00 1.00 No -
workflow-editing-diverse: alert-triage EditPreservation 2 0.40 0.50 -0.10 0.39 No -
workflow-editing-diverse: alert-triage EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage Latency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: alert-triage UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill Criteria 2 0.68 0.68 0.00 1.00 No -
workflow-editing-diverse: backfill EditPreservation 2 0.00 0.00 0.00 1.00 No -
workflow-editing-diverse: backfill EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill Latency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: backfill NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill StructuralCorrectness 2 0.50 0.50 0.00 1.00 No -
workflow-editing-diverse: backfill trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: backfill UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: enrichment-add-email Criteria 1 1.00 0.60 +0.40 - n/a -
workflow-editing-diverse: enrichment-add-email EditPreservation 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email Efficiency 1 1.00 0.49 +0.51 - n/a -
workflow-editing-diverse: enrichment-add-email Latency 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email StructuralCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email UsedExpectedTools 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: triaging Criteria 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging EditPreservation 2 0.76 0.76 0.00 1.00 No -
workflow-editing-diverse: triaging EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging Latency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: triaging UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-es: core UsedExpectedTools 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Criteria 2 0.80 0.80 0.00 1.00 No -
workflow-editing-multiturn: core EditPreservation 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Latency 2 0.70 1.00 -0.30 0.27 No -
workflow-editing-multiturn: core LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-multiturn: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing: conditional-indexing-on-fetch-success Criteria 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success EditPreservation 1 0.50 0.50 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success Latency 1 0.80 1.00 -0.20 - n/a -
workflow-editing: conditional-indexing-on-fetch-success LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success StructuralCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success UsedExpectedTools 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-editing: core Criteria 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core EditPreservation 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core Latency 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core StructuralCorrectness 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core trajectory 5 0.88 0.88 0.00 1.00 No -
workflow-editing: core UsedExpectedTools 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure BulkOperationsShape 3 0.33 0.67 -0.33 0.35 No -
workflow-es-ops: bulk-insert-structure Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure Efficiency 3 1.00 0.94 +0.06 0.19 No -
workflow-es-ops: bulk-insert-structure Latency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure trajectory 3 0.50 0.50 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path Criteria 3 0.67 0.63 +0.03 0.93 No -
workflow-es-ops: delete-id-in-path EditToolSuccess 3 0.67 1.00 -0.33 0.35 No -
workflow-es-ops: delete-id-in-path Efficiency 3 0.96 1.00 -0.04 0.35 No -
workflow-es-ops: delete-id-in-path Latency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path StructuralCorrectness 3 0.67 0.89 -0.22 0.35 No -
workflow-es-ops: delete-id-in-path trajectory 3 0.67 1.00 -0.33 0.35 No -
workflow-es-ops: delete-id-in-path ValidationPass 3 0.67 1.00 -0.33 0.35 No -
workflow-es-ops: index-with-id-in-path Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Latency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness Criteria 6 0.63 0.84 -0.21 0.33 No -
workflow-es-ops: search-correctness EditToolSuccess 6 0.83 1.00 -0.17 0.32 No -
workflow-es-ops: search-correctness Efficiency 6 0.93 0.85 +0.08 0.16 No -
workflow-es-ops: search-correctness Latency 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness StructuralCorrectness 6 0.72 0.94 -0.22 0.20 No -
workflow-es-ops: search-correctness trajectory 6 0.83 1.00 -0.17 0.32 No -
workflow-es-ops: search-correctness ValidationPass 6 0.83 1.00 -0.17 0.32 No -
workflow-es-ops: store-with-index-create Criteria 2 1.00 0.63 +0.38 0.39 No -
workflow-es-ops: store-with-index-create EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create Efficiency 2 0.84 0.86 -0.02 0.39 No -
workflow-es-ops: store-with-index-create Latency 2 0.95 1.00 -0.05 0.39 No -
workflow-es-ops: store-with-index-create NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create StructuralCorrectness 2 1.00 0.83 +0.17 0.39 No -
workflow-es-ops: store-with-index-create trajectory 2 0.50 0.50 0.00 1.00 No -
workflow-es-ops: store-with-index-create ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-negative: impossible-trigger Criteria 2 0.17 0.17 0.00 1.00 No -
workflow-negative: impossible-trigger Rejection 2 0.00 0.00 0.00 1.00 No -
workflow-negative: out-of-scope Criteria 2 0.42 0.42 0.00 1.00 No -
workflow-negative: out-of-scope Rejection 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsafe Criteria 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsafe Rejection 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsupported-features Criteria 3 0.33 0.33 0.00 1.00 No -
workflow-negative: unsupported-features Rejection 3 0.00 0.00 0.00 1.00 No -
workflow-self-correction: indent-mismatch Criteria 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: indent-mismatch NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: indent-mismatch SelfCorrection 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: semantic Criteria 2 0.30 0.53 -0.23 0.59 No -
workflow-self-correction: semantic NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-self-correction: semantic SelfCorrection 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::workflows::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::workflows::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (283 evaluator comparisons).

Note: 3 unpaired (ran in only one experiment), 257 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (283 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
workflow-alert-dedup: search-before-create Criteria 3 0.85 0.85 0.00 1.00 No -
workflow-alert-dedup: search-before-create EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create Latency 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing Criteria 6 0.92 1.00 -0.08 0.32 No -
workflow-alert-rule: rule-field-routing EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing Latency 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing StructuralCorrectness 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing trajectory 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core Criteria 2 0.81 0.81 0.00 1.00 No -
workflow-creation-alert: core EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core Latency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core LiquidPresence 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core StructuralCorrectness 2 0.75 0.63 +0.13 0.39 No -
workflow-creation-alert: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin Criteria 2 0.76 0.84 -0.08 0.39 No -
workflow-creation-builtin EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin Latency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation-builtin NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases Criteria 2 0.62 0.70 -0.07 0.39 No -
workflow-creation-cases EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases Latency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases StructuralCorrectness 2 0.71 0.71 0.00 1.00 No -
workflow-creation-cases trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation-connector Criteria 5 0.94 0.87 +0.07 0.54 No -
workflow-creation-connector EditToolSuccess 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector Efficiency 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector Latency 5 0.98 0.85 +0.12 0.16 No -
workflow-creation-connector LiquidCorrectness 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector NoErrors 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector StructuralCorrectness 5 0.75 0.73 +0.02 0.33 No -
workflow-creation-connector trajectory 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector ValidationPass 5 1.00 1.00 0.00 1.00 No -
workflow-creation-detection-rule: notify Criteria 1 1.00 0.68 +0.32 - n/a -
workflow-creation-detection-rule: notify EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify Latency 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify StructuralCorrectness 1 0.50 0.50 0.00 - n/a -
workflow-creation-detection-rule: notify trajectory 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-creation-es Criteria 3 0.72 0.72 0.00 1.00 No -
workflow-creation-es EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es Latency 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-es NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es StructuralCorrectness 3 0.88 0.88 0.00 1.00 No -
workflow-creation-es trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es: index-cleanup Criteria 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup Latency 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup StructuralCorrectness 1 0.88 0.88 0.00 - n/a -
workflow-creation-es: index-cleanup trajectory 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-creation-hard Criteria 5 0.65 0.68 -0.03 0.33 No -
workflow-creation-hard EditToolSuccess 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard Efficiency 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard Latency 5 0.80 0.79 +0.01 0.96 No -
workflow-creation-hard LiquidCorrectness 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard LiquidPresence 2 1.00 1.00 0.00 1.00 No -
workflow-creation-hard NoErrors 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard StructuralCorrectness 5 0.73 0.73 -0.01 0.33 No -
workflow-creation-hard trajectory 5 0.90 0.90 0.00 1.00 No -
workflow-creation-hard ValidationPass 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 Criteria 8 0.84 0.72 +0.13 0.32 No -
workflow-creation-hard-wave2 EditToolSuccess 8 1.00 0.88 +0.13 0.32 No -
workflow-creation-hard-wave2 Efficiency 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 Latency 8 0.76 0.60 +0.16 0.07 No -
workflow-creation-hard-wave2 LiquidCorrectness 7 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 LiquidPresence 4 0.75 0.75 0.00 1.00 No -
workflow-creation-hard-wave2 NoErrors 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 StructuralCorrectness 8 0.85 0.79 +0.06 0.66 No -
workflow-creation-hard-wave2 trajectory 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 ValidationPass 8 1.00 0.88 +0.13 0.32 No -
workflow-creation-long: core Criteria 2 0.76 0.81 -0.06 0.39 No -
workflow-creation-long: core EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core Efficiency 2 0.92 1.00 -0.08 0.39 No -
workflow-creation-long: core Latency 2 0.00 0.56 -0.56 0.20 No -
workflow-creation-long: core LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core LiquidPresence 1 0.67 0.67 0.00 - n/a -
workflow-creation-long: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core StructuralCorrectness 2 0.71 0.79 -0.08 0.39 No -
workflow-creation-long: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation: core Criteria 6 0.87 0.94 -0.07 0.32 No -
workflow-creation: core EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core Latency 6 1.00 1.00 -0.00 0.32 No -
workflow-creation: core LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation: core NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core StructuralCorrectness 6 0.96 0.96 -0.00 0.94 No -
workflow-creation: core trajectory 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core StructuralCorrectness 3 0.61 0.61 0.00 1.00 No -
workflow-editing-cases: core trajectory 2 0.85 0.85 0.00 1.00 No -
workflow-editing-cases: core UsedExpectedTools 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core Criteria 3 0.88 1.00 -0.12 0.35 No -
workflow-editing-connector: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-connector: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core UsedExpectedTools 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage Criteria 2 0.80 0.80 0.00 1.00 No -
workflow-editing-diverse: alert-triage EditPreservation 2 0.50 0.50 0.00 1.00 No -
workflow-editing-diverse: alert-triage EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage Latency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: alert-triage UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill Criteria 2 0.80 0.80 0.00 1.00 No -
workflow-editing-diverse: backfill EditPreservation 2 0.00 0.00 0.00 1.00 No -
workflow-editing-diverse: backfill EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill Latency 2 0.91 0.91 -0.00 0.39 No -
workflow-editing-diverse: backfill LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: backfill NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill StructuralCorrectness 2 0.50 0.50 0.00 1.00 No -
workflow-editing-diverse: backfill trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: backfill UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: enrichment-add-email Criteria 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email EditPreservation 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email Latency 1 1.00 0.82 +0.18 - n/a -
workflow-editing-diverse: enrichment-add-email LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email StructuralCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email UsedExpectedTools 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: triaging Criteria 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging EditPreservation 2 0.76 0.76 0.00 1.00 No -
workflow-editing-diverse: triaging EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging Latency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: triaging UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-es: core UsedExpectedTools 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Criteria 2 0.80 1.00 -0.20 0.39 No -
workflow-editing-multiturn: core EditPreservation 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Latency 2 0.49 0.08 +0.40 0.40 No -
workflow-editing-multiturn: core LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-multiturn: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing: conditional-indexing-on-fetch-success Criteria 1 1.00 0.60 +0.40 - n/a -
workflow-editing: conditional-indexing-on-fetch-success EditPreservation 1 0.50 0.00 +0.50 - n/a -
workflow-editing: conditional-indexing-on-fetch-success EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success Efficiency 1 0.90 1.00 -0.10 - n/a -
workflow-editing: conditional-indexing-on-fetch-success Latency 1 0.00 0.77 -0.77 - n/a -
workflow-editing: conditional-indexing-on-fetch-success LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success StructuralCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success UsedExpectedTools 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-editing: core Criteria 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core EditPreservation 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core Latency 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core StructuralCorrectness 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core trajectory 5 0.88 0.88 0.00 1.00 No -
workflow-editing: core UsedExpectedTools 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure BulkOperationsShape 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure Criteria 3 1.00 0.88 +0.12 0.35 No -
workflow-es-ops: bulk-insert-structure EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure Latency 3 0.68 0.71 -0.03 0.93 No -
workflow-es-ops: bulk-insert-structure NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure trajectory 3 0.50 0.50 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path Latency 3 0.67 1.00 -0.33 0.35 No -
workflow-es-ops: delete-id-in-path NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Latency 3 0.95 0.96 -0.01 0.91 No -
workflow-es-ops: index-with-id-in-path NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness Criteria 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness Latency 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness StructuralCorrectness 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness trajectory 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create Criteria 2 0.63 1.00 -0.38 0.39 No -
workflow-es-ops: store-with-index-create EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create Latency 2 0.65 0.50 +0.15 0.63 No -
workflow-es-ops: store-with-index-create NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create StructuralCorrectness 2 0.83 1.00 -0.17 0.39 No -
workflow-es-ops: store-with-index-create trajectory 2 0.50 0.50 0.00 1.00 No -
workflow-es-ops: store-with-index-create ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-negative: impossible-trigger Criteria 2 0.17 0.33 -0.17 0.39 No -
workflow-negative: impossible-trigger Rejection 2 0.00 0.00 0.00 1.00 No -
workflow-negative: out-of-scope Criteria 2 0.42 0.42 0.00 1.00 No -
workflow-negative: out-of-scope Rejection 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsafe Criteria 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsafe Rejection 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsupported-features Criteria 3 0.50 0.33 +0.17 0.35 No -
workflow-negative: unsupported-features Rejection 3 0.00 0.00 0.00 1.00 No -
workflow-self-correction: indent-mismatch Criteria 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: indent-mismatch NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: indent-mismatch SelfCorrection 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: semantic Criteria 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: semantic NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: semantic SelfCorrection 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::workflows::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::workflows::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (283 evaluator comparisons).

Note: 1 unpaired (ran in only one experiment), 250 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (283 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
workflow-alert-dedup: search-before-create Criteria 3 0.85 0.85 0.00 1.00 No -
workflow-alert-dedup: search-before-create EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create Latency 3 0.86 1.00 -0.14 0.35 No -
workflow-alert-dedup: search-before-create NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create StructuralCorrectness 3 1.00 0.96 +0.04 0.35 No -
workflow-alert-dedup: search-before-create trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing Criteria 6 0.78 0.87 -0.09 0.53 No -
workflow-alert-rule: rule-field-routing EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing Latency 6 1.00 1.00 +0.00 0.32 No -
workflow-alert-rule: rule-field-routing NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing StructuralCorrectness 6 1.00 0.98 +0.02 0.32 No -
workflow-alert-rule: rule-field-routing trajectory 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core Criteria 2 0.81 0.43 +0.39 0.19 No -
workflow-creation-alert: core EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core Latency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core LiquidPresence 2 1.00 0.75 +0.25 0.39 No -
workflow-creation-alert: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core StructuralCorrectness 2 0.75 0.71 +0.04 0.87 No -
workflow-creation-alert: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin Criteria 2 0.84 0.84 0.00 1.00 No -
workflow-creation-builtin EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin Latency 2 1.00 0.85 +0.15 0.39 No -
workflow-creation-builtin LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation-builtin NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases Criteria 2 0.70 0.70 0.00 1.00 No -
workflow-creation-cases EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases Latency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases StructuralCorrectness 2 0.71 0.71 0.00 1.00 No -
workflow-creation-cases trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation-connector Criteria 5 0.94 0.94 0.00 1.00 No -
workflow-creation-connector EditToolSuccess 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector Efficiency 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector Latency 5 1.00 0.95 +0.05 0.33 No -
workflow-creation-connector LiquidCorrectness 4 1.00 1.00 0.00 1.00 No -
workflow-creation-connector NoErrors 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector StructuralCorrectness 5 0.75 0.73 +0.02 0.33 No -
workflow-creation-connector trajectory 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector ValidationPass 5 1.00 1.00 0.00 1.00 No -
workflow-creation-detection-rule: notify Criteria 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify Latency 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify StructuralCorrectness 1 0.50 0.50 0.00 - n/a -
workflow-creation-detection-rule: notify trajectory 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-creation-es Criteria 3 0.88 0.89 -0.01 0.95 No -
workflow-creation-es EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es Latency 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-es NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es StructuralCorrectness 3 1.00 0.92 +0.08 0.35 No -
workflow-creation-es trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es: index-cleanup Criteria 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup Latency 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup StructuralCorrectness 1 1.00 0.75 +0.25 - n/a -
workflow-creation-es: index-cleanup trajectory 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-creation-hard Criteria 5 0.74 0.74 0.00 1.00 No -
workflow-creation-hard EditToolSuccess 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard Efficiency 5 1.00 0.98 +0.02 0.33 No -
workflow-creation-hard Latency 5 0.92 1.00 -0.08 0.30 No -
workflow-creation-hard LiquidCorrectness 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard LiquidPresence 2 1.00 1.00 0.00 1.00 No -
workflow-creation-hard NoErrors 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard StructuralCorrectness 5 0.69 0.70 -0.01 0.33 No -
workflow-creation-hard trajectory 5 0.90 0.90 0.00 1.00 No -
workflow-creation-hard ValidationPass 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 Criteria 8 0.97 0.79 +0.18 0.16 No -
workflow-creation-hard-wave2 EditToolSuccess 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 Efficiency 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 Latency 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 LiquidCorrectness 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 LiquidPresence 5 0.80 0.80 0.00 1.00 No -
workflow-creation-hard-wave2 NoErrors 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 StructuralCorrectness 8 0.86 0.84 +0.01 0.15 No -
workflow-creation-hard-wave2 trajectory 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 ValidationPass 8 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core Criteria 2 1.00 0.70 +0.30 0.39 No -
workflow-creation-long: core EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core Latency 2 0.75 0.45 +0.29 0.32 No -
workflow-creation-long: core LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core LiquidPresence 1 0.67 0.67 0.00 - n/a -
workflow-creation-long: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core StructuralCorrectness 2 0.85 0.85 0.00 1.00 No -
workflow-creation-long: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-creation: core Criteria 6 0.94 0.88 +0.06 0.32 No -
workflow-creation: core EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core Latency 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation: core NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core StructuralCorrectness 6 0.99 0.99 0.00 1.00 No -
workflow-creation: core trajectory 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core Criteria 3 0.45 0.45 0.00 1.00 No -
workflow-editing-cases: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core EditToolSuccess 3 0.33 0.33 0.00 1.00 No -
workflow-editing-cases: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core StructuralCorrectness 3 0.56 0.56 0.00 1.00 No -
workflow-editing-cases: core trajectory 2 0.35 0.35 0.00 1.00 No -
workflow-editing-cases: core UsedExpectedTools 3 0.67 0.67 0.00 1.00 No -
workflow-editing-cases: core ValidationPass 3 0.33 0.33 0.00 1.00 No -
workflow-editing-connector: core Criteria 3 0.88 1.00 -0.12 0.35 No -
workflow-editing-connector: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-connector: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core UsedExpectedTools 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage Criteria 2 0.80 0.80 0.00 1.00 No -
workflow-editing-diverse: alert-triage EditPreservation 2 0.60 0.50 +0.10 0.39 No -
workflow-editing-diverse: alert-triage EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage Latency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: alert-triage UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill Criteria 2 0.80 0.60 +0.20 0.39 No -
workflow-editing-diverse: backfill EditPreservation 2 0.00 0.00 0.00 1.00 No -
workflow-editing-diverse: backfill EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill Latency 2 0.98 1.00 -0.02 0.39 No -
workflow-editing-diverse: backfill LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: backfill NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill StructuralCorrectness 2 0.50 0.50 0.00 1.00 No -
workflow-editing-diverse: backfill trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: backfill UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: enrichment-add-email Criteria 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email EditPreservation 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email Latency 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email StructuralCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email UsedExpectedTools 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: triaging Criteria 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging EditPreservation 2 0.76 0.76 0.00 1.00 No -
workflow-editing-diverse: triaging EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging Latency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: triaging UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-es: core UsedExpectedTools 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Criteria 2 0.86 0.86 0.00 1.00 No -
workflow-editing-multiturn: core EditPreservation 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Latency 2 0.57 0.56 +0.02 0.41 No -
workflow-editing-multiturn: core LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-multiturn: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing: conditional-indexing-on-fetch-success Criteria 1 0.60 0.60 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success EditPreservation 1 0.00 0.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success EditToolSuccess 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success Latency 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success StructuralCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success UsedExpectedTools 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success ValidationPass 1 1.00 1.00 0.00 - n/a -
workflow-editing: core Criteria 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core EditPreservation 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core Efficiency 6 1.00 0.96 +0.04 0.32 No -
workflow-editing: core Latency 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core StructuralCorrectness 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core trajectory 5 0.88 0.88 0.00 1.00 No -
workflow-editing: core UsedExpectedTools 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure BulkOperationsShape 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure Criteria 3 0.88 0.89 -0.01 0.95 No -
workflow-es-ops: bulk-insert-structure EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure Latency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure trajectory 3 0.50 0.50 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path Criteria 3 0.82 0.75 +0.07 0.85 No -
workflow-es-ops: delete-id-in-path EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path Latency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Criteria 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path EditToolSuccess 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Latency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path ValidationPass 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness Criteria 6 0.92 1.00 -0.08 0.32 No -
workflow-es-ops: search-correctness EditToolSuccess 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness Latency 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness StructuralCorrectness 6 0.92 0.97 -0.06 0.32 No -
workflow-es-ops: search-correctness trajectory 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness ValidationPass 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create Criteria 2 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create Latency 2 1.00 0.50 +0.50 0.39 No -
workflow-es-ops: store-with-index-create NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create StructuralCorrectness 2 0.94 0.94 0.00 1.00 No -
workflow-es-ops: store-with-index-create trajectory 2 0.50 0.50 0.00 1.00 No -
workflow-es-ops: store-with-index-create ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-negative: impossible-trigger Criteria 2 0.17 0.33 -0.17 0.39 No -
workflow-negative: impossible-trigger Rejection 2 0.00 0.00 0.00 1.00 No -
workflow-negative: out-of-scope Criteria 2 0.42 0.42 0.00 1.00 No -
workflow-negative: out-of-scope Rejection 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsafe Criteria 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsafe Rejection 2 0.50 1.00 -0.50 0.39 No -
workflow-negative: unsupported-features Criteria 3 0.67 0.50 +0.17 0.69 No -
workflow-negative: unsupported-features Rejection 3 0.33 0.33 0.00 1.00 No -
workflow-self-correction: indent-mismatch Criteria 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: indent-mismatch NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: indent-mismatch SelfCorrection 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: semantic Criteria 2 0.63 0.63 0.00 1.00 No -
workflow-self-correction: semantic NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-self-correction: semantic SelfCorrection 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::workflows::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::workflows::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
2 significant difference(s) detected out of 272 comparisons.

Note: 8 unpaired (ran in only one experiment), 313 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
workflow-creation-builtin Efficiency 2 1.00 0.68 +0.32 0.00 Yes Improvement
workflow-creation-hard-wave2 Latency 8 0.98 0.46 +0.53 0.01 Yes Improvement
No significant changes (270 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
workflow-alert-dedup: search-before-create Criteria 3 0.42 0.76 -0.33 0.35 No -
workflow-alert-dedup: search-before-create EditToolSuccess 3 0.67 1.00 -0.33 0.35 No -
workflow-alert-dedup: search-before-create Efficiency 3 1.00 0.94 +0.06 0.35 No -
workflow-alert-dedup: search-before-create Latency 3 0.49 0.00 +0.49 0.14 No -
workflow-alert-dedup: search-before-create NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-alert-dedup: search-before-create StructuralCorrectness 3 0.67 1.00 -0.33 0.35 No -
workflow-alert-dedup: search-before-create trajectory 3 0.67 1.00 -0.33 0.35 No -
workflow-alert-dedup: search-before-create ValidationPass 3 0.67 1.00 -0.33 0.35 No -
workflow-alert-rule: rule-field-routing Criteria 6 0.25 0.45 -0.19 0.56 No -
workflow-alert-rule: rule-field-routing EditToolSuccess 6 0.33 0.50 -0.17 0.67 No -
workflow-alert-rule: rule-field-routing Efficiency 6 0.95 0.90 +0.05 0.32 No -
workflow-alert-rule: rule-field-routing Latency 6 0.69 0.73 -0.04 0.78 No -
workflow-alert-rule: rule-field-routing NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-alert-rule: rule-field-routing StructuralCorrectness 6 0.33 0.50 -0.17 0.67 No -
workflow-alert-rule: rule-field-routing trajectory 6 0.67 0.67 0.00 1.00 No -
workflow-alert-rule: rule-field-routing ValidationPass 6 0.33 0.50 -0.17 0.67 No -
workflow-creation-alert: core Criteria 2 0.22 0.22 0.00 1.00 No -
workflow-creation-alert: core EditToolSuccess 2 0.50 0.50 0.00 1.00 No -
workflow-creation-alert: core Efficiency 2 0.81 1.00 -0.19 0.39 No -
workflow-creation-alert: core Latency 2 0.79 0.96 -0.17 0.39 No -
workflow-creation-alert: core LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation-alert: core LiquidPresence 1 0.50 0.50 0.00 - n/a -
workflow-creation-alert: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core StructuralCorrectness 2 0.21 0.21 0.00 1.00 No -
workflow-creation-alert: core trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-alert: core ValidationPass 2 0.50 0.50 0.00 1.00 No -
workflow-creation-builtin Criteria 2 0.76 0.84 -0.08 0.39 No -
workflow-creation-builtin EditToolSuccess 2 1.00 0.50 +0.50 0.39 No -
workflow-creation-builtin Latency 2 1.00 0.53 +0.47 0.39 No -
workflow-creation-builtin LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation-builtin NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin trajectory 2 1.00 1.00 0.00 1.00 No -
workflow-creation-builtin ValidationPass 2 1.00 0.50 +0.50 0.39 No -
workflow-creation-cases Criteria 2 0.00 0.00 0.00 1.00 No -
workflow-creation-cases EditToolSuccess 2 0.00 0.00 0.00 1.00 No -
workflow-creation-cases Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases Latency 2 0.65 1.00 -0.35 0.39 No -
workflow-creation-cases NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-cases StructuralCorrectness 2 0.00 0.00 0.00 1.00 No -
workflow-creation-cases trajectory 2 0.50 1.00 -0.50 0.39 No -
workflow-creation-cases ValidationPass 2 0.00 0.00 0.00 1.00 No -
workflow-creation-connector Criteria 5 0.40 0.20 +0.20 0.58 No -
workflow-creation-connector EditToolSuccess 5 0.20 0.20 0.00 1.00 No -
workflow-creation-connector Efficiency 5 0.89 1.00 -0.11 0.15 No -
workflow-creation-connector Latency 5 0.90 1.00 -0.10 0.17 No -
workflow-creation-connector NoErrors 5 1.00 1.00 0.00 1.00 No -
workflow-creation-connector StructuralCorrectness 5 0.33 0.15 +0.18 0.54 No -
workflow-creation-connector trajectory 5 0.60 0.40 +0.20 0.33 No -
workflow-creation-connector ValidationPass 5 0.20 0.20 0.00 1.00 No -
workflow-creation-detection-rule: notify Criteria 1 0.00 0.00 0.00 - n/a -
workflow-creation-detection-rule: notify EditToolSuccess 1 0.00 0.00 0.00 - n/a -
workflow-creation-detection-rule: notify Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify Latency 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-creation-detection-rule: notify StructuralCorrectness 1 0.00 0.00 0.00 - n/a -
workflow-creation-detection-rule: notify trajectory 1 0.00 0.00 0.00 - n/a -
workflow-creation-detection-rule: notify ValidationPass 1 0.00 0.00 0.00 - n/a -
workflow-creation-es Criteria 3 1.00 0.44 +0.56 0.09 No -
workflow-creation-es EditToolSuccess 3 1.00 0.67 +0.33 0.35 No -
workflow-creation-es Efficiency 3 1.00 0.88 +0.12 0.35 No -
workflow-creation-es Latency 3 1.00 0.67 +0.33 0.35 No -
workflow-creation-es LiquidCorrectness 2 0.50 1.00 -0.50 0.39 No -
workflow-creation-es NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es StructuralCorrectness 3 1.00 0.58 +0.42 0.23 No -
workflow-creation-es trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-creation-es ValidationPass 3 1.00 0.67 +0.33 0.35 No -
workflow-creation-es: index-cleanup Criteria 1 1.00 0.00 +1.00 - n/a -
workflow-creation-es: index-cleanup EditToolSuccess 1 1.00 0.00 +1.00 - n/a -
workflow-creation-es: index-cleanup Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup Latency 1 1.00 0.77 +0.23 - n/a -
workflow-creation-es: index-cleanup NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-creation-es: index-cleanup StructuralCorrectness 1 1.00 0.00 +1.00 - n/a -
workflow-creation-es: index-cleanup trajectory 1 1.00 0.00 +1.00 - n/a -
workflow-creation-es: index-cleanup ValidationPass 1 1.00 0.00 +1.00 - n/a -
workflow-creation-hard Criteria 5 0.34 0.14 +0.20 0.49 No -
workflow-creation-hard EditToolSuccess 5 0.20 0.20 0.00 1.00 No -
workflow-creation-hard Efficiency 5 0.94 1.00 -0.06 0.33 No -
workflow-creation-hard Latency 5 0.87 0.93 -0.06 0.64 No -
workflow-creation-hard NoErrors 5 1.00 1.00 0.00 1.00 No -
workflow-creation-hard StructuralCorrectness 5 0.33 0.19 +0.14 0.67 No -
workflow-creation-hard trajectory 5 0.50 0.20 +0.30 0.42 No -
workflow-creation-hard ValidationPass 5 0.20 0.20 0.00 1.00 No -
workflow-creation-hard-wave2 Criteria 8 0.08 0.13 -0.05 0.76 No -
workflow-creation-hard-wave2 EditToolSuccess 8 0.13 0.13 0.00 1.00 No -
workflow-creation-hard-wave2 Efficiency 8 1.00 0.96 +0.04 0.32 No -
workflow-creation-hard-wave2 NoErrors 8 1.00 1.00 0.00 1.00 No -
workflow-creation-hard-wave2 StructuralCorrectness 8 0.07 0.10 -0.03 0.85 No -
workflow-creation-hard-wave2 trajectory 8 0.13 0.25 -0.13 0.32 No -
workflow-creation-hard-wave2 ValidationPass 8 0.13 0.13 0.00 1.00 No -
workflow-creation-long: core Criteria 2 0.00 0.00 0.00 1.00 No -
workflow-creation-long: core EditToolSuccess 2 0.00 0.00 0.00 1.00 No -
workflow-creation-long: core Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core Latency 2 0.50 0.49 +0.01 0.39 No -
workflow-creation-long: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-creation-long: core StructuralCorrectness 2 0.00 0.00 0.00 1.00 No -
workflow-creation-long: core trajectory 2 0.50 0.50 0.00 1.00 No -
workflow-creation-long: core ValidationPass 2 0.00 0.00 0.00 1.00 No -
workflow-creation: core Criteria 6 0.70 0.44 +0.26 0.40 No -
workflow-creation: core EditToolSuccess 6 0.83 0.50 +0.33 0.32 No -
workflow-creation: core Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core Latency 6 0.89 1.00 -0.11 0.15 No -
workflow-creation: core LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-creation: core NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-creation: core StructuralCorrectness 6 0.78 0.50 +0.28 0.37 No -
workflow-creation: core trajectory 6 0.83 0.83 0.00 1.00 No -
workflow-creation: core ValidationPass 6 0.83 0.50 +0.33 0.32 No -
workflow-editing-cases: core Criteria 3 0.12 0.12 0.00 1.00 No -
workflow-editing-cases: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core EditToolSuccess 3 0.00 0.00 0.00 1.00 No -
workflow-editing-cases: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-cases: core StructuralCorrectness 3 0.49 0.49 0.00 1.00 No -
workflow-editing-cases: core trajectory 2 0.00 0.00 0.00 1.00 No -
workflow-editing-cases: core UsedExpectedTools 3 0.67 0.67 0.00 1.00 No -
workflow-editing-cases: core ValidationPass 3 0.00 0.00 0.00 1.00 No -
workflow-editing-connector: core Criteria 3 0.37 0.68 -0.32 0.35 No -
workflow-editing-connector: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core EditToolSuccess 3 0.33 0.67 -0.33 0.35 No -
workflow-editing-connector: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-connector: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-connector: core StructuralCorrectness 3 0.92 0.92 0.00 1.00 No -
workflow-editing-connector: core trajectory 2 0.50 0.50 0.00 1.00 No -
workflow-editing-connector: core UsedExpectedTools 3 0.67 0.67 0.00 1.00 No -
workflow-editing-connector: core ValidationPass 3 0.33 0.67 -0.33 0.35 No -
workflow-editing-diverse: alert-triage Criteria 2 0.33 0.33 0.00 1.00 No -
workflow-editing-diverse: alert-triage EditPreservation 2 0.70 0.70 0.00 1.00 No -
workflow-editing-diverse: alert-triage EditToolSuccess 2 0.50 0.50 0.00 1.00 No -
workflow-editing-diverse: alert-triage Efficiency 2 1.00 0.90 +0.10 0.39 No -
workflow-editing-diverse: alert-triage Latency 2 0.98 0.72 +0.25 0.44 No -
workflow-editing-diverse: alert-triage LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage trajectory 1 1.00 0.00 +1.00 - n/a -
workflow-editing-diverse: alert-triage UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: alert-triage ValidationPass 2 0.50 0.50 0.00 1.00 No -
workflow-editing-diverse: backfill Criteria 2 0.22 0.30 -0.08 0.65 No -
workflow-editing-diverse: backfill EditPreservation 2 0.00 0.00 0.00 1.00 No -
workflow-editing-diverse: backfill EditToolSuccess 2 0.00 1.00 -1.00 1.00 No -
workflow-editing-diverse: backfill Efficiency 2 1.00 0.96 +0.04 0.39 No -
workflow-editing-diverse: backfill Latency 2 1.00 0.32 +0.68 0.20 No -
workflow-editing-diverse: backfill LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: backfill NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill StructuralCorrectness 2 0.50 0.50 0.00 1.00 No -
workflow-editing-diverse: backfill trajectory 1 0.00 1.00 -1.00 - n/a -
workflow-editing-diverse: backfill UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: backfill ValidationPass 2 0.00 1.00 -1.00 1.00 No -
workflow-editing-diverse: enrichment-add-email Criteria 1 0.35 0.35 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email EditPreservation 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email EditToolSuccess 1 0.00 1.00 -1.00 - n/a -
workflow-editing-diverse: enrichment-add-email Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email Latency 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email StructuralCorrectness 1 0.00 0.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email trajectory 1 0.00 1.00 -1.00 - n/a -
workflow-editing-diverse: enrichment-add-email UsedExpectedTools 1 1.00 1.00 0.00 - n/a -
workflow-editing-diverse: enrichment-add-email ValidationPass 1 0.00 1.00 -1.00 - n/a -
workflow-editing-diverse: triaging Criteria 2 0.40 0.40 0.00 1.00 No -
workflow-editing-diverse: triaging EditPreservation 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging EditToolSuccess 2 0.00 0.00 0.00 1.00 No -
workflow-editing-diverse: triaging Efficiency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging Latency 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging LiquidCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging trajectory 1 1.00 0.00 +1.00 - n/a -
workflow-editing-diverse: triaging UsedExpectedTools 2 1.00 1.00 0.00 1.00 No -
workflow-editing-diverse: triaging ValidationPass 2 0.00 0.00 0.00 1.00 No -
workflow-editing-es: core Criteria 3 0.67 1.00 -0.33 0.35 No -
workflow-editing-es: core EditPreservation 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core EditToolSuccess 3 0.67 1.00 -0.33 0.35 No -
workflow-editing-es: core Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core Latency 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core LiquidCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core StructuralCorrectness 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core trajectory 1 1.00 1.00 0.00 - n/a -
workflow-editing-es: core UsedExpectedTools 3 1.00 1.00 0.00 1.00 No -
workflow-editing-es: core ValidationPass 3 0.67 1.00 -0.33 0.35 No -
workflow-editing-multiturn: core Criteria 2 0.69 0.80 -0.11 0.62 No -
workflow-editing-multiturn: core EditPreservation 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core EditToolSuccess 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core Efficiency 2 0.86 1.00 -0.14 0.39 No -
workflow-editing-multiturn: core Latency 2 0.50 0.90 -0.40 0.39 No -
workflow-editing-multiturn: core LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing-multiturn: core NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core StructuralCorrectness 2 1.00 1.00 0.00 1.00 No -
workflow-editing-multiturn: core ValidationPass 2 1.00 1.00 0.00 1.00 No -
workflow-editing: conditional-indexing-on-fetch-success Criteria 1 0.10 0.10 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success EditPreservation 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success EditToolSuccess 1 0.00 0.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success Efficiency 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success Latency 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success LiquidCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success LiquidPresence 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success StructuralCorrectness 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success trajectory 1 1.00 0.00 +1.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success UsedExpectedTools 1 1.00 1.00 0.00 - n/a -
workflow-editing: conditional-indexing-on-fetch-success ValidationPass 1 0.00 0.00 0.00 - n/a -
workflow-editing: core Criteria 6 0.67 0.83 -0.17 0.32 No -
workflow-editing: core EditPreservation 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core EditToolSuccess 6 0.67 0.83 -0.17 0.32 No -
workflow-editing: core Efficiency 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core Latency 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-editing: core StructuralCorrectness 6 0.96 1.00 -0.04 0.32 No -
workflow-editing: core trajectory 5 0.68 0.88 -0.20 0.33 No -
workflow-editing: core UsedExpectedTools 6 0.83 1.00 -0.17 0.32 No -
workflow-editing: core ValidationPass 6 0.67 0.83 -0.17 0.32 No -
workflow-es-ops: bulk-insert-structure BulkOperationsShape 3 0.33 0.33 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure Criteria 3 0.33 0.33 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure EditToolSuccess 3 0.33 0.33 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure Latency 3 0.63 0.46 +0.17 0.75 No -
workflow-es-ops: bulk-insert-structure NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure StructuralCorrectness 3 0.33 0.33 0.00 1.00 No -
workflow-es-ops: bulk-insert-structure trajectory 3 0.33 0.17 +0.17 0.35 No -
workflow-es-ops: bulk-insert-structure ValidationPass 3 0.33 0.33 0.00 1.00 No -
workflow-es-ops: delete-id-in-path Criteria 3 0.63 0.67 -0.03 0.90 No -
workflow-es-ops: delete-id-in-path EditToolSuccess 3 1.00 0.67 +0.33 0.35 No -
workflow-es-ops: delete-id-in-path Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path Latency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path StructuralCorrectness 3 1.00 0.67 +0.33 0.35 No -
workflow-es-ops: delete-id-in-path trajectory 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: delete-id-in-path ValidationPass 3 1.00 0.67 +0.33 0.35 No -
workflow-es-ops: index-with-id-in-path Criteria 3 0.33 0.15 +0.18 0.65 No -
workflow-es-ops: index-with-id-in-path EditToolSuccess 3 0.33 0.33 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Efficiency 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path Latency 3 0.94 0.33 +0.60 0.20 No -
workflow-es-ops: index-with-id-in-path NoErrors 3 1.00 1.00 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path StructuralCorrectness 3 0.33 0.33 0.00 1.00 No -
workflow-es-ops: index-with-id-in-path trajectory 3 0.67 1.00 -0.33 0.35 No -
workflow-es-ops: index-with-id-in-path ValidationPass 3 0.33 0.33 0.00 1.00 No -
workflow-es-ops: search-correctness Criteria 6 0.50 0.83 -0.33 0.14 No -
workflow-es-ops: search-correctness EditToolSuccess 6 0.50 0.83 -0.33 0.14 No -
workflow-es-ops: search-correctness Efficiency 6 0.97 1.00 -0.03 0.32 No -
workflow-es-ops: search-correctness Latency 6 0.83 1.00 -0.17 0.32 No -
workflow-es-ops: search-correctness NoErrors 6 1.00 1.00 0.00 1.00 No -
workflow-es-ops: search-correctness StructuralCorrectness 6 0.50 0.81 -0.31 0.15 No -
workflow-es-ops: search-correctness trajectory 6 0.67 1.00 -0.33 0.14 No -
workflow-es-ops: search-correctness ValidationPass 6 0.50 0.83 -0.33 0.14 No -
workflow-es-ops: store-with-index-create Criteria 2 0.00 0.50 -0.50 0.39 No -
workflow-es-ops: store-with-index-create EditToolSuccess 2 0.00 0.50 -0.50 0.39 No -
workflow-es-ops: store-with-index-create Efficiency 2 0.90 1.00 -0.10 0.39 No -
workflow-es-ops: store-with-index-create Latency 2 0.50 0.83 -0.33 0.61 No -
workflow-es-ops: store-with-index-create NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-es-ops: store-with-index-create StructuralCorrectness 2 0.00 0.50 -0.50 0.39 No -
workflow-es-ops: store-with-index-create trajectory 2 0.25 0.25 0.00 1.00 No -
workflow-es-ops: store-with-index-create ValidationPass 2 0.00 0.50 -0.50 0.39 No -
workflow-negative: impossible-trigger Criteria 2 0.33 0.17 +0.17 0.39 No -
workflow-negative: impossible-trigger Rejection 2 1.00 0.50 +0.50 0.39 No -
workflow-negative: out-of-scope Criteria 2 0.75 0.42 +0.33 0.39 No -
workflow-negative: out-of-scope Rejection 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsafe Criteria 2 0.75 0.75 0.00 1.00 No -
workflow-negative: unsafe Rejection 2 1.00 1.00 0.00 1.00 No -
workflow-negative: unsupported-features Criteria 3 0.17 0.33 -0.17 0.60 No -
workflow-negative: unsupported-features Rejection 3 0.67 0.33 +0.33 0.35 No -
workflow-self-correction: indent-mismatch Criteria 1 0.60 1.00 -0.40 - n/a -
workflow-self-correction: indent-mismatch NoErrors 1 1.00 1.00 0.00 - n/a -
workflow-self-correction: indent-mismatch SelfCorrection 1 0.75 0.75 0.00 - n/a -
workflow-self-correction: semantic Criteria 2 0.50 0.53 -0.03 0.39 No -
workflow-self-correction: semantic NoErrors 2 1.00 1.00 0.00 1.00 No -
workflow-self-correction: semantic SelfCorrection 2 0.25 0.63 -0.38 0.39 No -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::skill-selection-benchmark::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::skill-selection-benchmark::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 26 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
skill-selection-benchmark: significant-events-management Skill Selection 5 0.80 0.00 +0.80 0.01 Yes Improvement
No significant changes (25 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
skill-selection-benchmark: alert-analysis Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: dashboard-management Skill Selection 10 0.80 0.80 0.00 1.00 No -
skill-selection-benchmark: detection-rule-edit Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: elastic-defend-configuration-troubleshooting Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: entity-analytics Skill Selection 5 0.80 0.80 0.00 1.00 No -
skill-selection-benchmark: find-security-ml-jobs Skill Selection 5 0.60 1.00 -0.40 0.14 No -
skill-selection-benchmark: find-security-rules Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: graph-creation Skill Selection 5 0.80 0.80 0.00 1.00 No -
skill-selection-benchmark: ki-identification-management Skill Selection 5 0.20 0.00 +0.20 0.33 No -
skill-selection-benchmark: knowledge-indicators-management Skill Selection 5 0.40 0.00 +0.40 0.14 No -
skill-selection-benchmark: observability.investigation Skill Selection 13 0.85 0.85 0.00 1.00 No -
skill-selection-benchmark: search.catalog-ecommerce Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.elasticsearch-onboarding Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.elasticsearch-tutorial Skill Selection 5 0.80 0.80 0.00 1.00 No -
skill-selection-benchmark: search.keyword-search Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.rag-chatbot Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.use-case-library Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.vector-hybrid-search Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: service-map Skill Selection 9 0.89 0.89 0.00 1.00 No -
skill-selection-benchmark: siem-readiness Skill Selection 5 0.20 0.20 0.00 1.00 No -
skill-selection-benchmark: significant-events-memory Skill Selection 5 1.00 0.00 +1.00 1.00 No -
skill-selection-benchmark: skill-authoring Skill Selection 5 0.20 0.20 0.00 1.00 No -
skill-selection-benchmark: streams-management Skill Selection 5 0.80 1.00 -0.20 0.33 No -
skill-selection-benchmark: threat-hunting Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: visualization-creation Skill Selection 5 0.80 0.80 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::skill-selection-benchmark::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::skill-selection-benchmark::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
2 significant difference(s) detected out of 26 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
skill-selection-benchmark: observability.investigation Skill Selection 13 0.38 0.85 -0.46 0.02 Yes Regression
skill-selection-benchmark: significant-events-management Skill Selection 5 0.80 0.00 +0.80 0.01 Yes Improvement
No significant changes (24 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
skill-selection-benchmark: alert-analysis Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: dashboard-management Skill Selection 10 0.80 0.90 -0.10 0.32 No -
skill-selection-benchmark: detection-rule-edit Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: elastic-defend-configuration-troubleshooting Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: entity-analytics Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: find-security-ml-jobs Skill Selection 5 0.60 1.00 -0.40 0.14 No -
skill-selection-benchmark: find-security-rules Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: graph-creation Skill Selection 5 0.40 0.80 -0.40 0.14 No -
skill-selection-benchmark: ki-identification-management Skill Selection 5 0.20 0.00 +0.20 0.33 No -
skill-selection-benchmark: knowledge-indicators-management Skill Selection 5 0.40 0.00 +0.40 0.14 No -
skill-selection-benchmark: search.catalog-ecommerce Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.elasticsearch-onboarding Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.elasticsearch-tutorial Skill Selection 5 0.80 1.00 -0.20 0.33 No -
skill-selection-benchmark: search.keyword-search Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.rag-chatbot Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.use-case-library Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.vector-hybrid-search Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: service-map Skill Selection 9 0.89 0.89 0.00 1.00 No -
skill-selection-benchmark: siem-readiness Skill Selection 5 0.20 0.20 0.00 1.00 No -
skill-selection-benchmark: significant-events-memory Skill Selection 5 1.00 0.00 +1.00 1.00 No -
skill-selection-benchmark: skill-authoring Skill Selection 5 0.20 0.20 0.00 1.00 No -
skill-selection-benchmark: streams-management Skill Selection 5 0.80 1.00 -0.20 0.33 No -
skill-selection-benchmark: threat-hunting Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: visualization-creation Skill Selection 5 0.80 0.80 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::skill-selection-benchmark::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::skill-selection-benchmark::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 26 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
skill-selection-benchmark: significant-events-management Skill Selection 5 0.80 0.00 +0.80 0.01 Yes Improvement
No significant changes (25 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
skill-selection-benchmark: alert-analysis Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: dashboard-management Skill Selection 10 0.70 0.40 +0.30 0.07 No -
skill-selection-benchmark: detection-rule-edit Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: elastic-defend-configuration-troubleshooting Skill Selection 5 1.00 0.80 +0.20 0.33 No -
skill-selection-benchmark: entity-analytics Skill Selection 5 0.80 0.80 0.00 1.00 No -
skill-selection-benchmark: find-security-ml-jobs Skill Selection 5 0.80 1.00 -0.20 0.33 No -
skill-selection-benchmark: find-security-rules Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: graph-creation Skill Selection 5 0.40 0.80 -0.40 0.14 No -
skill-selection-benchmark: ki-identification-management Skill Selection 5 0.20 0.00 +0.20 0.33 No -
skill-selection-benchmark: knowledge-indicators-management Skill Selection 5 0.40 0.00 +0.40 0.14 No -
skill-selection-benchmark: observability.investigation Skill Selection 13 0.54 0.85 -0.31 0.15 No -
skill-selection-benchmark: search.catalog-ecommerce Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.elasticsearch-onboarding Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.elasticsearch-tutorial Skill Selection 5 0.80 0.80 0.00 1.00 No -
skill-selection-benchmark: search.keyword-search Skill Selection 5 0.80 0.80 0.00 1.00 No -
skill-selection-benchmark: search.rag-chatbot Skill Selection 5 1.00 0.80 +0.20 0.33 No -
skill-selection-benchmark: search.use-case-library Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.vector-hybrid-search Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: service-map Skill Selection 9 0.89 0.67 +0.22 0.15 No -
skill-selection-benchmark: siem-readiness Skill Selection 5 0.20 0.20 0.00 1.00 No -
skill-selection-benchmark: significant-events-memory Skill Selection 5 1.00 0.00 +1.00 1.00 No -
skill-selection-benchmark: skill-authoring Skill Selection 5 0.20 0.20 0.00 1.00 No -
skill-selection-benchmark: streams-management Skill Selection 5 0.80 0.80 0.00 1.00 No -
skill-selection-benchmark: threat-hunting Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: visualization-creation Skill Selection 5 0.60 0.60 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::skill-selection-benchmark::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::skill-selection-benchmark::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 26 comparisons.

Note: 1 unpaired (ran in only one experiment) skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
skill-selection-benchmark: significant-events-management Skill Selection 5 0.80 0.00 +0.80 0.01 Yes Improvement
No significant changes (25 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
skill-selection-benchmark: alert-analysis Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: dashboard-management Skill Selection 10 0.90 0.60 +0.30 0.07 No -
skill-selection-benchmark: detection-rule-edit Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: elastic-defend-configuration-troubleshooting Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: entity-analytics Skill Selection 5 0.80 1.00 -0.20 0.33 No -
skill-selection-benchmark: find-security-ml-jobs Skill Selection 5 0.80 1.00 -0.20 0.33 No -
skill-selection-benchmark: find-security-rules Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: graph-creation Skill Selection 5 0.80 0.80 0.00 1.00 No -
skill-selection-benchmark: ki-identification-management Skill Selection 5 0.20 0.00 +0.20 0.33 No -
skill-selection-benchmark: knowledge-indicators-management Skill Selection 5 0.40 0.00 +0.40 0.14 No -
skill-selection-benchmark: observability.investigation Skill Selection 12 1.00 0.92 +0.08 0.32 No -
skill-selection-benchmark: search.catalog-ecommerce Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.elasticsearch-onboarding Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.elasticsearch-tutorial Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.keyword-search Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.rag-chatbot Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.use-case-library Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.vector-hybrid-search Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: service-map Skill Selection 9 0.89 0.89 0.00 1.00 No -
skill-selection-benchmark: siem-readiness Skill Selection 5 0.20 0.20 0.00 1.00 No -
skill-selection-benchmark: significant-events-memory Skill Selection 5 1.00 0.00 +1.00 1.00 No -
skill-selection-benchmark: skill-authoring Skill Selection 5 0.20 0.20 0.00 1.00 No -
skill-selection-benchmark: streams-management Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: threat-hunting Skill Selection 5 1.00 0.80 +0.20 0.33 No -
skill-selection-benchmark: visualization-creation Skill Selection 5 0.80 0.80 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::skill-selection-benchmark::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::skill-selection-benchmark::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 26 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
skill-selection-benchmark: significant-events-management Skill Selection 5 0.80 0.00 +0.80 0.01 Yes Improvement
No significant changes (25 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
skill-selection-benchmark: alert-analysis Skill Selection 5 1.00 0.80 +0.20 0.33 No -
skill-selection-benchmark: dashboard-management Skill Selection 10 0.90 0.80 +0.10 0.32 No -
skill-selection-benchmark: detection-rule-edit Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: elastic-defend-configuration-troubleshooting Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: entity-analytics Skill Selection 5 0.80 1.00 -0.20 0.33 No -
skill-selection-benchmark: find-security-ml-jobs Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: find-security-rules Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: graph-creation Skill Selection 5 0.80 0.60 +0.20 0.33 No -
skill-selection-benchmark: ki-identification-management Skill Selection 5 0.20 0.00 +0.20 0.33 No -
skill-selection-benchmark: knowledge-indicators-management Skill Selection 5 0.40 0.00 +0.40 0.14 No -
skill-selection-benchmark: observability.investigation Skill Selection 13 0.31 0.77 -0.46 0.07 No -
skill-selection-benchmark: search.catalog-ecommerce Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.elasticsearch-onboarding Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.elasticsearch-tutorial Skill Selection 5 0.80 0.80 0.00 1.00 No -
skill-selection-benchmark: search.keyword-search Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.rag-chatbot Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.use-case-library Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.vector-hybrid-search Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: service-map Skill Selection 9 0.89 0.89 0.00 1.00 No -
skill-selection-benchmark: siem-readiness Skill Selection 5 0.20 0.20 0.00 1.00 No -
skill-selection-benchmark: significant-events-memory Skill Selection 5 1.00 0.00 +1.00 1.00 No -
skill-selection-benchmark: skill-authoring Skill Selection 5 0.20 0.20 0.00 1.00 No -
skill-selection-benchmark: streams-management Skill Selection 5 0.80 0.80 0.00 1.00 No -
skill-selection-benchmark: threat-hunting Skill Selection 5 1.00 0.80 +0.20 0.33 No -
skill-selection-benchmark: visualization-creation Skill Selection 5 0.80 0.80 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::skill-selection-benchmark::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::skill-selection-benchmark::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 26 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
skill-selection-benchmark: significant-events-management Skill Selection 5 0.80 0.00 +0.80 0.01 Yes Improvement
No significant changes (25 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
skill-selection-benchmark: alert-analysis Skill Selection 5 0.60 0.40 +0.20 0.33 No -
skill-selection-benchmark: dashboard-management Skill Selection 10 0.70 0.70 0.00 1.00 No -
skill-selection-benchmark: detection-rule-edit Skill Selection 5 0.80 1.00 -0.20 0.33 No -
skill-selection-benchmark: elastic-defend-configuration-troubleshooting Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: entity-analytics Skill Selection 5 0.20 0.80 -0.60 0.05 No -
skill-selection-benchmark: find-security-ml-jobs Skill Selection 5 0.80 0.80 0.00 1.00 No -
skill-selection-benchmark: find-security-rules Skill Selection 5 0.80 0.80 0.00 1.00 No -
skill-selection-benchmark: graph-creation Skill Selection 5 0.60 0.60 0.00 1.00 No -
skill-selection-benchmark: ki-identification-management Skill Selection 5 0.20 0.00 +0.20 0.33 No -
skill-selection-benchmark: knowledge-indicators-management Skill Selection 5 0.40 0.00 +0.40 0.14 No -
skill-selection-benchmark: observability.investigation Skill Selection 13 0.31 0.23 +0.08 0.71 No -
skill-selection-benchmark: search.catalog-ecommerce Skill Selection 5 1.00 0.60 +0.40 0.14 No -
skill-selection-benchmark: search.elasticsearch-onboarding Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.elasticsearch-tutorial Skill Selection 5 0.60 0.60 0.00 1.00 No -
skill-selection-benchmark: search.keyword-search Skill Selection 5 1.00 0.60 +0.40 0.14 No -
skill-selection-benchmark: search.rag-chatbot Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.use-case-library Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: search.vector-hybrid-search Skill Selection 5 1.00 1.00 0.00 1.00 No -
skill-selection-benchmark: service-map Skill Selection 9 0.89 0.56 +0.33 0.07 No -
skill-selection-benchmark: siem-readiness Skill Selection 5 0.20 0.20 0.00 1.00 No -
skill-selection-benchmark: significant-events-memory Skill Selection 5 1.00 0.00 +1.00 1.00 No -
skill-selection-benchmark: skill-authoring Skill Selection 5 0.20 0.20 0.00 1.00 No -
skill-selection-benchmark: streams-management Skill Selection 5 0.60 0.40 +0.20 0.58 No -
skill-selection-benchmark: threat-hunting Skill Selection 5 1.00 0.60 +0.40 0.14 No -
skill-selection-benchmark: visualization-creation Skill Selection 5 0.80 0.80 0.00 1.00 No -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::endpoint::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::endpoint::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (66 evaluator comparisons).

Note: 1 unpaired (ran in only one experiment), 9 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (66 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
endpoint: currently_healthy_endpoint_no_active_issue Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: endpoint_alert_needs_endpoint_exception_not_siem_explicit_prompt Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: endpoint_alerts_missing_output_shipping_failure_explicit_prompt Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: endpoint_exception_field_mismatch_explicit_prompt Criteria 1 0.86 1.00 -0.14 - n/a -
endpoint: incompatible antivirus detection Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: incompatible_aws_vpc_cni_ebpf_conflict Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: linux_high_cpu_monitoring_scripts Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: linux_missed_checkins_selinux_203_exec Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: missing_endpoint_list_stopped_united_transform Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: output_kafka_message_size_rejection_explicit_prompt Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: policy response failure Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: trusted_app_wrong_condition_field Criteria 1 0.71 0.57 +0.14 - n/a -
endpoint: windows_bsod_network_driver_regression_explicit_prompt Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: windows_high_cpu_authentication_events Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: windows_missed_checkins_crash_dump Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Cached Tokens 1 110084.00 320911.00 -210827.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Input Tokens 1 274059.00 622434.00 -348375.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Latency 1 5.94 11.31 -5.37 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Output Tokens 1 1970.00 5696.00 -3726.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Tool Calls 1 15.00 33.00 -18.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Input Tokens 1 54090.00 4113.00 +49977.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Latency 1 3.91 0.92 +2.99 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Output Tokens 1 349.00 70.00 +279.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Tool Calls 1 3.00 0.00 +3.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Input Tokens 1 4386.00 4119.00 +267.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Latency 1 1.55 0.72 +0.82 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Output Tokens 1 34.00 34.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Tool Calls 1 0.00 0.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Cached Tokens 1 467114.00 365962.00 +101152.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Criteria 1 1.00 0.67 +0.33 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Input Tokens 1 687364.00 702304.00 -14940.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Latency 1 13.07 10.20 +2.87 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Output Tokens 1 7977.00 6895.00 +1082.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Tool Calls 1 39.00 29.00 +10.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Trajectory 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Cached Tokens 1 269396.00 244906.00 +24490.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Input Tokens 1 590016.00 393738.00 +196278.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Latency 1 8.10 12.25 -4.15 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Output Tokens 1 5003.00 6248.00 -1245.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Tool Calls 1 24.00 23.00 +1.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Trajectory 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Cached Tokens 1 185961.00 128319.00 +57642.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Input Tokens 1 364431.00 248120.00 +116311.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Latency 1 13.88 11.69 +2.20 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Output Tokens 1 5669.00 3347.00 +2322.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Tool Calls 1 20.00 10.00 +10.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Trajectory 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Cached Tokens 1 129250.00 129775.00 -525.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Input Tokens 1 287484.00 316161.00 -28677.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Latency 1 7.96 7.80 +0.16 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Output Tokens 1 2669.00 3293.00 -624.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Tool Calls 1 14.00 14.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Trajectory 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::endpoint::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::endpoint::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (66 evaluator comparisons).

Note: 10 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (66 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
endpoint: currently_healthy_endpoint_no_active_issue Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: endpoint_alert_needs_endpoint_exception_not_siem_explicit_prompt Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: endpoint_alerts_missing_output_shipping_failure_explicit_prompt Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: endpoint_exception_field_mismatch_explicit_prompt Criteria 1 1.00 0.86 +0.14 - n/a -
endpoint: incompatible antivirus detection Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: incompatible_aws_vpc_cni_ebpf_conflict Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: linux_high_cpu_monitoring_scripts Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: linux_missed_checkins_selinux_203_exec Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: missing_endpoint_list_stopped_united_transform Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: output_kafka_message_size_rejection_explicit_prompt Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: policy response failure Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: trusted_app_wrong_condition_field Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: windows_bsod_network_driver_regression_explicit_prompt Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: windows_high_cpu_authentication_events Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: windows_missed_checkins_crash_dump Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Cached Tokens 1 40651.00 22290.00 +18361.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Input Tokens 1 900124.00 809523.00 +90601.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Latency 1 7.95 8.18 -0.24 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Output Tokens 1 5678.00 4439.00 +1239.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Tool Calls 1 51.00 38.00 +13.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Input Tokens 1 211584.00 50671.00 +160913.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Latency 1 2.39 0.42 +1.97 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Output Tokens 1 638.00 245.00 +393.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Tool Calls 1 7.00 0.00 +7.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Input Tokens 1 20460.00 4119.00 +16341.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Latency 1 0.33 1.56 -1.23 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Output Tokens 1 146.00 34.00 +112.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Tool Calls 1 0.00 0.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Cached Tokens 1 10191.00 6098.00 +4093.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Input Tokens 1 365646.00 454877.00 -89231.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Latency 1 15.56 18.12 -2.57 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Output Tokens 1 7211.00 6460.00 +751.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Tool Calls 1 19.00 17.00 +2.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Trajectory 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Cached Tokens 1 10200.00 14231.00 -4031.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Input Tokens 1 206996.00 357802.00 -150806.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Latency 1 11.86 11.77 +0.10 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Output Tokens 1 3709.00 5668.00 -1959.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Tool Calls 1 10.00 19.00 -9.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Trajectory 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Cached Tokens 1 16303.00 93284.00 -76981.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Input Tokens 1 242797.00 521326.00 -278529.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Latency 1 7.04 6.55 +0.49 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Output Tokens 1 3329.00 5585.00 -2256.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Tool Calls 1 16.00 19.00 -3.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Trajectory 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Cached Tokens 1 22423.00 4059.00 +18364.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Input Tokens 1 362497.00 214305.00 +148192.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Latency 1 9.46 9.05 +0.41 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Output Tokens 1 4549.00 3448.00 +1101.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Tool Calls 1 18.00 12.00 +6.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Trajectory 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::endpoint::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::endpoint::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (67 evaluator comparisons).

Note: 1 unpaired (ran in only one experiment), 7 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (67 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
endpoint: currently_healthy_endpoint_no_active_issue Criteria 1 1.00 0.57 +0.43 - n/a -
endpoint: endpoint_alert_needs_endpoint_exception_not_siem_explicit_prompt Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: endpoint_alerts_missing_output_shipping_failure_explicit_prompt Criteria 1 0.86 1.00 -0.14 - n/a -
endpoint: endpoint_exception_field_mismatch_explicit_prompt Criteria 1 1.00 0.86 +0.14 - n/a -
endpoint: incompatible antivirus detection Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: incompatible_aws_vpc_cni_ebpf_conflict Criteria 1 0.86 0.86 0.00 - n/a -
endpoint: linux_high_cpu_monitoring_scripts Criteria 1 1.00 0.86 +0.14 - n/a -
endpoint: linux_missed_checkins_selinux_203_exec Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: missing_endpoint_list_stopped_united_transform Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: output_kafka_message_size_rejection_explicit_prompt Criteria 1 1.00 0.86 +0.14 - n/a -
endpoint: policy response failure Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: trusted_app_wrong_condition_field Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: windows_bsod_network_driver_regression_explicit_prompt Criteria 1 1.00 0.86 +0.14 - n/a -
endpoint: windows_high_cpu_authentication_events Criteria 1 0.86 0.86 0.00 - n/a -
endpoint: windows_missed_checkins_crash_dump Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Cached Tokens 1 484691.00 582284.00 -97593.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Input Tokens 1 990218.00 1228794.00 -238576.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Latency 1 14.13 16.80 -2.67 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Output Tokens 1 3890.00 4721.00 -831.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Tool Calls 1 48.00 50.00 -2.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Input Tokens 1 55530.00 4113.00 +51417.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Latency 1 1.88 1.31 +0.57 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Output Tokens 1 128.00 63.00 +65.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Tool Calls 1 3.00 0.00 +3.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Cached Tokens 1 21340.00 10666.00 +10674.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Input Tokens 1 29458.00 29184.00 +274.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Latency 1 0.79 0.94 -0.15 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Output Tokens 1 84.00 158.00 -74.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Tool Calls 1 0.00 0.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Cached Tokens 1 506219.00 454803.00 +51416.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Criteria 1 0.67 0.67 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Input Tokens 1 791108.00 721263.00 +69845.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Latency 1 11.18 5.19 +5.99 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Output Tokens 1 3522.00 5278.00 -1756.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Tool Calls 1 30.00 26.00 +4.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Trajectory 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Cached Tokens 1 629109.00 349778.00 +279331.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Input Tokens 1 1084049.00 474889.00 +609160.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Latency 1 5.54 2.61 +2.93 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Output Tokens 1 9135.00 2990.00 +6145.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Tool Calls 1 28.00 28.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Trajectory 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Cached Tokens 1 243058.00 219509.00 +23549.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Input Tokens 1 380371.00 312164.00 +68207.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Latency 1 5.07 4.29 +0.78 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Output Tokens 1 2351.00 2193.00 +158.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Tool Calls 1 15.00 19.00 -4.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Trajectory 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Cached Tokens 1 42565.00 157986.00 -115421.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Input Tokens 1 94480.00 356098.00 -261618.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Latency 1 2.82 3.78 -0.96 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Output Tokens 1 469.00 1970.00 -1501.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Tool Calls 1 4.00 15.00 -11.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Trajectory 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::endpoint::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::endpoint::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (67 evaluator comparisons).

Note: 7 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (67 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
endpoint: currently_healthy_endpoint_no_active_issue Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: endpoint_alert_needs_endpoint_exception_not_siem_explicit_prompt Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: endpoint_alerts_missing_output_shipping_failure_explicit_prompt Criteria 1 0.71 0.86 -0.14 - n/a -
endpoint: endpoint_exception_field_mismatch_explicit_prompt Criteria 1 0.86 1.00 -0.14 - n/a -
endpoint: incompatible antivirus detection Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: incompatible_aws_vpc_cni_ebpf_conflict Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: linux_high_cpu_monitoring_scripts Criteria 1 0.86 0.86 0.00 - n/a -
endpoint: linux_missed_checkins_selinux_203_exec Criteria 1 0.71 0.71 0.00 - n/a -
endpoint: missing_endpoint_list_stopped_united_transform Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: output_kafka_message_size_rejection_explicit_prompt Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: policy response failure Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: trusted_app_wrong_condition_field Criteria 1 0.71 0.71 0.00 - n/a -
endpoint: windows_bsod_network_driver_regression_explicit_prompt Criteria 1 1.00 0.86 +0.14 - n/a -
endpoint: windows_high_cpu_authentication_events Criteria 1 0.86 0.71 +0.14 - n/a -
endpoint: windows_missed_checkins_crash_dump Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Cached Tokens 1 187777.00 240402.00 -52625.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Input Tokens 1 323909.00 504068.00 -180159.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Latency 1 14.30 7.04 +7.26 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Output Tokens 1 4069.00 5243.00 -1174.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Tool Calls 1 16.00 27.00 -11.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Cached Tokens 1 7085.00 10632.00 -3547.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Input Tokens 1 16923.00 16687.00 +236.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Latency 1 3.54 3.93 -0.38 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Output Tokens 1 229.00 335.00 -106.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Tool Calls 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Input Tokens 1 4386.00 16611.00 -12225.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Latency 1 1.86 0.76 +1.10 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Output Tokens 1 33.00 359.00 -326.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Tool Calls 1 0.00 0.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Cached Tokens 1 535319.00 55492.00 +479827.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Input Tokens 1 1210079.00 138359.00 +1071720.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Latency 1 6.22 6.98 -0.76 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Output Tokens 1 4767.00 2259.00 +2508.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Tool Calls 1 30.00 6.00 +24.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Trajectory 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Cached Tokens 1 335157.00 75609.00 +259548.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Input Tokens 1 718478.00 192299.00 +526179.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Latency 1 7.61 4.96 +2.66 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Output Tokens 1 4704.00 2624.00 +2080.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Tool Calls 1 23.00 7.00 +16.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Trajectory 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Cached Tokens 1 264050.00 63504.00 +200546.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Input Tokens 1 595860.00 217881.00 +377979.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Latency 1 6.86 5.61 +1.25 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Output Tokens 1 3890.00 1684.00 +2206.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Tool Calls 1 14.00 5.00 +9.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Trajectory 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Cached Tokens 1 241374.00 78412.00 +162962.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Input Tokens 1 400380.00 193806.00 +206574.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Latency 1 5.39 5.46 -0.06 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Output Tokens 1 2511.00 2006.00 +505.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Tool Calls 1 11.00 6.00 +5.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Trajectory 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::endpoint::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::endpoint::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (67 evaluator comparisons).

Note: 7 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (67 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
endpoint: currently_healthy_endpoint_no_active_issue Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: endpoint_alert_needs_endpoint_exception_not_siem_explicit_prompt Criteria 1 0.86 1.00 -0.14 - n/a -
endpoint: endpoint_alerts_missing_output_shipping_failure_explicit_prompt Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: endpoint_exception_field_mismatch_explicit_prompt Criteria 1 1.00 0.86 +0.14 - n/a -
endpoint: incompatible antivirus detection Criteria 1 1.00 0.75 +0.25 - n/a -
endpoint: incompatible_aws_vpc_cni_ebpf_conflict Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: linux_high_cpu_monitoring_scripts Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: linux_missed_checkins_selinux_203_exec Criteria 1 0.71 0.71 0.00 - n/a -
endpoint: missing_endpoint_list_stopped_united_transform Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: output_kafka_message_size_rejection_explicit_prompt Criteria 1 0.86 0.86 0.00 - n/a -
endpoint: policy response failure Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: trusted_app_wrong_condition_field Criteria 1 1.00 0.57 +0.43 - n/a -
endpoint: windows_bsod_network_driver_regression_explicit_prompt Criteria 1 1.00 1.00 0.00 - n/a -
endpoint: windows_high_cpu_authentication_events Criteria 1 1.00 0.86 +0.14 - n/a -
endpoint: windows_missed_checkins_crash_dump Criteria 1 0.86 1.00 -0.14 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Cached Tokens 1 125786.00 262313.00 -136527.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Input Tokens 1 264377.00 440897.00 -176520.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Latency 1 5.26 15.43 -10.17 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Output Tokens 1 2378.00 5203.00 -2825.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Tool Calls 1 14.00 27.00 -13.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Cached Tokens 1 11904.00 12032.00 -128.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Input Tokens 1 16553.00 16319.00 +234.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Latency 1 2.90 2.25 +0.65 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Output Tokens 1 112.00 114.00 -2.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Tool Calls 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Input Tokens 1 241.00 16242.00 -16001.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Latency 1 0.65 5.14 -4.49 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Output Tokens 1 19.00 94.00 -75.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Tool Calls 1 0.00 1.00 -1.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Cached Tokens 1 258879.00 190433.00 +68446.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Input Tokens 1 359650.00 290825.00 +68825.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Latency 1 6.44 6.72 -0.29 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Output Tokens 1 3914.00 3432.00 +482.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Tool Calls 1 13.00 13.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Trajectory 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Cached Tokens 1 289182.00 351606.00 -62424.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Input Tokens 1 390821.00 527905.00 -137084.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Latency 1 9.18 8.53 +0.65 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Output Tokens 1 4813.00 5571.00 -758.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Tool Calls 1 12.00 17.00 -5.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Trajectory 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Cached Tokens 1 193048.00 187183.00 +5865.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Input Tokens 1 317741.00 297722.00 +20019.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Latency 1 8.29 7.59 +0.69 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Output Tokens 1 4893.00 3278.00 +1615.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Tool Calls 1 11.00 6.00 +5.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Trajectory 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Cached Tokens 1 355874.00 392212.00 -36338.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Input Tokens 1 483659.00 583102.00 -99443.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Latency 1 12.75 12.10 +0.64 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Output Tokens 1 7770.00 7284.00 +486.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Tool Calls 1 19.00 14.00 +5.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Trajectory 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::endpoint::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::endpoint::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (66 evaluator comparisons).

Note: 1 unpaired (ran in only one experiment), 9 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (66 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
endpoint: currently_healthy_endpoint_no_active_issue Criteria 1 0.29 0.29 0.00 - n/a -
endpoint: endpoint_alert_needs_endpoint_exception_not_siem_explicit_prompt Criteria 1 0.14 0.14 0.00 - n/a -
endpoint: endpoint_alerts_missing_output_shipping_failure_explicit_prompt Criteria 1 0.14 0.14 0.00 - n/a -
endpoint: endpoint_exception_field_mismatch_explicit_prompt Criteria 1 0.71 0.14 +0.57 - n/a -
endpoint: incompatible antivirus detection Criteria 1 0.25 0.25 0.00 - n/a -
endpoint: incompatible_aws_vpc_cni_ebpf_conflict Criteria 1 0.14 0.14 0.00 - n/a -
endpoint: linux_high_cpu_monitoring_scripts Criteria 1 0.14 0.29 -0.14 - n/a -
endpoint: linux_missed_checkins_selinux_203_exec Criteria 1 0.14 0.14 0.00 - n/a -
endpoint: missing_endpoint_list_stopped_united_transform Criteria 1 0.20 0.20 0.00 - n/a -
endpoint: output_kafka_message_size_rejection_explicit_prompt Criteria 1 0.14 0.14 0.00 - n/a -
endpoint: policy response failure Criteria 1 0.25 0.25 0.00 - n/a -
endpoint: trusted_app_wrong_condition_field Criteria 1 0.14 0.29 -0.14 - n/a -
endpoint: windows_bsod_network_driver_regression_explicit_prompt Criteria 1 0.14 0.14 0.00 - n/a -
endpoint: windows_high_cpu_authentication_events Criteria 1 0.14 0.14 0.00 - n/a -
endpoint: windows_missed_checkins_crash_dump Criteria 1 0.14 0.14 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Input Tokens 1 17024.00 124695.00 -107671.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Latency 1 23.32 2.95 +20.37 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Output Tokens 1 1042.00 1215.00 -173.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-antivirus Tool Calls 1 10.00 7.00 +3.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Cached Tokens 1 11143.00 12484.00 -1341.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Input Tokens 1 16928.00 39942.00 -23014.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Latency 1 2.38 2.30 +0.08 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Output Tokens 1 174.00 355.00 -181.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-dashboards Tool Calls 1 1.00 2.00 -1.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Input Tokens 1 4386.00 29179.00 -24793.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Latency 1 1.66 0.85 +0.81 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Output Tokens 1 34.00 565.00 -531.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-distractor-weather Tool Calls 1 0.00 0.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Cached Tokens 1 12058.00 12484.00 -426.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Criteria 1 1.00 0.33 +0.67 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Input Tokens 1 17228.00 30916.00 -13688.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Latency 1 3.06 4.64 -1.58 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Output Tokens 1 448.00 533.00 -85.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Tool Calls 1 1.00 2.00 -1.00 - n/a -
security: endpoint-forensic-analysis-smoke-lateral Trajectory 1 0.00 0.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Cached Tokens 1 12058.00 12486.00 -428.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Criteria 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Input Tokens 1 17111.00 29659.00 -12548.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Latency 1 2.20 4.05 -1.85 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Output Tokens 1 292.00 329.00 -37.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Tool Calls 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-patient-zero Trajectory 1 0.00 0.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Cached Tokens 1 12058.00 65044.00 -52986.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Criteria 1 0.75 1.00 -0.25 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Input Tokens 1 17135.00 529282.00 -512147.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Latency 1 1.67 4.16 -2.49 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Output Tokens 1 242.00 6611.00 -6369.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Tool Calls 1 1.00 14.00 -13.00 - n/a -
security: endpoint-forensic-analysis-smoke-persistence Trajectory 1 0.00 1.00 -1.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Cached Tokens 1 12058.00 12468.00 -410.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Criteria 1 0.50 0.50 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Input Tokens 1 17114.00 16877.00 +237.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Latency 1 2.07 3.22 -1.16 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Output Tokens 1 300.00 521.00 -221.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Skill Invoked (endpoint-forensic-analysis) 1 1.00 1.00 0.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Tool Calls 1 1.00 5.00 -4.00 - n/a -
security: endpoint-forensic-analysis-smoke-timeline Trajectory 1 0.00 0.00 0.00 - n/a -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::entity-analytics-v2::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::entity-analytics-v2::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (100 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (100 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics-v2: attachment side-effects Attachments 3 0.67 0.67 0.00 1.00 No -
entity-analytics-v2: attachment side-effects Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: attachment side-effects Groundedness 3 0.93 0.79 +0.14 0.15 No -
entity-analytics-v2: attachment side-effects Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: attachment side-effects ToolCalls 3 0.67 0.67 0.00 1.00 No -
entity-analytics-v2: entity relationship history Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history Criteria 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history Groundedness 4 0.97 0.97 +0.00 0.80 No -
entity-analytics-v2: entity relationship history Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history ToolCalls 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: enumerate watchlists Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: enumerate watchlists Criteria 1 0.67 0.67 0.00 - n/a -
entity-analytics-v2: enumerate watchlists Groundedness 1 0.99 1.00 -0.01 - n/a -
entity-analytics-v2: enumerate watchlists Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: enumerate watchlists ToolCalls 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: get entity Attachments 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity Criteria 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity Groundedness 6 0.93 0.88 +0.05 0.47 No -
entity-analytics-v2: get entity Sequence Accuracy 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity ToolCalls 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview Groundedness 3 0.87 0.73 +0.14 0.30 No -
entity-analytics-v2: graph preview Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Criteria 4 0.79 0.79 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Groundedness 4 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) ToolCalls 4 0.75 0.75 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Criteria 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Groundedness 4 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) ToolCalls 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing Criteria 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing Groundedness 4 0.95 0.90 +0.05 0.44 No -
entity-analytics-v2: multi-skill routing Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing ToolCalls 4 0.75 0.75 0.00 1.00 No -
entity-analytics-v2: risk score history Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: risk score history Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: risk score history Groundedness 3 0.97 0.97 +0.00 0.97 No -
entity-analytics-v2: risk score history Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: risk score history ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities Attachments 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities Criteria 9 0.93 1.00 -0.07 0.15 No -
entity-analytics-v2: search entities Groundedness 9 0.88 0.91 -0.04 0.46 No -
entity-analytics-v2: search entities Sequence Accuracy 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities ToolCalls 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Groundedness 3 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Groundedness 2 0.85 0.88 -0.03 0.65 No -
entity-analytics-v2: UI navigation — asset criticality bulk Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk ToolCalls 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — engine status Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status Criteria 1 1.00 0.00 +1.00 - n/a -
entity-analytics-v2: UI navigation — engine status Groundedness 1 0.71 0.93 -0.23 - n/a -
entity-analytics-v2: UI navigation — engine status Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status ToolCalls 1 1.00 0.00 +1.00 - n/a -
entity-analytics-v2: UI navigation — entity analytics settings Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings Criteria 2 0.17 0.17 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings Groundedness 2 0.17 0.11 +0.06 0.39 No -
entity-analytics-v2: UI navigation — entity analytics settings Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity resolution bulk Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Criteria 1 0.33 0.00 +0.33 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Groundedness 1 1.00 0.00 +1.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk ToolCalls 1 0.00 0.00 0.00 - n/a -
entity-analytics-v2: UI navigation — risk engine Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — risk engine Criteria 2 0.50 0.00 +0.50 0.39 No -
entity-analytics-v2: UI navigation — risk engine Groundedness 2 0.50 0.00 +0.50 0.39 No -
entity-analytics-v2: UI navigation — risk engine Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — risk engine ToolCalls 2 0.50 0.00 +0.50 0.39 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Groundedness 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead Groundedness 2 0.99 1.00 -0.01 0.39 No -
entity-analytics: dismiss_lead Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead ToolCalls 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Groundedness 2 0.00 0.00 0.00 1.00 No -
entity-analytics: generate_leads Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads ToolCalls 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Groundedness 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads ToolCalls 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::entity-analytics-v2::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::entity-analytics-v2::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (100 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (100 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics-v2: attachment side-effects Attachments 3 0.67 0.67 0.00 1.00 No -
entity-analytics-v2: attachment side-effects Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: attachment side-effects Groundedness 3 0.80 0.80 -0.00 0.99 No -
entity-analytics-v2: attachment side-effects Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: attachment side-effects ToolCalls 3 0.67 0.67 0.00 1.00 No -
entity-analytics-v2: entity relationship history Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history Criteria 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history Groundedness 4 0.90 0.91 -0.01 0.84 No -
entity-analytics-v2: entity relationship history Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history ToolCalls 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: enumerate watchlists Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: enumerate watchlists Criteria 1 0.67 0.67 0.00 - n/a -
entity-analytics-v2: enumerate watchlists Groundedness 1 0.97 1.00 -0.03 - n/a -
entity-analytics-v2: enumerate watchlists Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: enumerate watchlists ToolCalls 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: get entity Attachments 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity Criteria 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity Groundedness 6 0.83 0.89 -0.06 0.12 No -
entity-analytics-v2: get entity Sequence Accuracy 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity ToolCalls 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview Groundedness 3 0.85 0.80 +0.05 0.64 No -
entity-analytics-v2: graph preview Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Criteria 4 0.79 0.79 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Groundedness 4 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) ToolCalls 4 0.75 0.75 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Criteria 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Groundedness 4 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) ToolCalls 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing Criteria 4 1.00 0.88 +0.13 0.33 No -
entity-analytics-v2: multi-skill routing Groundedness 4 0.81 0.85 -0.04 0.60 No -
entity-analytics-v2: multi-skill routing Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing ToolCalls 4 0.75 0.50 +0.25 0.33 No -
entity-analytics-v2: risk score history Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: risk score history Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: risk score history Groundedness 3 0.86 0.78 +0.08 0.18 No -
entity-analytics-v2: risk score history Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: risk score history ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities Attachments 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities Criteria 9 1.00 0.91 +0.09 0.16 No -
entity-analytics-v2: search entities Groundedness 9 0.83 0.83 -0.00 0.98 No -
entity-analytics-v2: search entities Sequence Accuracy 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities ToolCalls 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Groundedness 3 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Criteria 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Groundedness 2 0.57 0.57 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — engine status Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status Criteria 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status Groundedness 1 0.79 0.97 -0.18 - n/a -
entity-analytics-v2: UI navigation — engine status Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status ToolCalls 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity analytics settings Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings Criteria 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings Groundedness 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity resolution bulk Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Criteria 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Groundedness 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk ToolCalls 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — risk engine Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — risk engine Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — risk engine Groundedness 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — risk engine Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — risk engine ToolCalls 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Groundedness 3 1.00 0.99 +0.01 0.35 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead Groundedness 2 1.00 1.00 -0.00 0.39 No -
entity-analytics: dismiss_lead Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead ToolCalls 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Groundedness 2 0.00 0.00 0.00 1.00 No -
entity-analytics: generate_leads Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads ToolCalls 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Groundedness 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads ToolCalls 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::entity-analytics-v2::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::entity-analytics-v2::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (100 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (100 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics-v2: attachment side-effects Attachments 3 1.00 0.67 +0.33 0.35 No -
entity-analytics-v2: attachment side-effects Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: attachment side-effects Groundedness 3 0.88 0.75 +0.12 0.38 No -
entity-analytics-v2: attachment side-effects Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: attachment side-effects ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history Criteria 4 1.00 0.88 +0.13 0.33 No -
entity-analytics-v2: entity relationship history Groundedness 4 0.98 0.91 +0.07 0.37 No -
entity-analytics-v2: entity relationship history Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history ToolCalls 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: enumerate watchlists Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: enumerate watchlists Criteria 1 0.67 0.67 0.00 - n/a -
entity-analytics-v2: enumerate watchlists Groundedness 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: enumerate watchlists Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: enumerate watchlists ToolCalls 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: get entity Attachments 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity Criteria 6 1.00 0.97 +0.03 0.32 No -
entity-analytics-v2: get entity Groundedness 6 0.96 0.98 -0.02 0.40 No -
entity-analytics-v2: get entity Sequence Accuracy 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity ToolCalls 6 0.83 1.00 -0.17 0.32 No -
entity-analytics-v2: graph preview Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview Criteria 3 1.00 0.89 +0.11 0.35 No -
entity-analytics-v2: graph preview Groundedness 3 0.85 0.81 +0.04 0.74 No -
entity-analytics-v2: graph preview Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview ToolCalls 3 1.00 0.67 +0.33 0.35 No -
entity-analytics-v2: manage watchlists (entity membership) Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Criteria 4 0.79 0.79 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Groundedness 4 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) ToolCalls 4 0.75 0.75 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Criteria 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Groundedness 4 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) ToolCalls 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing Criteria 4 0.75 0.88 -0.13 0.33 No -
entity-analytics-v2: multi-skill routing Groundedness 4 0.98 0.93 +0.05 0.10 No -
entity-analytics-v2: multi-skill routing Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing ToolCalls 4 0.50 0.75 -0.25 0.33 No -
entity-analytics-v2: risk score history Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: risk score history Criteria 3 0.89 1.00 -0.11 0.35 No -
entity-analytics-v2: risk score history Groundedness 3 0.93 0.84 +0.08 0.41 No -
entity-analytics-v2: risk score history Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: risk score history ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities Attachments 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities Criteria 9 0.89 0.96 -0.07 0.15 No -
entity-analytics-v2: search entities Groundedness 9 0.81 0.86 -0.05 0.32 No -
entity-analytics-v2: search entities Sequence Accuracy 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities ToolCalls 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Groundedness 3 0.00 0.33 -0.33 0.35 No -
entity-analytics-v2: set_asset_criticality Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality ToolCalls 3 1.00 0.22 +0.78 0.05 No -
entity-analytics-v2: UI navigation — asset criticality bulk Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Criteria 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Groundedness 2 0.50 0.85 -0.35 0.58 No -
entity-analytics-v2: UI navigation — asset criticality bulk Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — engine status Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status Criteria 1 0.00 0.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status Groundedness 1 0.97 1.00 -0.03 - n/a -
entity-analytics-v2: UI navigation — engine status Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status ToolCalls 1 0.00 0.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity analytics settings Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings Criteria 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings Groundedness 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity resolution bulk Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Criteria 1 0.00 0.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Groundedness 1 0.32 1.00 -0.68 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk ToolCalls 1 0.00 0.00 0.00 - n/a -
entity-analytics-v2: UI navigation — risk engine Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — risk engine Criteria 2 0.50 1.00 -0.50 0.39 No -
entity-analytics-v2: UI navigation — risk engine Groundedness 2 0.98 1.00 -0.02 0.39 No -
entity-analytics-v2: UI navigation — risk engine Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — risk engine ToolCalls 2 0.50 1.00 -0.50 0.39 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Groundedness 3 0.99 0.99 +0.00 0.85 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead Groundedness 2 0.99 0.99 +0.00 0.91 No -
entity-analytics: dismiss_lead Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead ToolCalls 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Groundedness 2 0.50 0.00 +0.50 0.39 No -
entity-analytics: generate_leads Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads ToolCalls 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Groundedness 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads ToolCalls 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::entity-analytics-v2::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::entity-analytics-v2::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (100 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (100 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics-v2: attachment side-effects Attachments 3 0.67 1.00 -0.33 0.35 No -
entity-analytics-v2: attachment side-effects Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: attachment side-effects Groundedness 3 0.78 0.94 -0.16 0.39 No -
entity-analytics-v2: attachment side-effects Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: attachment side-effects ToolCalls 3 0.67 1.00 -0.33 0.35 No -
entity-analytics-v2: entity relationship history Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history Criteria 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history Groundedness 4 0.75 1.00 -0.25 0.33 No -
entity-analytics-v2: entity relationship history Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history ToolCalls 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: enumerate watchlists Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: enumerate watchlists Criteria 1 0.67 0.67 0.00 - n/a -
entity-analytics-v2: enumerate watchlists Groundedness 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: enumerate watchlists Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: enumerate watchlists ToolCalls 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: get entity Attachments 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity Criteria 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity Groundedness 6 0.98 0.98 +0.00 0.84 No -
entity-analytics-v2: get entity Sequence Accuracy 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity ToolCalls 6 0.83 0.83 0.00 1.00 No -
entity-analytics-v2: graph preview Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview Groundedness 3 0.79 0.88 -0.09 0.23 No -
entity-analytics-v2: graph preview Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Criteria 4 0.79 0.79 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Groundedness 4 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) ToolCalls 4 0.75 0.75 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Criteria 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Groundedness 4 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) ToolCalls 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing Criteria 4 0.88 0.75 +0.13 0.33 No -
entity-analytics-v2: multi-skill routing Groundedness 4 0.99 1.00 -0.01 0.52 No -
entity-analytics-v2: multi-skill routing Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing ToolCalls 4 0.50 0.50 0.00 1.00 No -
entity-analytics-v2: risk score history Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: risk score history Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: risk score history Groundedness 3 0.92 0.89 +0.02 0.85 No -
entity-analytics-v2: risk score history Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: risk score history ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities Attachments 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities Criteria 9 0.96 0.93 +0.04 0.32 No -
entity-analytics-v2: search entities Groundedness 9 0.94 0.87 +0.07 0.46 No -
entity-analytics-v2: search entities Sequence Accuracy 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities ToolCalls 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Groundedness 3 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Criteria 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Groundedness 2 1.00 0.75 +0.25 0.39 No -
entity-analytics-v2: UI navigation — asset criticality bulk Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — engine status Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status Criteria 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status Groundedness 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status ToolCalls 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity analytics settings Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings Criteria 2 0.33 0.00 +0.33 0.39 No -
entity-analytics-v2: UI navigation — entity analytics settings Groundedness 2 0.50 0.00 +0.50 0.39 No -
entity-analytics-v2: UI navigation — entity analytics settings Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings ToolCalls 2 0.50 0.00 +0.50 0.39 No -
entity-analytics-v2: UI navigation — entity resolution bulk Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Criteria 1 0.00 0.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Groundedness 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk ToolCalls 1 0.00 0.00 0.00 - n/a -
entity-analytics-v2: UI navigation — risk engine Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — risk engine Criteria 2 1.00 0.50 +0.50 0.39 No -
entity-analytics-v2: UI navigation — risk engine Groundedness 2 1.00 0.50 +0.50 0.39 No -
entity-analytics-v2: UI navigation — risk engine Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — risk engine ToolCalls 2 1.00 0.50 +0.50 0.39 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Criteria 3 0.89 0.89 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Groundedness 3 0.98 1.00 -0.02 0.35 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead Groundedness 2 1.00 0.66 +0.34 0.39 No -
entity-analytics: dismiss_lead Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead ToolCalls 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Groundedness 2 0.00 0.00 0.00 1.00 No -
entity-analytics: generate_leads Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads ToolCalls 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Groundedness 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads ToolCalls 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::entity-analytics-v2::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::entity-analytics-v2::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (100 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (100 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics-v2: attachment side-effects Attachments 3 0.67 1.00 -0.33 0.35 No -
entity-analytics-v2: attachment side-effects Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: attachment side-effects Groundedness 3 0.99 1.00 -0.00 0.43 No -
entity-analytics-v2: attachment side-effects Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: attachment side-effects ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history Criteria 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history Groundedness 4 0.74 0.98 -0.23 0.34 No -
entity-analytics-v2: entity relationship history Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history ToolCalls 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: enumerate watchlists Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: enumerate watchlists Criteria 1 0.67 1.00 -0.33 - n/a -
entity-analytics-v2: enumerate watchlists Groundedness 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: enumerate watchlists Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: enumerate watchlists ToolCalls 1 0.00 0.00 0.00 - n/a -
entity-analytics-v2: get entity Attachments 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity Criteria 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity Groundedness 6 0.97 0.98 -0.01 0.35 No -
entity-analytics-v2: get entity Sequence Accuracy 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity ToolCalls 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview Groundedness 3 1.00 0.92 +0.08 0.32 No -
entity-analytics-v2: graph preview Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Criteria 4 0.79 0.79 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Groundedness 4 0.00 0.24 -0.24 0.33 No -
entity-analytics-v2: manage watchlists (entity membership) Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) ToolCalls 4 0.50 0.50 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Criteria 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Groundedness 4 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) ToolCalls 4 0.75 0.75 0.00 1.00 No -
entity-analytics-v2: multi-skill routing Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing Criteria 4 0.88 0.88 0.00 1.00 No -
entity-analytics-v2: multi-skill routing Groundedness 4 0.99 0.74 +0.25 0.34 No -
entity-analytics-v2: multi-skill routing Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing ToolCalls 4 0.50 0.50 0.00 1.00 No -
entity-analytics-v2: risk score history Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: risk score history Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: risk score history Groundedness 3 0.98 0.99 -0.00 0.84 No -
entity-analytics-v2: risk score history Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: risk score history ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities Attachments 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities Criteria 9 1.00 0.96 +0.04 0.32 No -
entity-analytics-v2: search entities Groundedness 9 0.98 0.99 -0.01 0.24 No -
entity-analytics-v2: search entities Sequence Accuracy 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities ToolCalls 9 0.78 1.00 -0.22 0.15 No -
entity-analytics-v2: set_asset_criticality Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Criteria 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Groundedness 3 0.00 0.33 -0.33 0.35 No -
entity-analytics-v2: set_asset_criticality Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality ToolCalls 3 0.67 0.33 +0.33 0.35 No -
entity-analytics-v2: UI navigation — asset criticality bulk Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Criteria 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Groundedness 2 1.00 0.68 +0.32 0.39 No -
entity-analytics-v2: UI navigation — asset criticality bulk Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — engine status Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status Criteria 1 0.00 1.00 -1.00 - n/a -
entity-analytics-v2: UI navigation — engine status Groundedness 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status ToolCalls 1 0.00 1.00 -1.00 - n/a -
entity-analytics-v2: UI navigation — entity analytics settings Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings Criteria 2 0.00 0.25 -0.25 0.39 No -
entity-analytics-v2: UI navigation — entity analytics settings Groundedness 2 0.98 0.49 +0.49 0.39 No -
entity-analytics-v2: UI navigation — entity analytics settings Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity resolution bulk Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Criteria 1 0.00 0.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Groundedness 1 0.00 0.32 -0.32 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk ToolCalls 1 0.00 0.00 0.00 - n/a -
entity-analytics-v2: UI navigation — risk engine Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — risk engine Criteria 2 0.00 0.25 -0.25 0.39 No -
entity-analytics-v2: UI navigation — risk engine Groundedness 2 0.49 0.93 -0.45 0.42 No -
entity-analytics-v2: UI navigation — risk engine Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — risk engine ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Criteria 3 0.78 0.89 -0.11 0.35 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Groundedness 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout ToolCalls 3 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead Criteria 2 1.00 0.75 +0.25 0.39 No -
entity-analytics: dismiss_lead Groundedness 2 0.99 0.99 -0.01 0.39 No -
entity-analytics: dismiss_lead Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead ToolCalls 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Groundedness 2 0.00 0.00 0.00 1.00 No -
entity-analytics: generate_leads Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads ToolCalls 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Groundedness 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads ToolCalls 2 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::entity-analytics-v2::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::entity-analytics-v2::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
6 significant difference(s) detected out of 100 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics-v2: get entity Criteria 6 0.52 0.94 -0.42 0.00 Yes Regression
entity-analytics-v2: graph preview Criteria 3 0.64 1.00 -0.36 0.03 Yes Regression
entity-analytics-v2: search entities Criteria 9 0.60 0.91 -0.31 0.00 Yes Regression
entity-analytics-v2: search entities ToolCalls 9 0.11 0.89 -0.78 0.00 Yes Regression
entity-analytics-v2: set_asset_criticality Criteria 3 0.33 1.00 -0.67 0.04 Yes Regression
entity-analytics-v2: set_asset_criticality Groundedness 3 0.60 0.00 +0.60 0.02 Yes Improvement
No significant changes (94 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics-v2: attachment side-effects Attachments 3 0.33 0.33 0.00 1.00 No -
entity-analytics-v2: attachment side-effects Criteria 3 0.50 0.50 0.00 1.00 No -
entity-analytics-v2: attachment side-effects Groundedness 3 0.95 0.99 -0.04 0.44 No -
entity-analytics-v2: attachment side-effects Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: attachment side-effects ToolCalls 3 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history Criteria 4 0.63 0.63 0.00 1.00 No -
entity-analytics-v2: entity relationship history Groundedness 4 0.88 0.82 +0.05 0.81 No -
entity-analytics-v2: entity relationship history Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: entity relationship history ToolCalls 4 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: enumerate watchlists Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: enumerate watchlists Criteria 1 0.67 0.67 0.00 - n/a -
entity-analytics-v2: enumerate watchlists Groundedness 1 1.00 0.84 +0.16 - n/a -
entity-analytics-v2: enumerate watchlists Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: enumerate watchlists ToolCalls 1 0.00 0.00 0.00 - n/a -
entity-analytics-v2: get entity Attachments 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity Groundedness 6 0.62 0.50 +0.12 0.63 No -
entity-analytics-v2: get entity Sequence Accuracy 6 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: get entity ToolCalls 6 0.00 1.00 -1.00 1.00 No -
entity-analytics-v2: graph preview Attachments 3 0.33 1.00 -0.67 0.13 No -
entity-analytics-v2: graph preview Groundedness 3 0.88 0.94 -0.06 0.69 No -
entity-analytics-v2: graph preview Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: graph preview ToolCalls 3 0.00 0.67 -0.67 0.13 No -
entity-analytics-v2: manage watchlists (entity membership) Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) Criteria 4 0.25 0.38 -0.13 0.59 No -
entity-analytics-v2: manage watchlists (entity membership) Groundedness 4 0.46 0.00 +0.46 0.14 No -
entity-analytics-v2: manage watchlists (entity membership) Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (entity membership) ToolCalls 4 0.25 0.00 +0.25 0.33 No -
entity-analytics-v2: manage watchlists (lifecycle) Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) Criteria 4 0.33 0.75 -0.42 0.11 No -
entity-analytics-v2: manage watchlists (lifecycle) Groundedness 4 0.75 0.12 +0.63 0.05 No -
entity-analytics-v2: manage watchlists (lifecycle) Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: manage watchlists (lifecycle) ToolCalls 4 0.00 0.50 -0.50 0.14 No -
entity-analytics-v2: multi-skill routing Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing Criteria 4 0.50 0.63 -0.13 0.33 No -
entity-analytics-v2: multi-skill routing Groundedness 4 0.66 0.84 -0.18 0.55 No -
entity-analytics-v2: multi-skill routing Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: multi-skill routing ToolCalls 4 0.00 0.25 -0.25 0.33 No -
entity-analytics-v2: risk score history Attachments 3 0.00 0.67 -0.67 0.13 No -
entity-analytics-v2: risk score history Criteria 3 0.67 0.89 -0.22 0.13 No -
entity-analytics-v2: risk score history Groundedness 3 0.90 0.83 +0.07 0.67 No -
entity-analytics-v2: risk score history Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: risk score history ToolCalls 3 0.00 0.67 -0.67 0.13 No -
entity-analytics-v2: search entities Attachments 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: search entities Groundedness 9 0.95 0.90 +0.04 0.65 No -
entity-analytics-v2: search entities Sequence Accuracy 9 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: set_asset_criticality ToolCalls 3 0.00 0.33 -0.33 0.35 No -
entity-analytics-v2: UI navigation — asset criticality bulk Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Criteria 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk Groundedness 2 0.75 1.00 -0.25 0.39 No -
entity-analytics-v2: UI navigation — asset criticality bulk Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — asset criticality bulk ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — engine status Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status Criteria 1 0.00 0.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status Groundedness 1 0.50 0.00 +0.50 - n/a -
entity-analytics-v2: UI navigation — engine status Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — engine status ToolCalls 1 0.00 0.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity analytics settings Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings Criteria 2 0.13 0.13 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings Groundedness 2 0.25 0.85 -0.60 0.28 No -
entity-analytics-v2: UI navigation — entity analytics settings Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity analytics settings ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — entity resolution bulk Attachments 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Criteria 1 0.00 0.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Groundedness 1 1.00 0.00 +1.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
entity-analytics-v2: UI navigation — entity resolution bulk ToolCalls 1 0.00 0.00 0.00 - n/a -
entity-analytics-v2: UI navigation — risk engine Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — risk engine Criteria 2 0.25 0.25 0.00 1.00 No -
entity-analytics-v2: UI navigation — risk engine Groundedness 2 0.85 0.35 +0.50 0.18 No -
entity-analytics-v2: UI navigation — risk engine Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — risk engine ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Criteria 3 0.11 0.11 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Groundedness 3 0.67 0.84 -0.17 0.67 No -
entity-analytics-v2: UI navigation — watchlist edit flyout Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics-v2: UI navigation — watchlist edit flyout ToolCalls 3 0.00 0.00 0.00 1.00 No -
entity-analytics: dismiss_lead Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead Criteria 2 0.75 0.50 +0.25 0.39 No -
entity-analytics: dismiss_lead Groundedness 2 0.92 1.00 -0.08 0.39 No -
entity-analytics: dismiss_lead Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: dismiss_lead ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics: generate_leads Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads Criteria 2 0.75 0.50 +0.25 0.39 No -
entity-analytics: generate_leads Groundedness 2 0.85 1.00 -0.15 0.36 No -
entity-analytics: generate_leads Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: generate_leads ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics: leads Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Criteria 2 0.58 0.58 0.00 1.00 No -
entity-analytics: leads Groundedness 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: leads ToolCalls 2 0.00 0.00 0.00 1.00 No -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::detection-watch-rule-creation::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::detection-watch-rule-creation::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 23 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
detection-watch-rule-creation: golden MITRE Accuracy 15 0.51 0.61 -0.10 0.01 Yes Regression
No significant changes (22 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
detection-watch-rule-creation: canary Canary Tripped 3 0.00 0.00 0.00 1.00 No -
detection-watch-rule-creation: golden Field Coverage 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Gap Addressed 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Interval Format 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Lookback Gap 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Query Executability 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Query Syntax Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Risk Score Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Rule Type & Language 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Severity Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Tool Routing 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Field Coverage 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Gap Addressed 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Interval Format 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Lookback Gap 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases MITRE Accuracy 15 0.71 0.62 +0.09 0.11 No -
detection-watch-rule-creation: hard-cases Query Executability 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Query Syntax Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Risk Score Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Rule Type & Language 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Severity Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Tool Routing 15 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::detection-watch-rule-creation::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::detection-watch-rule-creation::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (23 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (23 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
detection-watch-rule-creation: canary Canary Tripped 3 0.00 0.00 0.00 1.00 No -
detection-watch-rule-creation: golden Field Coverage 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Gap Addressed 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Interval Format 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Lookback Gap 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden MITRE Accuracy 15 0.67 0.55 +0.12 0.10 No -
detection-watch-rule-creation: golden Query Executability 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Query Syntax Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Risk Score Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Rule Type & Language 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Severity Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Tool Routing 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Field Coverage 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Gap Addressed 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Interval Format 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Lookback Gap 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases MITRE Accuracy 15 0.69 0.70 -0.00 0.92 No -
detection-watch-rule-creation: hard-cases Query Executability 15 0.93 0.93 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Query Syntax Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Risk Score Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Rule Type & Language 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Severity Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Tool Routing 15 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::detection-watch-rule-creation::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::detection-watch-rule-creation::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (23 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (23 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
detection-watch-rule-creation: canary Canary Tripped 3 0.33 0.00 +0.33 0.35 No -
detection-watch-rule-creation: golden Field Coverage 15 0.99 1.00 -0.01 0.32 No -
detection-watch-rule-creation: golden Gap Addressed 15 0.93 0.93 0.00 1.00 No -
detection-watch-rule-creation: golden Interval Format 15 0.93 0.93 0.00 1.00 No -
detection-watch-rule-creation: golden Lookback Gap 15 0.93 0.93 0.00 1.00 No -
detection-watch-rule-creation: golden MITRE Accuracy 15 0.64 0.66 -0.02 0.80 No -
detection-watch-rule-creation: golden Query Executability 15 0.93 0.93 0.00 1.00 No -
detection-watch-rule-creation: golden Query Syntax Validity 15 0.93 0.93 0.00 1.00 No -
detection-watch-rule-creation: golden Risk Score Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Rule Type & Language 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Severity Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Tool Routing 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Field Coverage 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Gap Addressed 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Interval Format 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Lookback Gap 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases MITRE Accuracy 15 0.70 0.71 -0.01 0.67 No -
detection-watch-rule-creation: hard-cases Query Executability 15 1.00 0.93 +0.07 0.32 No -
detection-watch-rule-creation: hard-cases Query Syntax Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Risk Score Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Rule Type & Language 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Severity Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Tool Routing 15 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::detection-watch-rule-creation::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::detection-watch-rule-creation::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 23 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
detection-watch-rule-creation: golden MITRE Accuracy 15 0.76 0.62 +0.13 0.01 Yes Improvement
No significant changes (22 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
detection-watch-rule-creation: canary Canary Tripped 3 0.00 0.00 0.00 1.00 No -
detection-watch-rule-creation: golden Field Coverage 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Gap Addressed 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Interval Format 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Lookback Gap 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Query Executability 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Query Syntax Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Risk Score Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Rule Type & Language 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Severity Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Tool Routing 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Field Coverage 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Gap Addressed 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Interval Format 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Lookback Gap 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases MITRE Accuracy 15 0.69 0.70 -0.00 0.93 No -
detection-watch-rule-creation: hard-cases Query Executability 15 1.00 0.87 +0.13 0.15 No -
detection-watch-rule-creation: hard-cases Query Syntax Validity 15 1.00 0.93 +0.07 0.32 No -
detection-watch-rule-creation: hard-cases Risk Score Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Rule Type & Language 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Severity Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Tool Routing 15 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::detection-watch-rule-creation::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::detection-watch-rule-creation::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (23 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (23 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
detection-watch-rule-creation: canary Canary Tripped 3 0.00 0.00 0.00 1.00 No -
detection-watch-rule-creation: golden Field Coverage 15 0.99 1.00 -0.01 0.32 No -
detection-watch-rule-creation: golden Gap Addressed 15 0.93 1.00 -0.07 0.32 No -
detection-watch-rule-creation: golden Interval Format 15 0.93 1.00 -0.07 0.32 No -
detection-watch-rule-creation: golden Lookback Gap 15 0.93 1.00 -0.07 0.32 No -
detection-watch-rule-creation: golden MITRE Accuracy 15 0.61 0.64 -0.03 0.65 No -
detection-watch-rule-creation: golden Query Executability 15 0.93 1.00 -0.07 0.32 No -
detection-watch-rule-creation: golden Query Syntax Validity 15 0.93 1.00 -0.07 0.32 No -
detection-watch-rule-creation: golden Risk Score Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Rule Type & Language 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Severity Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Tool Routing 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Field Coverage 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Gap Addressed 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Interval Format 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Lookback Gap 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases MITRE Accuracy 15 0.67 0.63 +0.04 0.36 No -
detection-watch-rule-creation: hard-cases Query Executability 15 0.93 1.00 -0.07 0.32 No -
detection-watch-rule-creation: hard-cases Query Syntax Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Risk Score Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Rule Type & Language 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Severity Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Tool Routing 15 1.00 1.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::detection-watch-rule-creation::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::detection-watch-rule-creation::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (23 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (23 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
detection-watch-rule-creation: canary Canary Tripped 3 0.00 0.00 0.00 1.00 No -
detection-watch-rule-creation: golden Field Coverage 15 1.00 0.99 +0.01 0.32 No -
detection-watch-rule-creation: golden Gap Addressed 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Interval Format 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Lookback Gap 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden MITRE Accuracy 15 0.66 0.71 -0.05 0.47 No -
detection-watch-rule-creation: golden Query Executability 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Query Syntax Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Risk Score Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Rule Type & Language 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Severity Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: golden Tool Routing 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Field Coverage 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Gap Addressed 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Interval Format 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Lookback Gap 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases MITRE Accuracy 15 0.60 0.63 -0.03 0.67 No -
detection-watch-rule-creation: hard-cases Query Executability 15 1.00 0.93 +0.07 0.32 No -
detection-watch-rule-creation: hard-cases Query Syntax Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Risk Score Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Rule Type & Language 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Severity Validity 15 1.00 1.00 0.00 1.00 No -
detection-watch-rule-creation: hard-cases Tool Routing 15 1.00 1.00 0.00 1.00 No -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::entity-analytics::anthropic-claude-4.6-opus | Baseline (main): bk-019ec894-2ea6-46b6-8f45-581d84147ea7::anthropic-claude-4.6-opus
Baseline: commit f3d7c43, 89 days ago

Warning: Baseline is 89 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
2 significant difference(s) detected out of 30 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics: asset criticality Groundedness 2 0.99 0.69 +0.30 0.04 Yes Improvement
entity-analytics: boundary cases Groundedness 23 0.62 0.79 -0.17 0.03 Yes Regression
No significant changes (28 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics: anomalous behavior without data Attachments 12 1.00 1.00 0.00 1.00 No -
entity-analytics: anomalous behavior without data Criteria 12 0.86 0.75 +0.11 0.32 No -
entity-analytics: anomalous behavior without data Groundedness 12 0.90 0.83 +0.07 0.07 No -
entity-analytics: anomalous behavior without data Sequence Accuracy 12 1.00 1.00 0.00 1.00 No -
entity-analytics: anomalous behavior without data ToolCalls 12 1.00 0.75 +0.25 0.07 No -
entity-analytics: asset criticality Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: asset criticality Criteria 2 1.00 0.75 +0.25 0.39 No -
entity-analytics: asset criticality Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: asset criticality ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics: boundary cases Attachments 23 1.00 1.00 0.00 1.00 No -
entity-analytics: boundary cases Criteria 23 0.62 0.68 -0.06 0.08 No -
entity-analytics: boundary cases Sequence Accuracy 23 1.00 1.00 0.00 1.00 No -
entity-analytics: boundary cases ToolCalls 23 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing Criteria 4 0.63 0.75 -0.13 0.33 No -
entity-analytics: multi-skill routing Groundedness 4 0.91 0.88 +0.03 0.25 No -
entity-analytics: multi-skill routing Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing ToolCalls 4 0.00 0.00 0.00 1.00 No -
entity-analytics: partial feasibility Attachments 9 1.00 1.00 0.00 1.00 No -
entity-analytics: partial feasibility Criteria 9 1.00 1.00 0.00 1.00 No -
entity-analytics: partial feasibility Groundedness 9 0.81 0.89 -0.08 0.50 No -
entity-analytics: partial feasibility Sequence Accuracy 9 1.00 1.00 0.00 1.00 No -
entity-analytics: partial feasibility ToolCalls 9 0.56 0.56 0.00 1.00 No -
entity-analytics: risk score without data Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics: risk score without data Criteria 3 0.78 0.78 0.00 1.00 No -
entity-analytics: risk score without data Groundedness 3 0.95 0.70 +0.26 0.15 No -
entity-analytics: risk score without data Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics: risk score without data ToolCalls 3 0.00 0.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::entity-analytics::anthropic-claude-4.6-sonnet | Baseline (main): bk-019ec894-2ea6-46b6-8f45-581d84147ea7::anthropic-claude-4.6-sonnet
Baseline: commit f3d7c43, 89 days ago

Warning: Baseline is 89 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
2 significant difference(s) detected out of 30 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics: boundary cases Criteria 23 0.67 0.56 +0.11 0.02 Yes Improvement
entity-analytics: boundary cases Groundedness 23 0.52 0.77 -0.25 0.01 Yes Regression
No significant changes (28 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics: anomalous behavior without data Attachments 12 1.00 1.00 0.00 1.00 No -
entity-analytics: anomalous behavior without data Criteria 12 0.90 0.72 +0.18 0.06 No -
entity-analytics: anomalous behavior without data Groundedness 12 0.84 0.75 +0.10 0.21 No -
entity-analytics: anomalous behavior without data Sequence Accuracy 12 1.00 1.00 0.00 1.00 No -
entity-analytics: anomalous behavior without data ToolCalls 12 0.92 0.75 +0.17 0.15 No -
entity-analytics: asset criticality Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: asset criticality Criteria 2 1.00 0.75 +0.25 0.39 No -
entity-analytics: asset criticality Groundedness 2 0.98 0.43 +0.55 0.33 No -
entity-analytics: asset criticality Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: asset criticality ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics: boundary cases Attachments 23 1.00 1.00 0.00 1.00 No -
entity-analytics: boundary cases Sequence Accuracy 23 1.00 1.00 0.00 1.00 No -
entity-analytics: boundary cases ToolCalls 23 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing Criteria 4 0.88 0.63 +0.25 0.14 No -
entity-analytics: multi-skill routing Groundedness 4 0.92 0.80 +0.12 0.12 No -
entity-analytics: multi-skill routing Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing ToolCalls 4 0.00 0.00 0.00 1.00 No -
entity-analytics: partial feasibility Attachments 9 1.00 1.00 0.00 1.00 No -
entity-analytics: partial feasibility Criteria 9 0.94 1.00 -0.06 0.32 No -
entity-analytics: partial feasibility Groundedness 9 0.76 0.82 -0.06 0.62 No -
entity-analytics: partial feasibility Sequence Accuracy 9 1.00 1.00 0.00 1.00 No -
entity-analytics: partial feasibility ToolCalls 9 0.56 0.56 0.00 1.00 No -
entity-analytics: risk score without data Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics: risk score without data Criteria 3 0.50 0.78 -0.28 0.15 No -
entity-analytics: risk score without data Groundedness 3 0.95 0.95 -0.00 0.94 No -
entity-analytics: risk score without data Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics: risk score without data ToolCalls 3 0.00 0.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::entity-analytics::google-gemini-3.0-flash | Baseline (main): bk-019ec894-2ea6-46b6-8f45-581d84147ea7::google-gemini-3.0-flash
Baseline: commit f3d7c43, 89 days ago

Warning: Baseline is 89 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (30 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (30 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics: anomalous behavior without data Attachments 12 1.00 1.00 0.00 1.00 No -
entity-analytics: anomalous behavior without data Criteria 12 0.63 0.78 -0.15 0.17 No -
entity-analytics: anomalous behavior without data Groundedness 12 0.98 0.95 +0.03 0.39 No -
entity-analytics: anomalous behavior without data Sequence Accuracy 12 1.00 1.00 0.00 1.00 No -
entity-analytics: anomalous behavior without data ToolCalls 12 0.83 0.83 0.00 1.00 No -
entity-analytics: asset criticality Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: asset criticality Criteria 2 1.00 1.00 0.00 1.00 No -
entity-analytics: asset criticality Groundedness 2 0.69 0.97 -0.28 0.43 No -
entity-analytics: asset criticality Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: asset criticality ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics: boundary cases Attachments 23 1.00 1.00 0.00 1.00 No -
entity-analytics: boundary cases Criteria 23 0.56 0.62 -0.07 0.18 No -
entity-analytics: boundary cases Groundedness 23 0.94 0.92 +0.02 0.57 No -
entity-analytics: boundary cases Sequence Accuracy 23 1.00 1.00 0.00 1.00 No -
entity-analytics: boundary cases ToolCalls 23 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing Criteria 4 0.75 0.63 +0.13 0.33 No -
entity-analytics: multi-skill routing Groundedness 4 0.90 0.98 -0.09 0.32 No -
entity-analytics: multi-skill routing Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing ToolCalls 4 0.00 0.00 0.00 1.00 No -
entity-analytics: partial feasibility Attachments 9 1.00 1.00 0.00 1.00 No -
entity-analytics: partial feasibility Criteria 9 0.94 0.94 0.00 1.00 No -
entity-analytics: partial feasibility Groundedness 9 0.94 0.95 -0.01 0.82 No -
entity-analytics: partial feasibility Sequence Accuracy 9 1.00 1.00 0.00 1.00 No -
entity-analytics: partial feasibility ToolCalls 9 0.56 0.56 0.00 1.00 No -
entity-analytics: risk score without data Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics: risk score without data Criteria 3 0.11 0.00 +0.11 0.35 No -
entity-analytics: risk score without data Groundedness 3 0.93 0.94 -0.01 0.68 No -
entity-analytics: risk score without data Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics: risk score without data ToolCalls 3 0.00 0.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::entity-analytics::google-gemini-3.1-pro | Baseline (main): bk-019ec894-2ea6-46b6-8f45-581d84147ea7::google-gemini-3.1-pro
Baseline: commit f3d7c43, 89 days ago

Warning: Baseline is 89 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 30 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics: multi-skill routing Criteria 4 0.88 0.50 +0.38 0.04 Yes Improvement
No significant changes (29 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics: anomalous behavior without data Attachments 12 1.00 1.00 0.00 1.00 No -
entity-analytics: anomalous behavior without data Criteria 12 0.78 0.97 -0.19 0.10 No -
entity-analytics: anomalous behavior without data Groundedness 12 0.98 0.99 -0.01 0.33 No -
entity-analytics: anomalous behavior without data Sequence Accuracy 12 1.00 1.00 0.00 1.00 No -
entity-analytics: anomalous behavior without data ToolCalls 12 0.83 1.00 -0.17 0.15 No -
entity-analytics: asset criticality Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: asset criticality Criteria 2 0.75 0.50 +0.25 0.39 No -
entity-analytics: asset criticality Groundedness 2 0.99 1.00 -0.01 0.39 No -
entity-analytics: asset criticality Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: asset criticality ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics: boundary cases Attachments 23 1.00 1.00 0.00 1.00 No -
entity-analytics: boundary cases Criteria 23 0.51 0.56 -0.04 0.15 No -
entity-analytics: boundary cases Groundedness 23 0.65 0.82 -0.18 0.10 No -
entity-analytics: boundary cases Sequence Accuracy 23 1.00 1.00 0.00 1.00 No -
entity-analytics: boundary cases ToolCalls 23 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing Groundedness 4 0.99 0.99 -0.00 0.98 No -
entity-analytics: multi-skill routing Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing ToolCalls 4 0.00 0.00 0.00 1.00 No -
entity-analytics: partial feasibility Attachments 9 1.00 1.00 0.00 1.00 No -
entity-analytics: partial feasibility Criteria 9 0.94 0.78 +0.17 0.17 No -
entity-analytics: partial feasibility Groundedness 9 0.88 0.94 -0.06 0.62 No -
entity-analytics: partial feasibility Sequence Accuracy 9 1.00 1.00 0.00 1.00 No -
entity-analytics: partial feasibility ToolCalls 9 0.56 0.56 0.00 1.00 No -
entity-analytics: risk score without data Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics: risk score without data Criteria 3 0.39 0.00 +0.39 0.14 No -
entity-analytics: risk score without data Groundedness 3 0.99 0.85 +0.14 0.10 No -
entity-analytics: risk score without data Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics: risk score without data ToolCalls 3 0.00 0.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::entity-analytics::openai-gpt-5.4 | Baseline (main): bk-019ec894-2ea6-46b6-8f45-581d84147ea7::openai-gpt-5.4
Baseline: commit f3d7c43, 89 days ago

Warning: Baseline is 89 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
2 significant difference(s) detected out of 30 comparisons.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics: boundary cases Groundedness 23 0.79 0.98 -0.19 0.03 Yes Regression
entity-analytics: multi-skill routing Groundedness 4 0.99 0.98 +0.02 0.01 Yes Improvement
No significant changes (28 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics: anomalous behavior without data Attachments 12 1.00 1.00 0.00 1.00 No -
entity-analytics: anomalous behavior without data Criteria 12 0.69 0.67 +0.03 0.81 No -
entity-analytics: anomalous behavior without data Groundedness 12 0.98 0.95 +0.03 0.31 No -
entity-analytics: anomalous behavior without data Sequence Accuracy 12 1.00 1.00 0.00 1.00 No -
entity-analytics: anomalous behavior without data ToolCalls 12 0.83 0.83 0.00 1.00 No -
entity-analytics: asset criticality Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: asset criticality Criteria 2 0.50 0.50 0.00 1.00 No -
entity-analytics: asset criticality Groundedness 2 0.48 0.99 -0.51 0.38 No -
entity-analytics: asset criticality Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: asset criticality ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics: boundary cases Attachments 23 1.00 1.00 0.00 1.00 No -
entity-analytics: boundary cases Criteria 23 0.62 0.67 -0.04 0.48 No -
entity-analytics: boundary cases Sequence Accuracy 23 1.00 1.00 0.00 1.00 No -
entity-analytics: boundary cases ToolCalls 23 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing Criteria 4 0.50 0.75 -0.25 0.14 No -
entity-analytics: multi-skill routing Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing ToolCalls 4 0.00 0.00 0.00 1.00 No -
entity-analytics: partial feasibility Attachments 9 1.00 1.00 0.00 1.00 No -
entity-analytics: partial feasibility Criteria 9 0.72 0.94 -0.22 0.15 No -
entity-analytics: partial feasibility Groundedness 9 0.77 0.94 -0.17 0.27 No -
entity-analytics: partial feasibility Sequence Accuracy 9 1.00 1.00 0.00 1.00 No -
entity-analytics: partial feasibility ToolCalls 9 0.56 0.56 0.00 1.00 No -
entity-analytics: risk score without data Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics: risk score without data Criteria 3 0.00 0.00 0.00 1.00 No -
entity-analytics: risk score without data Groundedness 3 0.96 0.99 -0.03 0.09 No -
entity-analytics: risk score without data Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics: risk score without data ToolCalls 3 0.00 0.00 0.00 1.00 No -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::entity-analytics::openai-gpt-oss-120b | Baseline (main): bk-019ec894-2ea6-46b6-8f45-581d84147ea7::openai-gpt-oss-120b
Baseline: commit f3d7c43, 89 days ago

Warning: Baseline is 89 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 30 comparisons.

Note: 35 unpaired (ran in only one experiment) skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics: boundary cases Criteria 23 0.49 0.58 -0.09 0.04 Yes Regression
No significant changes (29 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
entity-analytics: anomalous behavior without data Attachments 12 1.00 1.00 0.00 1.00 No -
entity-analytics: anomalous behavior without data Criteria 12 0.00 0.11 -0.11 0.15 No -
entity-analytics: anomalous behavior without data Groundedness 12 0.82 0.60 +0.22 0.19 No -
entity-analytics: anomalous behavior without data Sequence Accuracy 12 1.00 1.00 0.00 1.00 No -
entity-analytics: anomalous behavior without data ToolCalls 12 0.17 0.42 -0.25 0.07 No -
entity-analytics: asset criticality Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: asset criticality Criteria 2 0.25 0.25 0.00 1.00 No -
entity-analytics: asset criticality Groundedness 2 0.82 0.48 +0.33 0.49 No -
entity-analytics: asset criticality Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: asset criticality ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics: boundary cases Attachments 23 1.00 1.00 0.00 1.00 No -
entity-analytics: boundary cases Groundedness 23 0.73 0.81 -0.08 0.42 No -
entity-analytics: boundary cases Sequence Accuracy 23 1.00 1.00 0.00 1.00 No -
entity-analytics: boundary cases ToolCalls 23 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing Attachments 4 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing Criteria 4 0.63 0.50 +0.13 0.33 No -
entity-analytics: multi-skill routing Groundedness 4 1.00 0.99 +0.01 0.19 No -
entity-analytics: multi-skill routing Sequence Accuracy 4 1.00 1.00 0.00 1.00 No -
entity-analytics: multi-skill routing ToolCalls 4 0.00 0.00 0.00 1.00 No -
entity-analytics: partial feasibility Attachments 2 1.00 1.00 0.00 1.00 No -
entity-analytics: partial feasibility Criteria 2 0.00 1.00 -1.00 1.00 No -
entity-analytics: partial feasibility Groundedness 2 0.85 0.98 -0.13 0.48 No -
entity-analytics: partial feasibility Sequence Accuracy 2 1.00 1.00 0.00 1.00 No -
entity-analytics: partial feasibility ToolCalls 2 0.00 0.00 0.00 1.00 No -
entity-analytics: risk score without data Attachments 3 1.00 1.00 0.00 1.00 No -
entity-analytics: risk score without data Criteria 3 0.00 0.00 0.00 1.00 No -
entity-analytics: risk score without data Groundedness 3 0.75 0.32 +0.43 0.38 No -
entity-analytics: risk score without data Sequence Accuracy 3 1.00 1.00 0.00 1.00 No -
entity-analytics: risk score without data ToolCalls 3 0.00 0.00 0.00 1.00 No -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

🧪 LLM eval (PR) — Security Entity Analytics (entity-analytics) failed.

Failing models:

  • eis-anthropic-claude-4-6-opus
  • eis-anthropic-claude-4-6-sonnet
  • eis-google-gemini-3-0-flash
  • eis-google-gemini-3-1-pro
  • eis-openai-gpt-5-4
  • eis-openai-gpt-oss-120b

View build

Triage summary:

Triage summary could not be generated: Triage model did not call the report_triage tool. See the suite owner notify Buildkite step for details.

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::observability-ai::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::observability-ai::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (140 evaluator comparisons).

Note: 5 unpaired (ran in only one experiment), 114 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (140 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
ad-failure investigation Cached Tokens 1 14896.00 14897.00 -1.00 - n/a -
ad-failure investigation Criteria 1 1.00 0.50 +0.50 - n/a -
ad-failure investigation Input Tokens 1 158758.00 158941.00 -183.00 - n/a -
ad-failure investigation Output Tokens 1 1274.00 1425.00 -151.00 - n/a -
ad-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
ad-failure investigation Tool Calls 1 10.00 12.00 -2.00 - n/a -
ad-failure investigation trajectory 1 1.00 1.00 0.00 - n/a -
ad-high-cpu investigation Criteria 1 1.00 1.00 0.00 - n/a -
ad-high-cpu investigation Input Tokens 1 187266.00 141500.00 +45766.00 - n/a -
ad-high-cpu investigation Output Tokens 1 1901.00 2111.00 -210.00 - n/a -
ad-high-cpu investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
ad-high-cpu investigation Tool Calls 1 15.00 16.00 -1.00 - n/a -
ad-high-cpu investigation trajectory 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Factuality 1 0.40 0.27 +0.13 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Input Tokens 1 3662.00 3657.00 +5.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Output Tokens 1 383.00 399.00 -16.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Relevance 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Input Tokens 1 3181.00 3184.00 -3.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Output Tokens 1 400.00 402.00 -2.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Relevance 1 0.83 0.91 -0.08 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Input Tokens 1 3635.00 3639.00 -4.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Output Tokens 1 365.00 376.00 -11.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Relevance 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Factuality 1 0.60 0.38 +0.23 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Input Tokens 1 14864.00 14864.00 0.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Output Tokens 1 1327.00 1120.00 +207.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Relevance 1 0.73 0.83 -0.10 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Factuality 1 0.64 0.73 -0.09 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Input Tokens 1 16433.00 16434.00 -1.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Output Tokens 1 1101.00 1136.00 -35.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Relevance 1 0.85 0.77 +0.08 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Input Tokens 1 12215.00 12214.00 +1.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Output Tokens 1 1216.00 1305.00 -89.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Relevance 1 0.75 0.71 +0.04 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Factuality 1 0.73 0.00 +0.73 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Input Tokens 1 9629.00 9630.00 -1.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Output Tokens 1 641.00 626.00 +15.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Relevance 1 0.57 0.75 -0.18 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Factuality 1 0.00 0.84 -0.84 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Input Tokens 1 12659.00 12662.00 -3.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Output Tokens 1 1069.00 1190.00 -121.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Relevance 1 0.77 0.75 +0.02 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Input Tokens 1 7946.00 7950.00 -4.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Output Tokens 1 1251.00 1111.00 +140.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Relevance 1 0.86 0.78 +0.08 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
alert investigation — narrow screen context vs wide alert window Cached Tokens 1 15050.00 15052.00 -2.00 - n/a -
alert investigation — narrow screen context vs wide alert window Criteria 1 1.00 1.00 0.00 - n/a -
alert investigation — narrow screen context vs wide alert window Input Tokens 1 92347.00 93322.00 -975.00 - n/a -
alert investigation — narrow screen context vs wide alert window Output Tokens 1 1233.00 1079.00 +154.00 - n/a -
alert investigation — narrow screen context vs wide alert window Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
alert investigation — narrow screen context vs wide alert window Tool Calls 1 11.00 9.00 +2.00 - n/a -
alert investigation — narrow screen context vs wide alert window trajectory 1 1.00 1.00 0.00 - n/a -
alert time window — discovered via get_alerts Cached Tokens 1 85815.00 36532.00 +49283.00 - n/a -
alert time window — discovered via get_alerts Criteria 1 0.80 0.60 +0.20 - n/a -
alert time window — discovered via get_alerts Input Tokens 1 472096.00 321404.00 +150692.00 - n/a -
alert time window — discovered via get_alerts Output Tokens 1 6104.00 2937.00 +3167.00 - n/a -
alert time window — discovered via get_alerts Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
alert time window — discovered via get_alerts Tool Calls 1 16.00 13.00 +3.00 - n/a -
alert time window — discovered via get_alerts trajectory 1 1.00 1.00 0.00 - n/a -
cart-failure investigation Criteria 1 1.00 0.83 +0.17 - n/a -
cart-failure investigation Input Tokens 1 105382.00 105432.00 -50.00 - n/a -
cart-failure investigation Output Tokens 1 966.00 1009.00 -43.00 - n/a -
cart-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
cart-failure investigation Tool Calls 1 8.00 8.00 0.00 - n/a -
cart-failure investigation trajectory 1 1.00 1.00 0.00 - n/a -
email-memory-leak investigation Cached Tokens 1 299876.00 369507.00 -69631.00 - n/a -
email-memory-leak investigation Criteria 1 0.83 1.00 -0.17 - n/a -
email-memory-leak investigation Input Tokens 1 818856.00 970345.00 -151489.00 - n/a -
email-memory-leak investigation Output Tokens 1 6636.00 6714.00 -78.00 - n/a -
email-memory-leak investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
email-memory-leak investigation Tool Calls 1 37.00 38.00 -1.00 - n/a -
email-memory-leak investigation trajectory 1 1.00 1.00 0.00 - n/a -
kafka-queue-problems investigation Criteria 1 0.20 0.20 0.00 - n/a -
kafka-queue-problems investigation Input Tokens 1 110462.00 149221.00 -38759.00 - n/a -
kafka-queue-problems investigation Output Tokens 1 1169.00 1495.00 -326.00 - n/a -
kafka-queue-problems investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
kafka-queue-problems investigation Tool Calls 1 10.00 12.00 -2.00 - n/a -
kafka-queue-problems investigation trajectory 1 1.00 1.00 0.00 - n/a -
load-generator-flood-homepage investigation Criteria 1 1.00 1.00 0.00 - n/a -
load-generator-flood-homepage investigation Input Tokens 1 228530.00 196389.00 +32141.00 - n/a -
load-generator-flood-homepage investigation Output Tokens 1 2763.00 2165.00 +598.00 - n/a -
load-generator-flood-homepage investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
load-generator-flood-homepage investigation Tool Calls 1 18.00 16.00 +2.00 - n/a -
load-generator-flood-homepage investigation trajectory 1 1.00 1.00 0.00 - n/a -
obs alerts routing - no alerts present Criteria 3 0.33 0.00 +0.33 0.35 No -
obs alerts routing - no alerts present Input Tokens 3 94800.33 34080.67 +60719.67 0.10 No -
obs alerts routing - no alerts present Output Tokens 3 1891.33 911.00 +980.33 0.15 No -
obs alerts routing - no alerts present Skill Invoked (investigation) 3 0.00 0.00 0.00 1.00 No -
obs alerts routing - no alerts present Tool Calls 3 2.67 2.00 +0.67 0.35 No -
obs alerts routing - no alerts present trajectory 3 0.00 0.00 0.00 1.00 No -
obs alerts routing across screen contexts Criteria 5 0.67 0.67 0.00 1.00 No -
obs alerts routing across screen contexts Input Tokens 5 69292.40 77778.20 -8485.80 0.62 No -
obs alerts routing across screen contexts Output Tokens 5 1251.40 1807.60 -556.20 0.06 No -
obs alerts routing across screen contexts Skill Invoked (investigation) 5 0.00 0.00 0.00 1.00 No -
obs alerts routing across screen contexts Tool Calls 5 2.00 3.20 -1.20 0.05 No -
obs alerts routing across screen contexts trajectory 5 0.00 0.00 0.00 1.00 No -
payment-failure investigation Criteria 1 1.00 1.00 0.00 - n/a -
payment-failure investigation Input Tokens 1 104157.00 147988.00 -43831.00 - n/a -
payment-failure investigation Output Tokens 1 1256.00 1629.00 -373.00 - n/a -
payment-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
payment-failure investigation Tool Calls 1 10.00 12.00 -2.00 - n/a -
payment-failure investigation trajectory 1 1.00 1.00 0.00 - n/a -
payment-unreachable investigation Criteria 1 0.92 1.00 -0.08 - n/a -
payment-unreachable investigation Input Tokens 1 72718.00 72736.00 -18.00 - n/a -
payment-unreachable investigation Output Tokens 1 888.00 851.00 +37.00 - n/a -
payment-unreachable investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
payment-unreachable investigation Tool Calls 1 8.00 8.00 0.00 - n/a -
payment-unreachable investigation trajectory 1 1.00 1.00 0.00 - n/a -
service-map skill activation Cached Tokens 3 22684.00 22684.33 -0.33 0.86 No -
service-map skill activation Criteria 4 1.00 1.00 0.00 1.00 No -
service-map skill activation Input Tokens 4 58637.75 54164.25 +4473.50 0.33 No -
service-map skill activation Output Tokens 4 1068.25 875.50 +192.75 0.33 No -
service-map skill activation Skill Invoked (service-map) 4 0.00 0.00 0.00 1.00 No -
service-map skill activation Tool Calls 4 3.50 3.25 +0.25 0.33 No -
service-map skill activation trajectory 4 0.88 0.88 0.00 1.00 No -
tool-arguments: default time fallback Criteria 1 1.00 1.00 0.00 - n/a -
tool-arguments: default time fallback Input Tokens 1 70578.00 42686.00 +27892.00 - n/a -
tool-arguments: default time fallback Output Tokens 1 296.00 183.00 +113.00 - n/a -
tool-arguments: default time fallback Tool Calls 1 3.00 2.00 +1.00 - n/a -
tool-arguments: user time override Criteria 1 1.00 1.00 0.00 - n/a -
tool-arguments: user time override Input Tokens 1 43217.00 43213.00 +4.00 - n/a -
tool-arguments: user time override Output Tokens 1 227.00 232.00 -5.00 - n/a -
tool-arguments: user time override Tool Calls 1 3.00 3.00 0.00 - n/a -
tool-arguments: uses screen context time range Criteria 1 1.00 1.00 0.00 - n/a -
tool-arguments: uses screen context time range Input Tokens 1 30203.00 43196.00 -12993.00 - n/a -
tool-arguments: uses screen context time range Output Tokens 1 169.00 234.00 -65.00 - n/a -
tool-arguments: uses screen context time range Tool Calls 1 2.00 3.00 -1.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::observability-ai::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::observability-ai::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (137 evaluator comparisons).

Note: 6 unpaired (ran in only one experiment), 125 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (137 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
ad-failure investigation Criteria 1 1.00 1.00 0.00 - n/a -
ad-failure investigation Input Tokens 1 180627.00 142959.00 +37668.00 - n/a -
ad-failure investigation Output Tokens 1 1274.00 1244.00 +30.00 - n/a -
ad-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
ad-failure investigation Tool Calls 1 10.00 11.00 -1.00 - n/a -
ad-failure investigation trajectory 1 1.00 1.00 0.00 - n/a -
ad-high-cpu investigation Criteria 1 1.00 1.00 0.00 - n/a -
ad-high-cpu investigation Input Tokens 1 194805.00 146557.00 +48248.00 - n/a -
ad-high-cpu investigation Output Tokens 1 1960.00 1744.00 +216.00 - n/a -
ad-high-cpu investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
ad-high-cpu investigation Tool Calls 1 14.00 13.00 +1.00 - n/a -
ad-high-cpu investigation trajectory 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Factuality 1 0.56 0.30 +0.26 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Input Tokens 1 3657.00 3667.00 -10.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Output Tokens 1 386.00 432.00 -46.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Relevance 1 0.63 1.00 -0.38 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Input Tokens 1 3188.00 3186.00 +2.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Output Tokens 1 388.00 393.00 -5.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Relevance 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Input Tokens 1 3641.00 3639.00 +2.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Output Tokens 1 412.00 404.00 +8.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Relevance 1 0.78 1.00 -0.22 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Factuality 1 0.65 0.61 +0.03 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Input Tokens 1 14865.00 14866.00 -1.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Output Tokens 1 1191.00 1161.00 +30.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Relevance 1 0.67 0.75 -0.08 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Factuality 1 0.96 0.53 +0.43 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Input Tokens 1 16436.00 16436.00 0.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Output Tokens 1 1109.00 1081.00 +28.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Relevance 1 0.90 0.73 +0.17 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Input Tokens 1 12216.00 12217.00 -1.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Output Tokens 1 1307.00 1212.00 +95.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Relevance 1 0.70 0.54 +0.16 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Factuality 1 0.64 0.70 -0.06 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Input Tokens 1 9631.00 9631.00 0.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Output Tokens 1 556.00 632.00 -76.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Relevance 1 0.36 0.50 -0.14 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Input Tokens 1 12660.00 12664.00 -4.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Output Tokens 1 973.00 998.00 -25.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Relevance 1 0.75 0.73 +0.02 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Input Tokens 1 7947.00 7947.00 0.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Output Tokens 1 1198.00 1178.00 +20.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Relevance 1 0.50 0.83 -0.33 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
alert investigation — narrow screen context vs wide alert window Criteria 1 0.60 0.80 -0.20 - n/a -
alert investigation — narrow screen context vs wide alert window Input Tokens 1 136563.00 103461.00 +33102.00 - n/a -
alert investigation — narrow screen context vs wide alert window Output Tokens 1 1302.00 1056.00 +246.00 - n/a -
alert investigation — narrow screen context vs wide alert window Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
alert investigation — narrow screen context vs wide alert window Tool Calls 1 11.00 10.00 +1.00 - n/a -
alert investigation — narrow screen context vs wide alert window trajectory 1 1.00 1.00 0.00 - n/a -
alert time window — discovered via get_alerts Criteria 1 1.00 0.80 +0.20 - n/a -
alert time window — discovered via get_alerts Input Tokens 1 106335.00 133473.00 -27138.00 - n/a -
alert time window — discovered via get_alerts Output Tokens 1 878.00 1042.00 -164.00 - n/a -
alert time window — discovered via get_alerts Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
alert time window — discovered via get_alerts Tool Calls 1 8.00 10.00 -2.00 - n/a -
alert time window — discovered via get_alerts trajectory 1 1.00 1.00 0.00 - n/a -
cart-failure investigation Criteria 1 0.83 0.83 0.00 - n/a -
cart-failure investigation Input Tokens 1 103534.00 103583.00 -49.00 - n/a -
cart-failure investigation Output Tokens 1 770.00 820.00 -50.00 - n/a -
cart-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
cart-failure investigation Tool Calls 1 7.00 7.00 0.00 - n/a -
cart-failure investigation trajectory 1 1.00 1.00 0.00 - n/a -
email-memory-leak investigation Criteria 1 1.00 1.00 0.00 - n/a -
email-memory-leak investigation Input Tokens 1 762088.00 735261.00 +26827.00 - n/a -
email-memory-leak investigation Output Tokens 1 6783.00 5467.00 +1316.00 - n/a -
email-memory-leak investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
email-memory-leak investigation Tool Calls 1 40.00 31.00 +9.00 - n/a -
email-memory-leak investigation trajectory 1 1.00 1.00 0.00 - n/a -
kafka-queue-problems investigation Criteria 1 0.20 0.20 0.00 - n/a -
kafka-queue-problems investigation Input Tokens 1 131679.00 138419.00 -6740.00 - n/a -
kafka-queue-problems investigation Output Tokens 1 1460.00 1093.00 +367.00 - n/a -
kafka-queue-problems investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
kafka-queue-problems investigation Tool Calls 1 11.00 9.00 +2.00 - n/a -
kafka-queue-problems investigation trajectory 1 1.00 0.67 +0.33 - n/a -
load-generator-flood-homepage investigation Criteria 1 1.00 1.00 0.00 - n/a -
load-generator-flood-homepage investigation Input Tokens 1 208709.00 175225.00 +33484.00 - n/a -
load-generator-flood-homepage investigation Output Tokens 1 1923.00 1724.00 +199.00 - n/a -
load-generator-flood-homepage investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
load-generator-flood-homepage investigation Tool Calls 1 14.00 13.00 +1.00 - n/a -
load-generator-flood-homepage investigation trajectory 1 1.00 1.00 0.00 - n/a -
obs alerts routing - no alerts present Cached Tokens 1 32690.00 67430.00 -34740.00 - n/a -
obs alerts routing - no alerts present Criteria 3 0.00 0.00 0.00 1.00 No -
obs alerts routing - no alerts present Input Tokens 3 88134.67 118508.00 -30373.33 0.15 No -
obs alerts routing - no alerts present Output Tokens 3 1653.00 2227.33 -574.33 0.13 No -
obs alerts routing - no alerts present Skill Invoked (investigation) 3 0.00 0.00 0.00 1.00 No -
obs alerts routing - no alerts present Tool Calls 3 2.67 4.00 -1.33 0.13 No -
obs alerts routing - no alerts present trajectory 3 0.00 0.00 0.00 1.00 No -
obs alerts routing across screen contexts Cached Tokens 1 32690.00 34740.00 -2050.00 - n/a -
obs alerts routing across screen contexts Criteria 5 0.60 0.67 -0.07 0.33 No -
obs alerts routing across screen contexts Input Tokens 5 143296.40 80214.40 +63082.00 0.17 No -
obs alerts routing across screen contexts Output Tokens 5 2526.00 1403.00 +1123.00 0.25 No -
obs alerts routing across screen contexts Skill Invoked (investigation) 5 0.00 0.00 0.00 1.00 No -
obs alerts routing across screen contexts Tool Calls 5 3.60 3.20 +0.40 0.58 No -
obs alerts routing across screen contexts trajectory 5 0.00 0.00 0.00 1.00 No -
payment-failure investigation Criteria 1 1.00 1.00 0.00 - n/a -
payment-failure investigation Input Tokens 1 169594.00 131105.00 +38489.00 - n/a -
payment-failure investigation Output Tokens 1 1284.00 1187.00 +97.00 - n/a -
payment-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
payment-failure investigation Tool Calls 1 11.00 11.00 0.00 - n/a -
payment-failure investigation trajectory 1 1.00 1.00 0.00 - n/a -
payment-unreachable investigation Criteria 1 0.92 1.00 -0.08 - n/a -
payment-unreachable investigation Input Tokens 1 103218.00 72363.00 +30855.00 - n/a -
payment-unreachable investigation Output Tokens 1 814.00 803.00 +11.00 - n/a -
payment-unreachable investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
payment-unreachable investigation Tool Calls 1 7.00 7.00 0.00 - n/a -
payment-unreachable investigation trajectory 1 1.00 1.00 0.00 - n/a -
service-map skill activation Criteria 4 1.00 1.00 0.00 1.00 No -
service-map skill activation Input Tokens 4 55017.50 36381.50 +18636.00 0.07 No -
service-map skill activation Output Tokens 4 983.25 232.50 +750.75 0.12 No -
service-map skill activation Skill Invoked (service-map) 4 0.00 0.00 0.00 1.00 No -
service-map skill activation Tool Calls 4 3.25 2.25 +1.00 0.25 No -
service-map skill activation trajectory 4 1.00 1.00 0.00 1.00 No -
tool-arguments: default time fallback Criteria 1 1.00 1.00 0.00 - n/a -
tool-arguments: default time fallback Input Tokens 1 14873.00 70228.00 -55355.00 - n/a -
tool-arguments: default time fallback Output Tokens 1 72.00 283.00 -211.00 - n/a -
tool-arguments: default time fallback Tool Calls 1 1.00 3.00 -2.00 - n/a -
tool-arguments: user time override Criteria 1 1.00 1.00 0.00 - n/a -
tool-arguments: user time override Input Tokens 1 268560.00 219591.00 +48969.00 - n/a -
tool-arguments: user time override Output Tokens 1 1363.00 1209.00 +154.00 - n/a -
tool-arguments: user time override Tool Calls 1 10.00 10.00 0.00 - n/a -
tool-arguments: uses screen context time range Criteria 1 1.00 1.00 0.00 - n/a -
tool-arguments: uses screen context time range Input Tokens 1 272091.00 15027.00 +257064.00 - n/a -
tool-arguments: uses screen context time range Output Tokens 1 1113.00 133.00 +980.00 - n/a -
tool-arguments: uses screen context time range Tool Calls 1 10.00 2.00 +8.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::observability-ai::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::observability-ai::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (142 evaluator comparisons).

Note: 5 unpaired (ran in only one experiment), 106 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (142 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
ad-failure investigation Criteria 1 1.00 0.50 +0.50 - n/a -
ad-failure investigation Input Tokens 1 118142.00 566673.00 -448531.00 - n/a -
ad-failure investigation Output Tokens 1 1218.00 2278.00 -1060.00 - n/a -
ad-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
ad-failure investigation Tool Calls 1 13.00 18.00 -5.00 - n/a -
ad-failure investigation trajectory 1 1.00 1.00 0.00 - n/a -
ad-high-cpu investigation Cached Tokens 1 14755.00 11149.00 +3606.00 - n/a -
ad-high-cpu investigation Criteria 1 1.00 0.67 +0.33 - n/a -
ad-high-cpu investigation Input Tokens 1 260895.00 259251.00 +1644.00 - n/a -
ad-high-cpu investigation Output Tokens 1 1258.00 1930.00 -672.00 - n/a -
ad-high-cpu investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
ad-high-cpu investigation Tool Calls 1 17.00 17.00 0.00 - n/a -
ad-high-cpu investigation trajectory 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Factuality 1 0.61 0.30 +0.30 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Input Tokens 1 3653.00 3662.00 -9.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Output Tokens 1 363.00 406.00 -43.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Relevance 1 0.70 1.00 -0.30 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Input Tokens 1 3191.00 3190.00 +1.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Output Tokens 1 394.00 359.00 +35.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Relevance 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Input Tokens 1 3639.00 3640.00 -1.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Output Tokens 1 399.00 425.00 -26.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Relevance 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Factuality 1 0.32 0.00 +0.32 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Input Tokens 1 14865.00 14865.00 0.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Output Tokens 1 1232.00 1239.00 -7.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Relevance 1 0.91 0.77 +0.14 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Factuality 1 0.88 0.66 +0.23 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Input Tokens 1 16433.00 16434.00 -1.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Output Tokens 1 1068.00 1089.00 -21.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Relevance 1 0.89 0.80 +0.09 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Input Tokens 1 12215.00 12215.00 0.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Output Tokens 1 1359.00 1203.00 +156.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Relevance 1 1.00 0.75 +0.25 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Factuality 1 0.71 0.70 +0.01 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Input Tokens 1 9629.00 9629.00 0.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Output Tokens 1 604.00 620.00 -16.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Relevance 1 0.56 0.50 +0.06 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Factuality 1 0.75 0.80 -0.06 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Input Tokens 1 12661.00 12662.00 -1.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Output Tokens 1 1063.00 931.00 +132.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Relevance 1 0.64 0.50 +0.14 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Input Tokens 1 7948.00 7947.00 +1.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Output Tokens 1 1095.00 1216.00 -121.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Relevance 1 0.86 0.71 +0.14 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
alert investigation — narrow screen context vs wide alert window Cached Tokens 1 592439.00 535366.00 +57073.00 - n/a -
alert investigation — narrow screen context vs wide alert window Criteria 1 1.00 1.00 0.00 - n/a -
alert investigation — narrow screen context vs wide alert window Input Tokens 1 1131466.00 1234191.00 -102725.00 - n/a -
alert investigation — narrow screen context vs wide alert window Output Tokens 1 4981.00 3670.00 +1311.00 - n/a -
alert investigation — narrow screen context vs wide alert window Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
alert investigation — narrow screen context vs wide alert window Tool Calls 1 42.00 34.00 +8.00 - n/a -
alert investigation — narrow screen context vs wide alert window trajectory 1 1.00 1.00 0.00 - n/a -
alert time window — discovered via get_alerts Criteria 1 1.00 0.80 +0.20 - n/a -
alert time window — discovered via get_alerts Input Tokens 1 358838.00 26080.00 +332758.00 - n/a -
alert time window — discovered via get_alerts Output Tokens 1 3323.00 54.00 +3269.00 - n/a -
alert time window — discovered via get_alerts Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
alert time window — discovered via get_alerts Tool Calls 1 16.00 1.00 +15.00 - n/a -
alert time window — discovered via get_alerts trajectory 1 1.00 0.00 +1.00 - n/a -
cart-failure investigation Criteria 1 0.67 0.83 -0.17 - n/a -
cart-failure investigation Input Tokens 1 57254.00 104389.00 -47135.00 - n/a -
cart-failure investigation Output Tokens 1 429.00 694.00 -265.00 - n/a -
cart-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
cart-failure investigation Tool Calls 1 6.00 10.00 -4.00 - n/a -
cart-failure investigation trajectory 1 1.00 1.00 0.00 - n/a -
email-memory-leak investigation Cached Tokens 1 108553.00 737359.00 -628806.00 - n/a -
email-memory-leak investigation Criteria 1 0.83 0.67 +0.17 - n/a -
email-memory-leak investigation Input Tokens 1 668946.00 1268572.00 -599626.00 - n/a -
email-memory-leak investigation Output Tokens 1 3644.00 5128.00 -1484.00 - n/a -
email-memory-leak investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
email-memory-leak investigation Tool Calls 1 32.00 40.00 -8.00 - n/a -
email-memory-leak investigation trajectory 1 1.00 1.00 0.00 - n/a -
kafka-queue-problems investigation Criteria 1 0.20 0.20 0.00 - n/a -
kafka-queue-problems investigation Input Tokens 1 110886.00 634902.00 -524016.00 - n/a -
kafka-queue-problems investigation Output Tokens 1 863.00 3250.00 -2387.00 - n/a -
kafka-queue-problems investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
kafka-queue-problems investigation Tool Calls 1 8.00 22.00 -14.00 - n/a -
kafka-queue-problems investigation trajectory 1 1.00 1.00 0.00 - n/a -
load-generator-flood-homepage investigation Criteria 1 0.60 1.00 -0.40 - n/a -
load-generator-flood-homepage investigation Input Tokens 1 110721.00 81692.00 +29029.00 - n/a -
load-generator-flood-homepage investigation Output Tokens 1 1230.00 602.00 +628.00 - n/a -
load-generator-flood-homepage investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
load-generator-flood-homepage investigation Tool Calls 1 11.00 8.00 +3.00 - n/a -
load-generator-flood-homepage investigation trajectory 1 0.67 1.00 -0.33 - n/a -
obs alerts routing - no alerts present Cached Tokens 1 101949.00 32951.00 +68998.00 - n/a -
obs alerts routing - no alerts present Criteria 3 0.33 0.67 -0.33 0.35 No -
obs alerts routing - no alerts present Input Tokens 3 110562.00 358706.00 -248144.00 0.42 No -
obs alerts routing - no alerts present Output Tokens 3 1154.33 1824.67 -670.33 0.59 No -
obs alerts routing - no alerts present Skill Invoked (investigation) 3 0.00 0.00 0.00 1.00 No -
obs alerts routing - no alerts present Tool Calls 3 4.00 11.00 -7.00 0.40 No -
obs alerts routing - no alerts present trajectory 3 0.00 0.33 -0.33 0.35 No -
obs alerts routing across screen contexts Cached Tokens 3 18304.33 27500.33 -9196.00 0.68 No -
obs alerts routing across screen contexts Criteria 5 0.93 0.80 +0.13 0.14 No -
obs alerts routing across screen contexts Input Tokens 5 83134.60 101715.80 -18581.20 0.70 No -
obs alerts routing across screen contexts Output Tokens 5 902.80 1092.60 -189.80 0.75 No -
obs alerts routing across screen contexts Skill Invoked (investigation) 5 0.00 0.00 0.00 1.00 No -
obs alerts routing across screen contexts Tool Calls 5 3.60 3.80 -0.20 0.88 No -
obs alerts routing across screen contexts trajectory 5 0.80 0.40 +0.40 0.14 No -
payment-failure investigation Criteria 1 1.00 1.00 0.00 - n/a -
payment-failure investigation Input Tokens 1 56641.00 252591.00 -195950.00 - n/a -
payment-failure investigation Output Tokens 1 325.00 1300.00 -975.00 - n/a -
payment-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
payment-failure investigation Tool Calls 1 6.00 14.00 -8.00 - n/a -
payment-failure investigation trajectory 1 1.00 1.00 0.00 - n/a -
payment-unreachable investigation Criteria 1 0.69 0.69 0.00 - n/a -
payment-unreachable investigation Input Tokens 1 34367.00 130818.00 -96451.00 - n/a -
payment-unreachable investigation Output Tokens 1 265.00 875.00 -610.00 - n/a -
payment-unreachable investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
payment-unreachable investigation Tool Calls 1 4.00 9.00 -5.00 - n/a -
payment-unreachable investigation trajectory 1 1.00 1.00 0.00 - n/a -
service-map skill activation Cached Tokens 1 43503.00 21840.00 +21663.00 - n/a -
service-map skill activation Criteria 4 1.00 1.00 0.00 1.00 No -
service-map skill activation Input Tokens 4 57860.00 58577.50 -717.50 0.90 No -
service-map skill activation Output Tokens 4 155.75 214.00 -58.25 0.54 No -
service-map skill activation Skill Invoked (service-map) 4 0.00 0.00 0.00 1.00 No -
service-map skill activation Tool Calls 4 3.00 3.00 0.00 1.00 No -
service-map skill activation trajectory 4 1.00 1.00 0.00 1.00 No -
tool-arguments: default time fallback Criteria 1 1.00 1.00 0.00 - n/a -
tool-arguments: default time fallback Input Tokens 1 237769.00 55816.00 +181953.00 - n/a -
tool-arguments: default time fallback Output Tokens 1 1014.00 221.00 +793.00 - n/a -
tool-arguments: default time fallback Tool Calls 1 10.00 3.00 +7.00 - n/a -
tool-arguments: user time override Criteria 1 1.00 1.00 0.00 - n/a -
tool-arguments: user time override Input Tokens 1 227053.00 12266.00 +214787.00 - n/a -
tool-arguments: user time override Output Tokens 1 872.00 42.00 +830.00 - n/a -
tool-arguments: user time override Tool Calls 1 8.00 1.00 +7.00 - n/a -
tool-arguments: uses screen context time range Cached Tokens 1 43030.00 11147.00 +31883.00 - n/a -
tool-arguments: uses screen context time range Criteria 1 0.00 1.00 -1.00 - n/a -
tool-arguments: uses screen context time range Input Tokens 1 179593.00 12256.00 +167337.00 - n/a -
tool-arguments: uses screen context time range Output Tokens 1 486.00 40.00 +446.00 - n/a -
tool-arguments: uses screen context time range Tool Calls 1 8.00 1.00 +7.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::observability-ai::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::observability-ai::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (146 evaluator comparisons).

Note: 1 unpaired (ran in only one experiment), 99 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (146 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
ad-failure investigation Criteria 1 0.83 1.00 -0.17 - n/a -
ad-failure investigation Input Tokens 1 1051195.00 57600.00 +993595.00 - n/a -
ad-failure investigation Output Tokens 1 1418.00 713.00 +705.00 - n/a -
ad-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
ad-failure investigation Tool Calls 1 26.00 7.00 +19.00 - n/a -
ad-failure investigation trajectory 1 1.00 1.00 0.00 - n/a -
ad-high-cpu investigation Cached Tokens 1 255277.00 569026.00 -313749.00 - n/a -
ad-high-cpu investigation Criteria 1 1.00 1.00 0.00 - n/a -
ad-high-cpu investigation Input Tokens 1 1063925.00 929783.00 +134142.00 - n/a -
ad-high-cpu investigation Output Tokens 1 1722.00 3164.00 -1442.00 - n/a -
ad-high-cpu investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
ad-high-cpu investigation Tool Calls 1 31.00 29.00 +2.00 - n/a -
ad-high-cpu investigation trajectory 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Factuality 1 0.39 0.00 +0.39 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Input Tokens 1 3666.00 3651.00 +15.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Output Tokens 1 403.00 385.00 +18.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Relevance 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Factuality 1 0.00 0.69 -0.69 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Input Tokens 1 3189.00 3190.00 -1.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Output Tokens 1 396.00 397.00 -1.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Relevance 1 0.90 0.89 +0.01 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Input Tokens 1 3641.00 3635.00 +6.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Output Tokens 1 348.00 353.00 -5.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Relevance 1 1.00 0.78 +0.22 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Factuality 1 0.62 0.43 +0.19 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Input Tokens 1 14862.00 14862.00 0.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Output Tokens 1 1181.00 1067.00 +114.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Relevance 1 0.64 0.80 -0.16 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Factuality 1 0.85 0.72 +0.13 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Input Tokens 1 16434.00 16432.00 +2.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Output Tokens 1 1152.00 1178.00 -26.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Relevance 1 0.82 0.57 +0.25 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Factuality 1 0.55 0.00 +0.55 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Input Tokens 1 12216.00 12216.00 0.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Output Tokens 1 1237.00 1259.00 -22.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Relevance 1 0.75 0.88 -0.13 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Factuality 1 0.70 0.75 -0.06 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Input Tokens 1 9628.00 9630.00 -2.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Output Tokens 1 605.00 634.00 -29.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Relevance 1 0.50 0.67 -0.17 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Factuality 1 0.66 0.00 +0.66 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Input Tokens 1 12662.00 12663.00 -1.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Output Tokens 1 1012.00 1051.00 -39.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Relevance 1 0.75 0.78 -0.03 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Input Tokens 1 7945.00 7947.00 -2.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Output Tokens 1 1307.00 1088.00 +219.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Relevance 1 0.89 0.83 +0.06 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
alert investigation — narrow screen context vs wide alert window Cached Tokens 1 18006.00 18011.00 -5.00 - n/a -
alert investigation — narrow screen context vs wide alert window Criteria 1 0.80 0.80 0.00 - n/a -
alert investigation — narrow screen context vs wide alert window Input Tokens 1 239170.00 109859.00 +129311.00 - n/a -
alert investigation — narrow screen context vs wide alert window Output Tokens 1 1799.00 1387.00 +412.00 - n/a -
alert investigation — narrow screen context vs wide alert window Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
alert investigation — narrow screen context vs wide alert window Tool Calls 1 14.00 9.00 +5.00 - n/a -
alert investigation — narrow screen context vs wide alert window trajectory 1 1.00 1.00 0.00 - n/a -
alert time window — discovered via get_alerts Criteria 1 1.00 0.60 +0.40 - n/a -
alert time window — discovered via get_alerts Input Tokens 1 84941.00 370309.00 -285368.00 - n/a -
alert time window — discovered via get_alerts Output Tokens 1 1130.00 9966.00 -8836.00 - n/a -
alert time window — discovered via get_alerts Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
alert time window — discovered via get_alerts Tool Calls 1 9.00 8.00 +1.00 - n/a -
alert time window — discovered via get_alerts trajectory 1 1.00 0.00 +1.00 - n/a -
cart-failure investigation Criteria 1 0.50 0.83 -0.33 - n/a -
cart-failure investigation Input Tokens 1 34324.00 56536.00 -22212.00 - n/a -
cart-failure investigation Output Tokens 1 544.00 620.00 -76.00 - n/a -
cart-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
cart-failure investigation Tool Calls 1 4.00 4.00 0.00 - n/a -
cart-failure investigation trajectory 1 0.67 0.67 0.00 - n/a -
email-memory-leak investigation Cached Tokens 1 674925.00 743261.00 -68336.00 - n/a -
email-memory-leak investigation Criteria 1 0.83 0.67 +0.17 - n/a -
email-memory-leak investigation Input Tokens 1 978169.00 1595290.00 -617121.00 - n/a -
email-memory-leak investigation Output Tokens 1 2635.00 6694.00 -4059.00 - n/a -
email-memory-leak investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
email-memory-leak investigation Tool Calls 1 36.00 60.00 -24.00 - n/a -
email-memory-leak investigation trajectory 1 0.50 1.00 -0.50 - n/a -
kafka-queue-problems investigation Cached Tokens 1 268346.00 14686.00 +253660.00 - n/a -
kafka-queue-problems investigation Criteria 1 0.20 0.20 0.00 - n/a -
kafka-queue-problems investigation Input Tokens 1 666556.00 56973.00 +609583.00 - n/a -
kafka-queue-problems investigation Output Tokens 1 2833.00 1270.00 +1563.00 - n/a -
kafka-queue-problems investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
kafka-queue-problems investigation Tool Calls 1 25.00 8.00 +17.00 - n/a -
kafka-queue-problems investigation trajectory 1 0.67 1.00 -0.33 - n/a -
load-generator-flood-homepage investigation Cached Tokens 1 160912.00 480702.00 -319790.00 - n/a -
load-generator-flood-homepage investigation Criteria 1 0.20 0.80 -0.60 - n/a -
load-generator-flood-homepage investigation Input Tokens 1 478465.00 1833968.00 -1355503.00 - n/a -
load-generator-flood-homepage investigation Output Tokens 1 1806.00 4434.00 -2628.00 - n/a -
load-generator-flood-homepage investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
load-generator-flood-homepage investigation Tool Calls 1 23.00 66.00 -43.00 - n/a -
load-generator-flood-homepage investigation trajectory 1 0.67 1.00 -0.33 - n/a -
obs alerts routing - no alerts present Cached Tokens 2 55281.50 65640.50 -10359.00 0.77 No -
obs alerts routing - no alerts present Criteria 3 0.67 0.67 0.00 1.00 No -
obs alerts routing - no alerts present Input Tokens 3 113313.33 145383.67 -32070.33 0.56 No -
obs alerts routing - no alerts present Output Tokens 3 2084.00 3681.33 -1597.33 0.29 No -
obs alerts routing - no alerts present Skill Invoked (investigation) 3 0.00 0.00 0.00 1.00 No -
obs alerts routing - no alerts present Tool Calls 3 3.33 5.33 -2.00 0.29 No -
obs alerts routing - no alerts present trajectory 3 0.67 0.00 +0.67 0.13 No -
obs alerts routing across screen contexts Cached Tokens 3 91056.00 130642.33 -39586.33 0.68 No -
obs alerts routing across screen contexts Criteria 5 0.87 0.80 +0.07 0.67 No -
obs alerts routing across screen contexts Input Tokens 5 142677.40 277680.60 -135003.20 0.35 No -
obs alerts routing across screen contexts Output Tokens 5 3617.20 4074.00 -456.80 0.86 No -
obs alerts routing across screen contexts Skill Invoked (investigation) 5 0.00 0.00 0.00 1.00 No -
obs alerts routing across screen contexts Tool Calls 5 4.40 7.60 -3.20 0.37 No -
obs alerts routing across screen contexts trajectory 5 0.40 0.80 -0.40 0.14 No -
payment-failure investigation Cached Tokens 1 43305.00 189460.00 -146155.00 - n/a -
payment-failure investigation Criteria 1 1.00 1.00 0.00 - n/a -
payment-failure investigation Input Tokens 1 192056.00 511467.00 -319411.00 - n/a -
payment-failure investigation Output Tokens 1 668.00 1079.00 -411.00 - n/a -
payment-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
payment-failure investigation Tool Calls 1 9.00 16.00 -7.00 - n/a -
payment-failure investigation trajectory 1 1.00 1.00 0.00 - n/a -
payment-unreachable investigation Cached Tokens 1 17992.00 130260.00 -112268.00 - n/a -
payment-unreachable investigation Criteria 1 0.77 0.92 -0.15 - n/a -
payment-unreachable investigation Input Tokens 1 164217.00 403308.00 -239091.00 - n/a -
payment-unreachable investigation Output Tokens 1 812.00 1886.00 -1074.00 - n/a -
payment-unreachable investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
payment-unreachable investigation Tool Calls 1 8.00 15.00 -7.00 - n/a -
payment-unreachable investigation trajectory 1 0.67 0.67 0.00 - n/a -
service-map skill activation Cached Tokens 1 22178.00 7384.00 +14794.00 - n/a -
service-map skill activation Criteria 4 1.00 1.00 0.00 1.00 No -
service-map skill activation Input Tokens 4 36524.00 41765.25 -5241.25 0.64 No -
service-map skill activation Output Tokens 4 508.00 1305.75 -797.75 0.12 No -
service-map skill activation Skill Invoked (service-map) 4 0.00 0.00 0.00 1.00 No -
service-map skill activation Tool Calls 4 2.75 3.25 -0.50 0.44 No -
service-map skill activation trajectory 4 1.00 0.88 +0.13 0.33 No -
tool-arguments: default time fallback Criteria 1 1.00 0.00 +1.00 - n/a -
tool-arguments: default time fallback Input Tokens 1 12109.00 12109.00 0.00 - n/a -
tool-arguments: default time fallback Output Tokens 1 461.00 160.00 +301.00 - n/a -
tool-arguments: default time fallback Tool Calls 1 1.00 0.00 +1.00 - n/a -
tool-arguments: user time override Criteria 1 1.00 1.00 0.00 - n/a -
tool-arguments: user time override Input Tokens 1 34309.00 242.00 +34067.00 - n/a -
tool-arguments: user time override Output Tokens 1 352.00 24.00 +328.00 - n/a -
tool-arguments: user time override Tool Calls 1 3.00 0.00 +3.00 - n/a -
tool-arguments: uses screen context time range Cached Tokens 1 103926.00 146586.00 -42660.00 - n/a -
tool-arguments: uses screen context time range Criteria 1 1.00 1.00 0.00 - n/a -
tool-arguments: uses screen context time range Input Tokens 1 390299.00 322735.00 +67564.00 - n/a -
tool-arguments: uses screen context time range Output Tokens 1 1037.00 938.00 +99.00 - n/a -
tool-arguments: uses screen context time range Tool Calls 1 16.00 13.00 +3.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::observability-ai::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::observability-ai::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 148 comparisons.

Note: 91 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
service-map skill activation Cached Tokens 4 21056.00 6080.00 +14976.00 0.01 Yes -
No significant changes (147 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
ad-failure investigation Cached Tokens 1 14592.00 9728.00 +4864.00 - n/a -
ad-failure investigation Criteria 1 0.67 1.00 -0.33 - n/a -
ad-failure investigation Input Tokens 1 81571.00 56749.00 +24822.00 - n/a -
ad-failure investigation Output Tokens 1 751.00 530.00 +221.00 - n/a -
ad-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
ad-failure investigation Tool Calls 1 9.00 7.00 +2.00 - n/a -
ad-failure investigation trajectory 1 1.00 1.00 0.00 - n/a -
ad-high-cpu investigation Cached Tokens 1 24320.00 31872.00 -7552.00 - n/a -
ad-high-cpu investigation Criteria 1 1.00 1.00 0.00 - n/a -
ad-high-cpu investigation Input Tokens 1 146711.00 123207.00 +23504.00 - n/a -
ad-high-cpu investigation Output Tokens 1 1617.00 1237.00 +380.00 - n/a -
ad-high-cpu investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
ad-high-cpu investigation Tool Calls 1 19.00 14.00 +5.00 - n/a -
ad-high-cpu investigation trajectory 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Input Tokens 1 3659.00 3667.00 -8.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Output Tokens 1 361.00 373.00 -12.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Relevance 1 1.00 0.75 +0.25 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Input Tokens 1 3196.00 3192.00 +4.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Output Tokens 1 351.00 390.00 -39.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Relevance 1 1.00 0.90 +0.10 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Input Tokens 1 3637.00 3636.00 +1.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Output Tokens 1 472.00 389.00 +83.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Relevance 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Factuality 1 0.51 0.48 +0.03 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Input Tokens 1 14865.00 14868.00 -3.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Output Tokens 1 1174.00 1203.00 -29.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Relevance 1 0.79 0.77 +0.02 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Factuality 1 0.65 0.57 +0.08 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Input Tokens 1 16433.00 16436.00 -3.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Output Tokens 1 999.00 1093.00 -94.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Relevance 1 0.89 0.85 +0.05 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Factuality 1 0.62 0.00 +0.62 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Input Tokens 1 12215.00 12219.00 -4.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Output Tokens 1 1179.00 1370.00 -191.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Relevance 1 0.75 0.88 -0.13 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Factuality 1 0.53 0.78 -0.25 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Input Tokens 1 9629.00 9632.00 -3.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Output Tokens 1 536.00 614.00 -78.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Relevance 1 0.44 0.67 -0.22 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Factuality 1 0.00 0.56 -0.56 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Input Tokens 1 12662.00 12664.00 -2.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Output Tokens 1 1223.00 947.00 +276.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Relevance 1 0.80 0.69 +0.11 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Input Tokens 1 7948.00 7951.00 -3.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Output Tokens 1 1249.00 1122.00 +127.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Relevance 1 1.00 0.67 +0.33 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
alert investigation — narrow screen context vs wide alert window Criteria 1 0.80 1.00 -0.20 - n/a -
alert investigation — narrow screen context vs wide alert window Input Tokens 1 11906.00 228369.00 -216463.00 - n/a -
alert investigation — narrow screen context vs wide alert window Output Tokens 1 124.00 2783.00 -2659.00 - n/a -
alert investigation — narrow screen context vs wide alert window Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
alert investigation — narrow screen context vs wide alert window Tool Calls 1 3.00 11.00 -8.00 - n/a -
alert investigation — narrow screen context vs wide alert window trajectory 1 0.00 0.00 0.00 - n/a -
alert time window — discovered via get_alerts Cached Tokens 1 19584.00 24434.00 -4850.00 - n/a -
alert time window — discovered via get_alerts Criteria 1 0.60 0.80 -0.20 - n/a -
alert time window — discovered via get_alerts Input Tokens 1 150441.00 113511.00 +36930.00 - n/a -
alert time window — discovered via get_alerts Output Tokens 1 2739.00 1967.00 +772.00 - n/a -
alert time window — discovered via get_alerts Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
alert time window — discovered via get_alerts Tool Calls 1 4.00 4.00 0.00 - n/a -
alert time window — discovered via get_alerts trajectory 1 0.00 0.00 0.00 - n/a -
cart-failure investigation Cached Tokens 1 26240.00 9728.00 +16512.00 - n/a -
cart-failure investigation Criteria 1 0.83 0.83 0.00 - n/a -
cart-failure investigation Input Tokens 1 83058.00 56384.00 +26674.00 - n/a -
cart-failure investigation Output Tokens 1 921.00 650.00 +271.00 - n/a -
cart-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
cart-failure investigation Tool Calls 1 11.00 8.00 +3.00 - n/a -
cart-failure investigation trajectory 1 1.00 1.00 0.00 - n/a -
email-memory-leak investigation Cached Tokens 1 34816.00 20352.00 +14464.00 - n/a -
email-memory-leak investigation Criteria 1 0.83 0.83 0.00 - n/a -
email-memory-leak investigation Input Tokens 1 56409.00 163609.00 -107200.00 - n/a -
email-memory-leak investigation Output Tokens 1 880.00 1540.00 -660.00 - n/a -
email-memory-leak investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
email-memory-leak investigation Tool Calls 1 11.00 19.00 -8.00 - n/a -
email-memory-leak investigation trajectory 1 1.00 0.50 +0.50 - n/a -
kafka-queue-problems investigation Cached Tokens 1 29952.00 14592.00 +15360.00 - n/a -
kafka-queue-problems investigation Criteria 1 0.20 0.20 0.00 - n/a -
kafka-queue-problems investigation Input Tokens 1 55985.00 79083.00 -23098.00 - n/a -
kafka-queue-problems investigation Output Tokens 1 727.00 729.00 -2.00 - n/a -
kafka-queue-problems investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
kafka-queue-problems investigation Tool Calls 1 9.00 9.00 0.00 - n/a -
kafka-queue-problems investigation trajectory 1 1.00 1.00 0.00 - n/a -
load-generator-flood-homepage investigation Cached Tokens 1 39680.00 45952.00 -6272.00 - n/a -
load-generator-flood-homepage investigation Criteria 1 0.60 0.40 +0.20 - n/a -
load-generator-flood-homepage investigation Input Tokens 1 81851.00 103296.00 -21445.00 - n/a -
load-generator-flood-homepage investigation Output Tokens 1 762.00 1136.00 -374.00 - n/a -
load-generator-flood-homepage investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
load-generator-flood-homepage investigation Tool Calls 1 10.00 13.00 -3.00 - n/a -
load-generator-flood-homepage investigation trajectory 1 0.67 1.00 -0.33 - n/a -
obs alerts routing - no alerts present Cached Tokens 3 25344.00 20906.67 +4437.33 0.46 No -
obs alerts routing - no alerts present Criteria 3 0.00 0.00 0.00 1.00 No -
obs alerts routing - no alerts present Input Tokens 3 79355.67 37162.67 +42193.00 0.18 No -
obs alerts routing - no alerts present Output Tokens 3 1669.00 394.67 +1274.33 0.10 No -
obs alerts routing - no alerts present Skill Invoked (investigation) 3 0.00 0.00 0.00 1.00 No -
obs alerts routing - no alerts present Tool Calls 3 2.67 2.00 +0.67 0.35 No -
obs alerts routing - no alerts present trajectory 3 0.00 0.00 0.00 1.00 No -
obs alerts routing across screen contexts Cached Tokens 4 33496.25 16896.00 +16600.25 0.46 No -
obs alerts routing across screen contexts Criteria 5 0.60 0.67 -0.07 0.33 No -
obs alerts routing across screen contexts Input Tokens 5 85238.20 90761.60 -5523.40 0.84 No -
obs alerts routing across screen contexts Output Tokens 5 1360.40 1678.40 -318.00 0.53 No -
obs alerts routing across screen contexts Skill Invoked (investigation) 5 0.00 0.00 0.00 1.00 No -
obs alerts routing across screen contexts Tool Calls 5 2.40 2.80 -0.40 0.58 No -
obs alerts routing across screen contexts trajectory 5 0.00 0.00 0.00 1.00 No -
payment-failure investigation Cached Tokens 1 25856.00 14592.00 +11264.00 - n/a -
payment-failure investigation Criteria 1 1.00 1.00 0.00 - n/a -
payment-failure investigation Input Tokens 1 83308.00 80191.00 +3117.00 - n/a -
payment-failure investigation Output Tokens 1 927.00 831.00 +96.00 - n/a -
payment-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
payment-failure investigation Tool Calls 1 11.00 10.00 +1.00 - n/a -
payment-failure investigation trajectory 1 1.00 1.00 0.00 - n/a -
payment-unreachable investigation Cached Tokens 1 9728.00 35072.00 -25344.00 - n/a -
payment-unreachable investigation Criteria 1 0.92 0.92 0.00 - n/a -
payment-unreachable investigation Input Tokens 1 56468.00 56461.00 +7.00 - n/a -
payment-unreachable investigation Output Tokens 1 663.00 659.00 +4.00 - n/a -
payment-unreachable investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
payment-unreachable investigation Tool Calls 1 8.00 8.00 0.00 - n/a -
payment-unreachable investigation trajectory 1 1.00 1.00 0.00 - n/a -
service-map skill activation Criteria 4 1.00 1.00 0.00 1.00 No -
service-map skill activation Input Tokens 4 28324.25 32920.00 -4595.75 0.62 No -
service-map skill activation Output Tokens 4 248.75 178.00 +70.75 0.53 No -
service-map skill activation Skill Invoked (service-map) 4 0.00 0.00 0.00 1.00 No -
service-map skill activation Tool Calls 4 1.75 2.25 -0.50 0.62 No -
service-map skill activation trajectory 4 0.88 0.88 0.00 1.00 No -
tool-arguments: default time fallback Cached Tokens 1 9728.00 4864.00 +4864.00 - n/a -
tool-arguments: default time fallback Criteria 1 1.00 1.00 0.00 - n/a -
tool-arguments: default time fallback Input Tokens 1 56979.00 34243.00 +22736.00 - n/a -
tool-arguments: default time fallback Output Tokens 1 234.00 151.00 +83.00 - n/a -
tool-arguments: default time fallback Tool Calls 1 4.00 3.00 +1.00 - n/a -
tool-arguments: user time override Criteria 1 1.00 1.00 0.00 - n/a -
tool-arguments: user time override Input Tokens 1 242.00 11792.00 -11550.00 - n/a -
tool-arguments: user time override Output Tokens 1 18.00 109.00 -91.00 - n/a -
tool-arguments: user time override Tool Calls 1 0.00 2.00 -2.00 - n/a -
tool-arguments: uses screen context time range Criteria 1 1.00 1.00 0.00 - n/a -
tool-arguments: uses screen context time range Input Tokens 1 11781.00 78182.00 -66401.00 - n/a -
tool-arguments: uses screen context time range Output Tokens 1 76.00 449.00 -373.00 - n/a -
tool-arguments: uses screen context time range Tool Calls 1 1.00 6.00 -5.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::observability-ai::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::observability-ai::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
1 significant difference(s) detected out of 138 comparisons.

Note: 13 unpaired (ran in only one experiment), 110 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
service-map skill activation Cached Tokens 3 8266.00 11702.67 -3436.67 0.00 Yes -
No significant changes (137 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
ad-failure investigation Criteria 1 0.17 0.17 0.00 - n/a -
ad-failure investigation Input Tokens 1 12102.00 50605.00 -38503.00 - n/a -
ad-failure investigation Output Tokens 1 253.00 493.00 -240.00 - n/a -
ad-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
ad-failure investigation Tool Calls 1 2.00 3.00 -1.00 - n/a -
ad-failure investigation trajectory 1 0.00 0.00 0.00 - n/a -
ad-high-cpu investigation Criteria 1 0.50 0.17 +0.33 - n/a -
ad-high-cpu investigation Input Tokens 1 316445.00 14632.00 +301813.00 - n/a -
ad-high-cpu investigation Output Tokens 1 5860.00 104.00 +5756.00 - n/a -
ad-high-cpu investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
ad-high-cpu investigation Tool Calls 1 18.00 1.00 +17.00 - n/a -
ad-high-cpu investigation trajectory 1 0.00 0.00 0.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Factuality 1 0.32 0.00 +0.32 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Input Tokens 1 3656.00 3660.00 -4.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Output Tokens 1 414.00 390.00 +24.00 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Relevance 1 0.86 0.64 +0.22 - n/a -
ai insights: alert analysis (ad-high-cpu-alert, ad-high-cpu) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Input Tokens 1 3188.00 3195.00 -7.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Output Tokens 1 363.00 363.00 0.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Relevance 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-error-count-alert, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Input Tokens 1 3637.00 3635.00 +2.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Output Tokens 1 380.00 380.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Relevance 1 1.00 1.00 0.00 - n/a -
ai insights: alert analysis (payment-unreachable-alert, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Factuality 1 0.46 0.00 +0.46 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Input Tokens 1 14866.00 14862.00 +4.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Output Tokens 1 1195.00 1217.00 -22.00 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Relevance 1 0.80 0.69 +0.11 - n/a -
ai insights: APM error analysis (payment-service-failure, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Factuality 1 0.76 0.72 +0.04 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Input Tokens 1 16435.00 16433.00 +2.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Output Tokens 1 1095.00 1102.00 -7.00 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Relevance 1 0.92 0.67 +0.26 - n/a -
ai insights: APM error analysis (payment-unreachable, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Input Tokens 1 12216.00 12216.00 0.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Output Tokens 1 1179.00 1159.00 +20.00 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Relevance 1 0.69 0.73 -0.04 - n/a -
ai insights: APM error analysis (product-catalog-failure, product-catalog) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Factuality 1 0.00 0.77 -0.77 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Input Tokens 1 9632.00 9627.00 +5.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Output Tokens 1 654.00 565.00 +89.00 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Relevance 1 0.50 0.67 -0.17 - n/a -
ai insights: log analysis (info-log, payment-service-failures) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Factuality 1 0.61 0.76 -0.15 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Input Tokens 1 12663.00 12662.00 +1.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Output Tokens 1 1009.00 1007.00 +2.00 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Relevance 1 0.58 0.64 -0.05 - n/a -
ai insights: log analysis (payment-unreachable-error-log, payment-unreachable) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Factuality 1 0.00 0.00 0.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Input Tokens 1 7949.00 7947.00 +2.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Output Tokens 1 1140.00 1055.00 +85.00 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Relevance 1 0.67 0.92 -0.26 - n/a -
ai insights: log analysis (product-catalog-error-log, product-catalog) Sequence Accuracy 1 1.00 1.00 0.00 - n/a -
alert investigation — narrow screen context vs wide alert window Criteria 1 0.80 0.60 +0.20 - n/a -
alert investigation — narrow screen context vs wide alert window Input Tokens 1 12208.00 77005.00 -64797.00 - n/a -
alert investigation — narrow screen context vs wide alert window Output Tokens 1 2348.00 1745.00 +603.00 - n/a -
alert investigation — narrow screen context vs wide alert window Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
alert investigation — narrow screen context vs wide alert window Tool Calls 1 21.00 7.00 +14.00 - n/a -
alert investigation — narrow screen context vs wide alert window trajectory 1 0.00 0.00 0.00 - n/a -
alert time window — discovered via get_alerts Criteria 1 0.00 0.80 -0.80 - n/a -
alert time window — discovered via get_alerts Input Tokens 1 12095.00 124855.00 -112760.00 - n/a -
alert time window — discovered via get_alerts Output Tokens 1 192.00 1381.00 -1189.00 - n/a -
alert time window — discovered via get_alerts Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
alert time window — discovered via get_alerts Tool Calls 1 2.00 4.00 -2.00 - n/a -
alert time window — discovered via get_alerts trajectory 1 0.00 0.00 0.00 - n/a -
cart-failure investigation Criteria 1 0.17 0.17 0.00 - n/a -
cart-failure investigation Input Tokens 1 12101.00 12088.00 +13.00 - n/a -
cart-failure investigation Output Tokens 1 260.00 219.00 +41.00 - n/a -
cart-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
cart-failure investigation Tool Calls 1 2.00 1.00 +1.00 - n/a -
cart-failure investigation trajectory 1 0.00 0.00 0.00 - n/a -
email-memory-leak investigation Criteria 1 0.50 0.67 -0.17 - n/a -
email-memory-leak investigation Input Tokens 1 14613.00 12111.00 +2502.00 - n/a -
email-memory-leak investigation Output Tokens 1 257.00 173.00 +84.00 - n/a -
email-memory-leak investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
email-memory-leak investigation Tool Calls 1 1.00 2.00 -1.00 - n/a -
email-memory-leak investigation trajectory 1 0.00 0.00 0.00 - n/a -
kafka-queue-problems investigation Criteria 1 0.00 0.00 0.00 - n/a -
kafka-queue-problems investigation Input Tokens 1 14596.00 14596.00 0.00 - n/a -
kafka-queue-problems investigation Output Tokens 1 288.00 2451.00 -2163.00 - n/a -
kafka-queue-problems investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
kafka-queue-problems investigation Tool Calls 1 1.00 3.00 -2.00 - n/a -
kafka-queue-problems investigation trajectory 1 0.00 0.00 0.00 - n/a -
load-generator-flood-homepage investigation Criteria 1 0.20 0.20 0.00 - n/a -
load-generator-flood-homepage investigation Input Tokens 1 12095.00 12109.00 -14.00 - n/a -
load-generator-flood-homepage investigation Output Tokens 1 109.00 575.00 -466.00 - n/a -
load-generator-flood-homepage investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
load-generator-flood-homepage investigation Tool Calls 1 1.00 9.00 -8.00 - n/a -
load-generator-flood-homepage investigation trajectory 1 0.00 0.00 0.00 - n/a -
obs alerts routing - no alerts present Cached Tokens 1 43008.00 11739.00 +31269.00 - n/a -
obs alerts routing - no alerts present Criteria 3 0.00 0.00 0.00 1.00 No -
obs alerts routing - no alerts present Input Tokens 3 36034.33 8963.00 +27071.33 0.26 No -
obs alerts routing - no alerts present Output Tokens 3 414.33 1453.00 -1038.67 0.37 No -
obs alerts routing - no alerts present Skill Invoked (investigation) 3 0.00 0.00 0.00 1.00 No -
obs alerts routing - no alerts present Tool Calls 3 3.33 0.33 +3.00 0.41 No -
obs alerts routing - no alerts present trajectory 3 0.00 0.00 0.00 1.00 No -
obs alerts routing across screen contexts Criteria 5 0.40 0.33 +0.07 0.33 No -
obs alerts routing across screen contexts Input Tokens 5 28722.60 5472.20 +23250.40 0.08 No -
obs alerts routing across screen contexts Output Tokens 5 543.80 107.40 +436.40 0.12 No -
obs alerts routing across screen contexts Skill Invoked (investigation) 5 0.00 0.00 0.00 1.00 No -
obs alerts routing across screen contexts Tool Calls 5 1.40 0.80 +0.60 0.60 No -
obs alerts routing across screen contexts trajectory 5 0.00 0.00 0.00 1.00 No -
payment-failure investigation Criteria 1 0.17 0.17 0.00 - n/a -
payment-failure investigation Input Tokens 1 12093.00 243.00 +11850.00 - n/a -
payment-failure investigation Output Tokens 1 159.00 18.00 +141.00 - n/a -
payment-failure investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
payment-failure investigation Tool Calls 1 1.00 0.00 +1.00 - n/a -
payment-failure investigation trajectory 1 0.00 0.00 0.00 - n/a -
payment-unreachable investigation Cached Tokens 1 8279.00 11699.00 -3420.00 - n/a -
payment-unreachable investigation Criteria 1 0.15 0.15 0.00 - n/a -
payment-unreachable investigation Input Tokens 1 12090.00 12091.00 -1.00 - n/a -
payment-unreachable investigation Output Tokens 1 301.00 196.00 +105.00 - n/a -
payment-unreachable investigation Skill Invoked (investigation) 1 0.00 0.00 0.00 - n/a -
payment-unreachable investigation Tool Calls 1 2.00 2.00 0.00 - n/a -
payment-unreachable investigation trajectory 1 0.00 0.00 0.00 - n/a -
service-map skill activation Criteria 4 0.00 0.00 0.00 1.00 No -
service-map skill activation Input Tokens 4 11952.25 9023.50 +2928.75 0.33 No -
service-map skill activation Output Tokens 4 342.25 626.75 -284.50 0.31 No -
service-map skill activation Skill Invoked (service-map) 4 0.00 0.00 0.00 1.00 No -
service-map skill activation Tool Calls 4 1.50 7.25 -5.75 0.21 No -
service-map skill activation trajectory 4 0.00 0.00 0.00 1.00 No -
tool-arguments: default time fallback Criteria 1 1.00 1.00 0.00 - n/a -
tool-arguments: default time fallback Input Tokens 1 38884.00 35464.00 +3420.00 - n/a -
tool-arguments: default time fallback Output Tokens 1 837.00 750.00 +87.00 - n/a -
tool-arguments: default time fallback Tool Calls 1 9.00 1.00 +8.00 - n/a -
tool-arguments: user time override Criteria 1 1.00 0.00 +1.00 - n/a -
tool-arguments: user time override Input Tokens 1 14586.00 242.00 +14344.00 - n/a -
tool-arguments: user time override Output Tokens 1 181.00 18.00 +163.00 - n/a -
tool-arguments: user time override Tool Calls 1 1.00 0.00 +1.00 - n/a -
tool-arguments: uses screen context time range Criteria 1 1.00 0.00 +1.00 - n/a -
tool-arguments: uses screen context time range Input Tokens 1 38589.00 158535.00 -119946.00 - n/a -
tool-arguments: uses screen context time range Output Tokens 1 692.00 2196.00 -1504.00 - n/a -
tool-arguments: uses screen context time range Tool Calls 1 3.00 10.00 -7.00 - n/a -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

🧪 LLM eval (PR) — Agent Builder (agent-builder) failed.

Failing models:

  • eis-google-gemini-3-0-flash
  • eis-google-gemini-3-1-pro
  • eis-openai-gpt-5-4
  • eis-openai-gpt-oss-120b

View build

Triage summary:

ERROR Error: Playwright exited with code 1

x-pack/platform/packages/shared/kbn-evals/src/cli/commands/run.ts:153:16 — eis-google-gemini-3-0-flash, eis-google-gemini-3-1-pro, eis-openai-gpt-5-4, eis-openai-gpt-oss-120b
Root cause: Evaluation specs failed across models due to assertion mismatches in security skill tests. Inspect error-context artifacts and update prompts or assertions as needed.

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::significant-events::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::significant-events::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
4 significant difference(s) detected out of 92 comparisons.

Note: 16 unpaired (ran in only one experiment), 51 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: KI feature exclusion (otel-demo) initial_feature_count 2 20.50 16.00 +4.50 0.05 Yes Improvement
sigevents: KI feature exclusion (otel-demo) Input Tokens 2 568687.50 56837.00 +511850.50 0.03 Yes Regression
sigevents: KI feature exclusion (otel-demo) reported_input_tokens 2 613106.50 75849.50 +537257.00 0.01 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) reported_input_tokens 2 766760.00 82952.00 +683808.00 0.04 Yes Regression
No significant changes (88 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) Cached Tokens 3 166706.00 165895.00 +811.00 0.97 No -
sigevents: Discovery (bank-of-anthos) confidence_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) confirmation_alignment 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_collection 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_description_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) execute_esql_grounding 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) grouping_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) Input Tokens 3 278859.00 277904.67 +954.33 0.97 No -
sigevents: Discovery (bank-of-anthos) Latency 3 19.74 20.81 -1.07 0.23 No -
sigevents: Discovery (bank-of-anthos) narrative_fields_quality 3 0.94 0.89 +0.06 0.35 No -
sigevents: Discovery (bank-of-anthos) Output Tokens 3 3091.67 3132.33 -40.67 0.79 No -
sigevents: Discovery (bank-of-anthos) scenario_criteria 3 0.39 0.39 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_exact 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) signal_evidence_consistency 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) status_correctness 3 0.33 0.33 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Tool Calls 3 7.33 7.33 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) topology_correctness 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) trajectory 3 0.75 0.83 -0.08 0.35 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Cached Tokens 2 342655.00 368288.50 -25633.50 0.29 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_routing 2 1.00 1.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_stability 2 1.00 1.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Input Tokens 2 562258.00 588455.00 -26197.00 0.28 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Latency 2 61.20 60.66 +0.54 0.64 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Output Tokens 2 3266.50 3686.00 -419.50 0.24 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Tool Calls 2 13.50 13.00 +0.50 0.39 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) trajectory 2 0.50 0.75 -0.25 0.22 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Cached Tokens 2 324985.00 346538.00 -21553.00 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_routing 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_stability 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Input Tokens 2 544472.00 566344.00 -21872.00 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Latency 2 60.34 61.77 -1.43 0.71 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Output Tokens 2 3001.50 3581.50 -580.00 0.20 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Tool Calls 2 13.00 13.50 -0.50 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) trajectory 2 0.75 0.69 +0.06 0.39 No -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Cached Tokens 1 282255.00 325051.00 -42796.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_routing 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_stability 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Input Tokens 1 501333.00 544612.00 -43279.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Latency 1 54.62 63.41 -8.79 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Output Tokens 1 2609.00 3010.00 -401.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Tool Calls 1 12.00 13.00 -1.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) trajectory 1 0.38 0.38 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Cached Tokens 1 165431.00 165528.00 -97.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confidence_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confirmation_alignment 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_collection 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_description_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Input Tokens 1 310527.00 310710.00 -183.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Latency 1 21.37 24.14 -2.77 - n/a -
sigevents: Discovery memory (bank-of-anthos) narrative_fields_quality 1 0.83 0.83 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Output Tokens 1 1957.00 2011.00 -54.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) scenario_criteria 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_exact 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) signal_evidence_consistency 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) status_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Tool Calls 1 8.00 8.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) topology_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) trajectory 1 0.75 0.75 0.00 - n/a -
sigevents: KI feature deduplication context contracts dedup_context_contract 2 1.00 0.50 +0.50 0.39 No -
sigevents: KI feature exclusion (bank-of-anthos) Chat Calls 1 31.00 3.00 +28.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_retained_count 1 4.00 7.00 -3.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_returned_count 1 4.00 7.00 -3.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) initial_feature_count 1 9.00 13.00 -4.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) Input Tokens 1 692167.00 58992.00 +633175.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) Latency 1 342.55 183.15 +159.40 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) llm_exclude_compliance 1 1.00 1.00 0.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) Output Tokens 1 16982.00 13287.00 +3695.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) reported_input_tokens 1 717900.00 78777.00 +639123.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) reported_output_tokens 1 17174.00 18062.00 -888.00 - n/a -
sigevents: KI feature exclusion (otel-demo) Chat Calls 2 24.50 3.00 +21.50 0.05 No -
sigevents: KI feature exclusion (otel-demo) follow_up_retained_count 2 14.50 14.50 0.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) follow_up_returned_count 2 14.50 14.50 0.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) Latency 2 481.84 213.89 +267.95 0.05 No -
sigevents: KI feature exclusion (otel-demo) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) Output Tokens 2 29373.00 16573.00 +12800.00 0.09 No -
sigevents: KI feature exclusion (otel-demo) reported_output_tokens 2 28587.50 22236.00 +6351.50 0.12 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Chat Calls 2 31.00 3.50 +27.50 0.06 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_retained_count 2 5.00 6.00 -1.00 1.00 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_returned_count 2 5.00 6.00 -1.00 1.00 No -
sigevents: KI feature exclusion (quarkus-super-heroes) initial_feature_count 2 9.00 10.00 -1.00 0.39 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Input Tokens 2 715197.50 72609.50 +642588.00 0.05 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Latency 2 301.15 164.00 +137.15 0.15 No -
sigevents: KI feature exclusion (quarkus-super-heroes) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Output Tokens 2 14895.00 11794.50 +3100.50 0.35 No -
sigevents: KI feature exclusion (quarkus-super-heroes) reported_output_tokens 2 15231.50 13291.50 +1940.00 0.21 No -
sigevents: KI query generation: empty datastream expected_generation_outcome 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::significant-events::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::significant-events::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
7 significant difference(s) detected out of 88 comparisons.

Note: 74 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Input Tokens 2 564561.00 521947.50 +42613.50 0.01 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) Chat Calls 2 19.50 3.00 +16.50 0.01 Yes -
sigevents: KI feature exclusion (bank-of-anthos) Input Tokens 2 421360.50 58861.00 +362499.50 0.02 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) reported_input_tokens 2 469467.50 78568.50 +390899.00 0.01 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) Chat Calls 2 17.00 3.00 +14.00 0.03 Yes -
sigevents: KI feature exclusion (quarkus-super-heroes) Input Tokens 2 376785.50 62196.00 +314589.50 0.03 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) reported_input_tokens 2 424846.50 83014.00 +341832.50 0.03 Yes Regression
No significant changes (81 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) confidence_calibration 3 0.89 1.00 -0.11 0.35 No -
sigevents: Discovery (bank-of-anthos) confirmation_alignment 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_collection 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_description_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) execute_esql_grounding 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) grouping_correctness 1 1.00 0.00 +1.00 - n/a -
sigevents: Discovery (bank-of-anthos) Input Tokens 3 306311.00 307402.67 -1091.67 0.27 No -
sigevents: Discovery (bank-of-anthos) Latency 3 15.98 15.94 +0.04 0.94 No -
sigevents: Discovery (bank-of-anthos) narrative_fields_quality 3 0.94 0.94 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Output Tokens 3 2984.67 3519.00 -534.33 0.16 No -
sigevents: Discovery (bank-of-anthos) scenario_criteria 3 0.39 0.38 +0.02 0.35 No -
sigevents: Discovery (bank-of-anthos) severity_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_exact 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) signal_evidence_consistency 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) status_correctness 3 0.33 0.33 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Tool Calls 3 8.00 8.33 -0.33 0.35 No -
sigevents: Discovery (bank-of-anthos) topology_correctness 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) trajectory 3 0.83 0.83 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_routing 2 1.00 1.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_stability 2 1.00 1.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Input Tokens 2 521867.00 521851.00 +16.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Latency 2 39.61 43.21 -3.60 0.43 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Output Tokens 2 3636.00 3449.00 +187.00 0.83 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Tool Calls 2 14.00 13.00 +1.00 0.39 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) trajectory 2 0.75 0.75 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_routing 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_stability 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Latency 2 41.41 45.04 -3.63 0.44 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Output Tokens 2 3608.00 3272.00 +336.00 0.17 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Tool Calls 2 14.50 13.50 +1.00 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) trajectory 2 0.75 0.69 +0.06 0.39 No -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_routing 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_stability 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Input Tokens 1 543257.00 586520.00 -43263.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Latency 1 44.67 44.87 -0.20 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Output Tokens 1 3346.00 3961.00 -615.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Tool Calls 1 14.00 15.00 -1.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) trajectory 1 0.38 0.38 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confidence_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confirmation_alignment 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confirmed_evidences 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_collection 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_description_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Input Tokens 1 355463.00 310649.00 +44814.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Latency 1 12.41 13.30 -0.89 - n/a -
sigevents: Discovery memory (bank-of-anthos) narrative_fields_quality 1 0.67 0.67 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Output Tokens 1 2548.00 2758.00 -210.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) scenario_criteria 1 0.50 0.75 -0.25 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_calibration 1 0.86 1.00 -0.14 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_exact 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) signal_evidence_consistency 1 0.50 1.00 -0.50 - n/a -
sigevents: Discovery memory (bank-of-anthos) status_correctness 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Tool Calls 1 10.00 9.00 +1.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) topology_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) trajectory 1 0.75 0.75 0.00 - n/a -
sigevents: KI feature deduplication context contracts dedup_context_contract 2 0.00 0.00 0.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_retained_count 2 6.50 8.50 -2.00 0.22 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_returned_count 2 6.50 8.50 -2.00 0.22 No -
sigevents: KI feature exclusion (bank-of-anthos) initial_feature_count 2 8.00 13.00 -5.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) Latency 2 180.23 126.44 +53.79 0.05 No -
sigevents: KI feature exclusion (bank-of-anthos) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) Output Tokens 2 14206.50 14658.50 -452.00 0.62 No -
sigevents: KI feature exclusion (bank-of-anthos) reported_output_tokens 2 15855.00 19104.00 -3249.00 0.26 No -
sigevents: KI feature exclusion (otel-demo) Chat Calls 2 22.50 3.00 +19.50 0.06 No -
sigevents: KI feature exclusion (otel-demo) follow_up_retained_count 2 14.50 12.50 +2.00 0.22 No -
sigevents: KI feature exclusion (otel-demo) follow_up_returned_count 2 14.50 12.50 +2.00 0.22 No -
sigevents: KI feature exclusion (otel-demo) initial_feature_count 2 15.50 18.00 -2.50 0.39 No -
sigevents: KI feature exclusion (otel-demo) Input Tokens 2 526275.00 56809.00 +469466.00 0.08 No -
sigevents: KI feature exclusion (otel-demo) Latency 2 321.97 143.09 +178.88 0.07 No -
sigevents: KI feature exclusion (otel-demo) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) Output Tokens 2 28575.00 16788.50 +11786.50 0.07 No -
sigevents: KI feature exclusion (otel-demo) reported_input_tokens 2 569554.00 75814.50 +493739.50 0.09 No -
sigevents: KI feature exclusion (otel-demo) reported_output_tokens 2 28888.00 22338.50 +6549.50 0.08 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_retained_count 2 4.00 5.00 -1.00 0.39 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_returned_count 2 4.00 5.00 -1.00 0.39 No -
sigevents: KI feature exclusion (quarkus-super-heroes) initial_feature_count 2 7.00 9.00 -2.00 1.00 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Latency 2 120.84 86.60 +34.23 0.22 No -
sigevents: KI feature exclusion (quarkus-super-heroes) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Output Tokens 2 9459.50 9374.00 +85.50 0.94 No -
sigevents: KI feature exclusion (quarkus-super-heroes) reported_output_tokens 2 11448.00 12253.50 -805.50 0.64 No -
sigevents: KI query generation: empty datastream expected_generation_outcome 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::significant-events::anthropic-claude-4.7-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::significant-events::anthropic-claude-4.7-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
13 significant difference(s) detected out of 92 comparisons.

Note: 6 unpaired (ran in only one experiment), 52 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Latency 2 39.52 36.38 +3.14 0.04 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) Input Tokens 2 519030.50 78983.00 +440047.50 0.00 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) reported_input_tokens 2 583462.00 105418.50 +478043.50 0.00 Yes Regression
sigevents: KI feature exclusion (otel-demo) Chat Calls 2 16.50 3.00 +13.50 0.05 Yes -
sigevents: KI feature exclusion (otel-demo) Input Tokens 2 483264.00 76331.00 +406933.00 0.05 Yes Regression
sigevents: KI feature exclusion (otel-demo) Latency 2 224.27 142.57 +81.70 0.04 Yes Regression
sigevents: KI feature exclusion (otel-demo) reported_input_tokens 2 549934.50 101871.50 +448063.00 0.04 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) Chat Calls 2 17.00 3.00 +14.00 0.03 Yes -
sigevents: KI feature exclusion (quarkus-super-heroes) Input Tokens 2 512270.50 83295.00 +428975.50 0.03 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) Latency 2 166.03 86.75 +79.28 0.02 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) Output Tokens 2 15607.00 10526.00 +5081.00 0.03 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) reported_input_tokens 2 579192.00 111162.00 +468030.00 0.03 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) reported_output_tokens 2 19579.50 13728.00 +5851.50 0.05 Yes Regression
No significant changes (79 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) Cached Tokens 3 208650.33 249999.67 -41349.33 0.12 No -
sigevents: Discovery (bank-of-anthos) confidence_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) confirmation_alignment 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_collection 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_description_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) execute_esql_grounding 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) grouping_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) Input Tokens 3 362287.67 404257.00 -41969.33 0.12 No -
sigevents: Discovery (bank-of-anthos) Latency 3 14.20 16.43 -2.23 0.26 No -
sigevents: Discovery (bank-of-anthos) narrative_fields_quality 3 0.89 0.89 +0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Output Tokens 3 3131.00 3914.33 -783.33 0.13 No -
sigevents: Discovery (bank-of-anthos) scenario_criteria 3 0.38 0.38 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_exact 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) signal_evidence_consistency 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) status_correctness 3 0.33 0.33 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Tool Calls 3 7.00 7.67 -0.67 0.13 No -
sigevents: Discovery (bank-of-anthos) topology_correctness 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) trajectory 3 0.83 0.83 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Cached Tokens 2 509094.50 479659.00 +29435.50 0.39 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_routing 2 1.00 1.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_stability 2 1.00 1.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Input Tokens 2 812293.00 782709.00 +29584.00 0.39 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Output Tokens 2 4275.00 4303.50 -28.50 0.93 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Tool Calls 2 14.00 13.50 +0.50 0.39 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) trajectory 2 0.75 0.69 +0.06 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Cached Tokens 2 479402.00 420628.00 +58774.00 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_routing 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_stability 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Input Tokens 2 782286.50 723167.50 +59119.00 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Latency 2 37.50 33.84 +3.66 0.17 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Output Tokens 2 4031.50 3581.50 +450.00 0.48 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Tool Calls 2 13.00 12.50 +0.50 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) trajectory 2 0.69 0.63 +0.06 0.39 No -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Cached Tokens 1 509052.00 390970.00 +118082.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_routing 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_stability 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Input Tokens 1 812153.00 693307.00 +118846.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Latency 1 38.87 41.00 -2.14 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Output Tokens 1 3982.00 3172.00 +810.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Tool Calls 1 13.00 12.00 +1.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) trajectory 1 0.38 0.38 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Cached Tokens 1 228892.00 168595.00 +60297.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confidence_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confirmation_alignment 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_collection 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_description_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Input Tokens 1 429374.00 368490.00 +60884.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Latency 1 10.17 4.75 +5.41 - n/a -
sigevents: Discovery memory (bank-of-anthos) narrative_fields_quality 1 0.83 0.83 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Output Tokens 1 2458.00 1460.00 +998.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) scenario_criteria 1 0.25 0.00 +0.25 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_exact 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) signal_evidence_consistency 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) status_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Tool Calls 1 8.00 7.00 +1.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) topology_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) trajectory 1 0.75 0.75 0.00 - n/a -
sigevents: KI feature deduplication context contracts dedup_context_contract 2 1.00 0.50 +0.50 0.39 No -
sigevents: KI feature exclusion (bank-of-anthos) Chat Calls 2 18.00 3.00 +15.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_retained_count 2 10.00 11.00 -1.00 0.61 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_returned_count 2 10.00 11.00 -1.00 0.61 No -
sigevents: KI feature exclusion (bank-of-anthos) initial_feature_count 2 14.00 15.50 -1.50 0.15 No -
sigevents: KI feature exclusion (bank-of-anthos) Latency 2 190.81 130.60 +60.21 0.09 No -
sigevents: KI feature exclusion (bank-of-anthos) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) Output Tokens 2 18355.50 16167.00 +2188.50 0.29 No -
sigevents: KI feature exclusion (bank-of-anthos) reported_output_tokens 2 22668.00 20850.50 +1817.50 0.44 No -
sigevents: KI feature exclusion (otel-demo) follow_up_retained_count 2 16.00 15.00 +1.00 0.39 No -
sigevents: KI feature exclusion (otel-demo) follow_up_returned_count 2 16.00 15.00 +1.00 0.39 No -
sigevents: KI feature exclusion (otel-demo) initial_feature_count 2 17.50 19.50 -2.00 0.22 No -
sigevents: KI feature exclusion (otel-demo) llm_exclude_compliance 2 0.96 1.00 -0.04 0.39 No -
sigevents: KI feature exclusion (otel-demo) Output Tokens 2 22937.50 17534.00 +5403.50 0.07 No -
sigevents: KI feature exclusion (otel-demo) reported_output_tokens 2 29137.00 23262.00 +5875.00 0.06 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_retained_count 2 8.50 7.00 +1.50 0.15 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_returned_count 2 8.50 7.00 +1.50 0.15 No -
sigevents: KI feature exclusion (quarkus-super-heroes) initial_feature_count 2 11.50 10.00 +1.50 0.55 No -
sigevents: KI feature exclusion (quarkus-super-heroes) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI query generation: empty datastream expected_generation_outcome 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::significant-events::anthropic-claude-4.8-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::significant-events::anthropic-claude-4.8-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
8 significant difference(s) detected out of 92 comparisons.

Note: 6 unpaired (ran in only one experiment), 52 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: KI feature exclusion (bank-of-anthos) Chat Calls 2 16.50 3.00 +13.50 0.05 Yes -
sigevents: KI feature exclusion (bank-of-anthos) Latency 2 198.80 130.04 +68.77 0.02 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) reported_input_tokens 2 527466.00 103655.50 +423810.50 0.05 Yes Regression
sigevents: KI feature exclusion (otel-demo) Input Tokens 2 421040.50 75143.00 +345897.50 0.00 Yes Regression
sigevents: KI feature exclusion (otel-demo) reported_input_tokens 2 481806.50 100289.50 +381517.00 0.00 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) Chat Calls 2 16.00 3.50 +12.50 0.02 Yes -
sigevents: KI feature exclusion (quarkus-super-heroes) Input Tokens 2 468635.50 96081.00 +372554.50 0.02 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) reported_input_tokens 2 516144.50 123566.50 +392578.00 0.03 Yes Regression
No significant changes (84 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) Cached Tokens 3 268172.67 248420.00 +19752.67 0.70 No -
sigevents: Discovery (bank-of-anthos) confidence_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) confirmation_alignment 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_collection 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_description_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) execute_esql_grounding 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) grouping_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) Input Tokens 3 421538.00 401606.67 +19931.33 0.70 No -
sigevents: Discovery (bank-of-anthos) Latency 3 22.50 22.45 +0.04 0.97 No -
sigevents: Discovery (bank-of-anthos) narrative_fields_quality 3 0.94 0.94 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Output Tokens 3 4425.67 4309.33 +116.33 0.82 No -
sigevents: Discovery (bank-of-anthos) scenario_criteria 3 0.39 0.38 +0.02 0.35 No -
sigevents: Discovery (bank-of-anthos) severity_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_exact 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) signal_evidence_consistency 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) status_correctness 3 0.33 0.33 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Tool Calls 3 7.67 7.67 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) topology_correctness 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) trajectory 3 0.75 0.75 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Cached Tokens 2 475801.50 417932.50 +57869.00 0.40 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_routing 2 1.00 1.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_stability 2 1.00 1.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Input Tokens 2 776449.50 718502.00 +57947.50 0.40 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Latency 2 45.70 51.02 -5.32 0.35 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Output Tokens 2 4661.00 4117.00 +544.00 0.50 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Tool Calls 2 13.50 12.50 +1.00 0.39 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) trajectory 2 0.69 0.56 +0.13 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Cached Tokens 2 505674.50 447210.00 +58464.50 0.40 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_routing 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_stability 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Input Tokens 2 806761.50 748301.00 +58460.50 0.40 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Latency 2 50.12 49.03 +1.09 0.22 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Output Tokens 2 4887.00 4774.50 +112.50 0.64 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Tool Calls 2 14.00 13.00 +1.00 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) trajectory 2 0.69 0.50 +0.19 0.15 No -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Cached Tokens 1 446362.00 507617.00 -61255.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_routing 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_stability 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Input Tokens 1 746685.00 808889.00 -62204.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Latency 1 47.22 53.44 -6.22 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Output Tokens 1 4444.00 4822.00 -378.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Tool Calls 1 13.00 14.00 -1.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) trajectory 1 0.38 0.38 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Cached Tokens 1 288979.00 289335.00 -356.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confidence_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confirmation_alignment 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_collection 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_description_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Input Tokens 1 489154.00 489505.00 -351.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Latency 1 12.70 16.60 -3.90 - n/a -
sigevents: Discovery memory (bank-of-anthos) narrative_fields_quality 1 0.83 0.83 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Output Tokens 1 3435.00 3608.00 -173.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) scenario_criteria 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_exact 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) signal_evidence_consistency 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) status_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Tool Calls 1 9.00 9.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) topology_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) trajectory 1 0.75 0.75 0.00 - n/a -
sigevents: KI feature deduplication context contracts dedup_context_contract 2 0.50 0.50 0.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_retained_count 2 9.00 11.00 -2.00 0.22 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_returned_count 2 9.00 11.00 -2.00 0.22 No -
sigevents: KI feature exclusion (bank-of-anthos) initial_feature_count 2 11.00 14.50 -3.50 0.30 No -
sigevents: KI feature exclusion (bank-of-anthos) Input Tokens 2 466312.50 77670.00 +388642.50 0.05 No -
sigevents: KI feature exclusion (bank-of-anthos) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) Output Tokens 2 15764.00 13292.50 +2471.50 0.21 No -
sigevents: KI feature exclusion (bank-of-anthos) reported_output_tokens 2 20198.00 17287.50 +2910.50 0.19 No -
sigevents: KI feature exclusion (otel-demo) Chat Calls 2 15.00 3.00 +12.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) follow_up_retained_count 2 10.50 12.50 -2.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) follow_up_returned_count 2 10.50 12.50 -2.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) initial_feature_count 2 16.00 16.00 0.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) Latency 2 194.29 141.72 +52.57 0.15 No -
sigevents: KI feature exclusion (otel-demo) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) Output Tokens 2 16040.00 14376.50 +1663.50 0.28 No -
sigevents: KI feature exclusion (otel-demo) reported_output_tokens 2 21043.50 18622.00 +2421.50 0.25 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_retained_count 2 5.50 6.00 -0.50 0.72 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_returned_count 2 5.50 6.00 -0.50 0.72 No -
sigevents: KI feature exclusion (quarkus-super-heroes) initial_feature_count 2 8.00 9.00 -1.00 0.39 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Latency 2 166.06 84.77 +81.29 0.09 No -
sigevents: KI feature exclusion (quarkus-super-heroes) llm_exclude_compliance 2 1.00 0.83 +0.17 0.39 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Output Tokens 2 12800.50 8269.50 +4531.00 0.14 No -
sigevents: KI feature exclusion (quarkus-super-heroes) reported_output_tokens 2 14539.50 10973.00 +3566.50 0.34 No -
sigevents: KI query generation: empty datastream expected_generation_outcome 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::significant-events::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::significant-events::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
12 significant difference(s) detected out of 95 comparisons.

Note: 3 unpaired (ran in only one experiment), 49 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) Latency 3 7.16 2.47 +4.69 0.02 Yes Regression
sigevents: Discovery (bank-of-anthos) Output Tokens 3 2171.00 743.33 +1427.67 0.00 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) Chat Calls 2 21.50 2.50 +19.00 0.05 Yes -
sigevents: KI feature exclusion (bank-of-anthos) Input Tokens 2 400606.50 49170.00 +351436.50 0.02 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) Latency 2 71.52 22.34 +49.18 0.03 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) reported_input_tokens 2 431495.50 78824.50 +352671.00 0.01 Yes Regression
sigevents: KI feature exclusion (otel-demo) Cached Tokens 2 162342.00 24053.50 +138288.50 0.01 Yes -
sigevents: KI feature exclusion (otel-demo) Input Tokens 2 512689.50 57345.00 +455344.50 0.05 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) Input Tokens 2 415521.50 52599.50 +362922.00 0.04 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) Latency 2 69.02 19.76 +49.26 0.02 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) Output Tokens 2 5078.00 2605.50 +2472.50 0.02 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) reported_input_tokens 2 447785.50 84314.50 +363471.00 0.01 Yes Regression
No significant changes (83 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) Cached Tokens 2 145146.00 92925.00 +52221.00 0.39 No -
sigevents: Discovery (bank-of-anthos) confidence_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) confirmation_alignment 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_collection 3 1.00 0.67 +0.33 0.35 No -
sigevents: Discovery (bank-of-anthos) evidence_description_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) execute_esql_grounding 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) grouping_correctness 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) Input Tokens 3 247604.67 153057.67 +94547.00 0.06 No -
sigevents: Discovery (bank-of-anthos) narrative_fields_quality 3 0.89 0.78 +0.11 0.13 No -
sigevents: Discovery (bank-of-anthos) scenario_criteria 3 0.36 0.39 -0.03 0.35 No -
sigevents: Discovery (bank-of-anthos) severity_calibration 3 1.00 0.76 +0.24 0.35 No -
sigevents: Discovery (bank-of-anthos) severity_exact 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) signal_evidence_consistency 3 1.00 0.67 +0.33 0.13 No -
sigevents: Discovery (bank-of-anthos) status_correctness 3 0.33 0.67 -0.33 0.35 No -
sigevents: Discovery (bank-of-anthos) Tool Calls 3 7.00 5.00 +2.00 0.05 No -
sigevents: Discovery (bank-of-anthos) topology_correctness 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) trajectory 3 0.83 0.58 +0.25 0.35 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Cached Tokens 2 146719.00 210624.00 -63905.00 0.57 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_routing 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_stability 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Input Tokens 2 295465.50 420249.00 -124783.50 0.39 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Latency 2 13.34 19.19 -5.85 0.59 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Output Tokens 2 1253.00 1684.00 -431.00 0.42 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Tool Calls 2 9.00 12.00 -3.00 0.48 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) trajectory 2 0.56 0.69 -0.13 0.61 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Cached Tokens 2 245289.00 193215.00 +52074.00 0.37 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_routing 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_stability 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Input Tokens 2 453751.00 385133.00 +68618.00 0.22 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Latency 2 22.40 20.05 +2.35 0.26 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Output Tokens 2 1929.00 1605.50 +323.50 0.54 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Tool Calls 2 13.50 11.50 +2.00 0.22 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) trajectory 2 0.75 0.69 +0.06 0.39 No -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Cached Tokens 1 239580.00 235846.00 +3734.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_routing 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_stability 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Input Tokens 1 436779.00 403365.00 +33414.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Latency 1 24.39 22.17 +2.22 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Output Tokens 1 1765.00 1884.00 -119.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Tool Calls 1 13.00 12.00 +1.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) trajectory 1 0.38 0.38 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confidence_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confirmation_alignment 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_collection 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_description_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) grouping_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Input Tokens 1 382.00 80532.00 -80150.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Latency 1 4.72 1.67 +3.05 - n/a -
sigevents: Discovery memory (bank-of-anthos) narrative_fields_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Output Tokens 1 18.00 166.00 -148.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) scenario_criteria 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_exact 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) signal_evidence_consistency 1 1.00 0.50 +0.50 - n/a -
sigevents: Discovery memory (bank-of-anthos) status_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Tool Calls 1 0.00 3.00 -3.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) topology_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) trajectory 1 0.00 0.00 0.00 - n/a -
sigevents: KI feature deduplication context contracts dedup_context_contract 2 0.50 0.00 +0.50 0.39 No -
sigevents: KI feature exclusion (bank-of-anthos) Cached Tokens 1 134459.00 32089.00 +102370.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_retained_count 2 4.50 5.00 -0.50 0.39 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_returned_count 2 4.50 5.00 -0.50 0.39 No -
sigevents: KI feature exclusion (bank-of-anthos) initial_feature_count 2 6.00 7.00 -1.00 0.39 No -
sigevents: KI feature exclusion (bank-of-anthos) llm_exclude_compliance 2 0.96 1.00 -0.04 0.39 No -
sigevents: KI feature exclusion (bank-of-anthos) Output Tokens 2 6312.00 3123.50 +3188.50 0.09 No -
sigevents: KI feature exclusion (bank-of-anthos) reported_output_tokens 2 7087.00 4828.50 +2258.50 0.12 No -
sigevents: KI feature exclusion (otel-demo) Chat Calls 2 23.50 3.00 +20.50 0.05 No -
sigevents: KI feature exclusion (otel-demo) follow_up_retained_count 2 6.50 8.50 -2.00 0.22 No -
sigevents: KI feature exclusion (otel-demo) follow_up_returned_count 2 6.50 8.50 -2.00 0.22 No -
sigevents: KI feature exclusion (otel-demo) initial_feature_count 2 8.50 11.00 -2.50 0.09 No -
sigevents: KI feature exclusion (otel-demo) Latency 2 98.31 37.36 +60.94 0.16 No -
sigevents: KI feature exclusion (otel-demo) llm_exclude_compliance 2 1.00 0.96 +0.04 0.39 No -
sigevents: KI feature exclusion (otel-demo) Output Tokens 2 10950.50 5723.00 +5227.50 0.31 No -
sigevents: KI feature exclusion (otel-demo) reported_input_tokens 2 522942.00 76506.00 +446436.00 0.09 No -
sigevents: KI feature exclusion (otel-demo) reported_output_tokens 2 9941.50 7487.00 +2454.50 0.20 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Cached Tokens 1 82641.00 40165.00 +42476.00 - n/a -
sigevents: KI feature exclusion (quarkus-super-heroes) Chat Calls 2 21.00 2.50 +18.50 0.06 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_retained_count 2 3.50 4.00 -0.50 0.72 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_returned_count 2 3.50 4.00 -0.50 0.72 No -
sigevents: KI feature exclusion (quarkus-super-heroes) initial_feature_count 2 4.50 5.50 -1.00 1.00 No -
sigevents: KI feature exclusion (quarkus-super-heroes) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI feature exclusion (quarkus-super-heroes) reported_output_tokens 2 5942.50 4222.50 +1720.00 0.29 No -
sigevents: KI query generation: empty datastream expected_generation_outcome 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::significant-events::google-gemini-3.1-flash-lite | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::significant-events::google-gemini-3.1-flash-lite
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
3 significant difference(s) detected out of 93 comparisons.

Note: 8 unpaired (ran in only one experiment), 57 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) Cached Tokens 2 167492.00 242575.00 -75083.00 0.04 Yes -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Input Tokens 2 270302.00 556342.50 -286040.50 0.02 Yes Improvement
sigevents: KI feature exclusion (quarkus-super-heroes) reported_input_tokens 2 778426.00 52447.00 +725979.00 0.03 Yes Regression
No significant changes (90 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) confidence_calibration 3 0.78 1.00 -0.22 0.35 No -
sigevents: Discovery (bank-of-anthos) confirmation_alignment 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_collection 3 0.67 1.00 -0.33 0.35 No -
sigevents: Discovery (bank-of-anthos) evidence_description_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) execute_esql_grounding 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) grouping_correctness 1 0.00 1.00 -1.00 - n/a -
sigevents: Discovery (bank-of-anthos) Input Tokens 3 226252.33 289422.00 -63169.67 0.14 No -
sigevents: Discovery (bank-of-anthos) Latency 3 3.50 2.87 +0.63 0.19 No -
sigevents: Discovery (bank-of-anthos) narrative_fields_quality 3 1.00 0.78 +0.22 0.21 No -
sigevents: Discovery (bank-of-anthos) Output Tokens 3 2539.00 3951.33 -1412.33 0.23 No -
sigevents: Discovery (bank-of-anthos) scenario_criteria 3 0.13 0.33 -0.20 0.46 No -
sigevents: Discovery (bank-of-anthos) severity_calibration 3 0.81 1.00 -0.19 0.35 No -
sigevents: Discovery (bank-of-anthos) severity_exact 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) signal_evidence_consistency 3 0.67 1.00 -0.33 0.13 No -
sigevents: Discovery (bank-of-anthos) status_correctness 3 0.00 0.33 -0.33 0.35 No -
sigevents: Discovery (bank-of-anthos) Tool Calls 3 4.67 5.67 -1.00 1.00 No -
sigevents: Discovery (bank-of-anthos) topology_correctness 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) trajectory 3 0.17 0.50 -0.33 0.35 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Cached Tokens 2 368008.50 128480.50 +239528.00 0.10 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_routing 2 1.00 1.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_stability 2 1.00 1.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Input Tokens 2 539680.00 235208.50 +304471.50 0.13 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Latency 2 23.72 16.89 +6.83 0.61 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Output Tokens 2 3201.50 1236.50 +1965.00 0.16 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Tool Calls 2 8.50 5.00 +3.50 0.06 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) trajectory 2 0.44 0.38 +0.06 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Cached Tokens 2 122734.00 352573.00 -229839.00 0.06 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_routing 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_stability 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Latency 2 15.56 23.59 -8.03 0.51 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Output Tokens 2 990.00 2881.00 -1891.00 0.06 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Tool Calls 2 7.00 10.50 -3.50 0.06 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) trajectory 2 0.44 0.31 +0.13 0.39 No -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Cached Tokens 1 270281.00 59917.00 +210364.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Input Tokens 1 497910.00 263461.00 +234449.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Latency 1 14.68 15.77 -1.08 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Output Tokens 1 1627.00 1000.00 +627.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Tool Calls 1 13.00 8.00 +5.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) trajectory 1 0.25 0.00 +0.25 - n/a -
sigevents: Discovery memory (bank-of-anthos) confidence_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confirmation_alignment 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_collection 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_description_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) grouping_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Input Tokens 1 80508.00 143437.00 -62929.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Latency 1 2.04 1.69 +0.35 - n/a -
sigevents: Discovery memory (bank-of-anthos) narrative_fields_quality 1 0.50 1.00 -0.50 - n/a -
sigevents: Discovery memory (bank-of-anthos) Output Tokens 1 177.00 272.00 -95.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) scenario_criteria 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_exact 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) signal_evidence_consistency 1 0.00 1.00 -1.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) status_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Tool Calls 1 3.00 5.00 -2.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) topology_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) trajectory 1 0.50 0.00 +0.50 - n/a -
sigevents: KI feature deduplication context contracts dedup_context_contract 2 0.50 0.50 0.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) Cached Tokens 1 38676.00 18049.00 +20627.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) Chat Calls 2 19.50 2.00 +17.50 0.32 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_retained_count 1 2.00 5.00 -3.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_returned_count 1 2.00 5.00 -3.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) initial_feature_count 2 2.00 6.00 -4.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) Input Tokens 2 333833.50 39332.00 +294501.50 0.32 No -
sigevents: KI feature exclusion (bank-of-anthos) Latency 2 57.39 8.33 +49.05 0.33 No -
sigevents: KI feature exclusion (bank-of-anthos) llm_exclude_compliance 1 1.00 1.00 0.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) Output Tokens 2 1227.00 1738.00 -511.00 0.54 No -
sigevents: KI feature exclusion (bank-of-anthos) reported_input_tokens 2 293534.50 78788.00 +214746.50 0.24 No -
sigevents: KI feature exclusion (bank-of-anthos) reported_output_tokens 2 1572.50 2904.50 -1332.00 0.09 No -
sigevents: KI feature exclusion (otel-demo) Cached Tokens 2 91650.00 18038.00 +73612.00 0.33 No -
sigevents: KI feature exclusion (otel-demo) Chat Calls 2 21.00 2.00 +19.00 0.21 No -
sigevents: KI feature exclusion (otel-demo) follow_up_retained_count 2 3.00 6.00 -3.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) follow_up_returned_count 2 3.00 8.50 -5.50 0.12 No -
sigevents: KI feature exclusion (otel-demo) initial_feature_count 2 6.50 10.00 -3.50 0.06 No -
sigevents: KI feature exclusion (otel-demo) Input Tokens 2 360867.50 38135.50 +322732.00 0.21 No -
sigevents: KI feature exclusion (otel-demo) Latency 2 68.34 12.54 +55.80 0.34 No -
sigevents: KI feature exclusion (otel-demo) llm_exclude_compliance 2 1.00 0.00 +1.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) Output Tokens 2 3464.50 3300.50 +164.00 0.95 No -
sigevents: KI feature exclusion (otel-demo) reported_input_tokens 2 371370.50 76429.50 +294941.00 0.10 No -
sigevents: KI feature exclusion (otel-demo) reported_output_tokens 2 4282.00 5958.50 -1676.50 0.38 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Cached Tokens 1 172119.00 20082.00 +152037.00 - n/a -
sigevents: KI feature exclusion (quarkus-super-heroes) Chat Calls 2 33.00 1.50 +31.50 0.09 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_retained_count 1 2.00 2.00 0.00 - n/a -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_returned_count 1 2.00 3.00 -1.00 - n/a -
sigevents: KI feature exclusion (quarkus-super-heroes) initial_feature_count 2 3.00 3.00 0.00 1.00 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Input Tokens 2 890829.50 31443.00 +859386.50 0.09 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Latency 2 103.78 5.01 +98.77 0.12 No -
sigevents: KI feature exclusion (quarkus-super-heroes) llm_exclude_compliance 1 1.00 0.33 +0.67 - n/a -
sigevents: KI feature exclusion (quarkus-super-heroes) Output Tokens 2 2709.00 844.50 +1864.50 0.12 No -
sigevents: KI feature exclusion (quarkus-super-heroes) reported_output_tokens 2 3055.50 1481.00 +1574.50 0.34 No -
sigevents: KI query generation: empty datastream expected_generation_outcome 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::significant-events::google-gemini-3.5-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::significant-events::google-gemini-3.5-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
5 significant difference(s) detected out of 95 comparisons.

Note: 46 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: KI feature exclusion (otel-demo) Cached Tokens 2 343678.50 29061.50 +314617.00 0.02 Yes -
sigevents: KI feature exclusion (otel-demo) Chat Calls 2 37.50 3.00 +34.50 0.02 Yes -
sigevents: KI feature exclusion (otel-demo) initial_feature_count 2 11.50 16.00 -4.50 0.05 Yes Regression
sigevents: KI feature exclusion (otel-demo) Input Tokens 2 821605.00 57391.00 +764214.00 0.04 Yes Regression
sigevents: KI feature exclusion (otel-demo) reported_input_tokens 2 904041.50 76575.00 +827466.50 0.03 Yes Regression
No significant changes (90 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) Cached Tokens 3 144519.00 162885.67 -18366.67 0.81 No -
sigevents: Discovery (bank-of-anthos) confidence_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) confirmation_alignment 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_collection 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_description_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) execute_esql_grounding 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) grouping_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) Input Tokens 3 288356.33 350477.33 -62121.00 0.58 No -
sigevents: Discovery (bank-of-anthos) Latency 3 9.26 17.55 -8.29 0.44 No -
sigevents: Discovery (bank-of-anthos) narrative_fields_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Output Tokens 3 5402.33 12526.33 -7124.00 0.44 No -
sigevents: Discovery (bank-of-anthos) scenario_criteria 3 0.39 0.39 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_exact 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) signal_evidence_consistency 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) status_correctness 3 0.33 0.33 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Tool Calls 3 8.67 9.33 -0.67 0.74 No -
sigevents: Discovery (bank-of-anthos) topology_correctness 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) trajectory 3 0.83 0.83 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Cached Tokens 2 283056.50 199908.50 +83148.00 0.29 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_routing 2 1.00 1.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_stability 2 1.00 1.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Input Tokens 2 528679.00 511843.00 +16836.00 0.43 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Latency 2 32.25 33.89 -1.64 0.63 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Output Tokens 2 7559.50 6896.00 +663.50 0.70 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Tool Calls 2 16.00 15.00 +1.00 0.39 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) trajectory 2 0.75 0.75 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Cached Tokens 2 519384.50 206704.50 +312680.00 0.26 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_routing 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_stability 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Input Tokens 2 808881.00 547304.50 +261576.50 0.36 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Latency 2 66.54 32.58 +33.96 0.38 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Output Tokens 2 14916.50 6670.50 +8246.00 0.45 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Tool Calls 2 16.00 15.50 +0.50 0.72 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) trajectory 2 0.38 0.75 -0.38 1.00 No -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Cached Tokens 1 703599.00 253216.00 +450383.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_routing 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_stability 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Input Tokens 1 1012540.00 548122.00 +464418.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Latency 1 56.33 32.78 +23.55 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Output Tokens 1 15909.00 7034.00 +8875.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Tool Calls 1 24.00 16.00 +8.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) trajectory 1 0.38 0.38 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Cached Tokens 1 129623.00 154745.00 -25122.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confidence_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confirmation_alignment 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_collection 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_description_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Input Tokens 1 290279.00 324695.00 -34416.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Latency 1 5.84 8.31 -2.47 - n/a -
sigevents: Discovery memory (bank-of-anthos) narrative_fields_quality 1 0.83 1.00 -0.17 - n/a -
sigevents: Discovery memory (bank-of-anthos) Output Tokens 1 3870.00 2737.00 +1133.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) scenario_criteria 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_exact 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) signal_evidence_consistency 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) status_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Tool Calls 1 9.00 10.00 -1.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) topology_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) trajectory 1 0.75 0.75 0.00 - n/a -
sigevents: KI feature deduplication context contracts dedup_context_contract 2 0.00 0.00 0.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) Cached Tokens 2 334285.00 29079.50 +305205.50 0.11 No -
sigevents: KI feature exclusion (bank-of-anthos) Chat Calls 2 31.50 3.00 +28.50 0.06 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_retained_count 2 8.50 9.50 -1.00 0.61 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_returned_count 2 8.50 9.50 -1.00 0.61 No -
sigevents: KI feature exclusion (bank-of-anthos) initial_feature_count 2 12.50 13.50 -1.00 0.39 No -
sigevents: KI feature exclusion (bank-of-anthos) Input Tokens 2 636199.50 59120.00 +577079.50 0.06 No -
sigevents: KI feature exclusion (bank-of-anthos) Latency 2 260.04 151.72 +108.32 0.13 No -
sigevents: KI feature exclusion (bank-of-anthos) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) Output Tokens 2 48228.50 35265.00 +12963.50 0.20 No -
sigevents: KI feature exclusion (bank-of-anthos) reported_input_tokens 2 702274.00 78876.00 +623398.00 0.07 No -
sigevents: KI feature exclusion (bank-of-anthos) reported_output_tokens 2 54893.50 46748.50 +8145.00 0.08 No -
sigevents: KI feature exclusion (otel-demo) follow_up_retained_count 2 11.50 14.00 -2.50 0.09 No -
sigevents: KI feature exclusion (otel-demo) follow_up_returned_count 2 11.50 14.00 -2.50 0.09 No -
sigevents: KI feature exclusion (otel-demo) Latency 2 322.42 173.04 +149.37 0.07 No -
sigevents: KI feature exclusion (otel-demo) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) Output Tokens 2 65674.50 44704.00 +20970.50 0.14 No -
sigevents: KI feature exclusion (otel-demo) reported_output_tokens 2 67392.00 59327.00 +8065.00 0.08 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Cached Tokens 2 540414.00 40166.00 +500248.00 0.11 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Chat Calls 2 39.00 4.00 +35.00 0.08 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_retained_count 2 5.50 5.50 0.00 1.00 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_returned_count 2 5.50 5.50 0.00 1.00 No -
sigevents: KI feature exclusion (quarkus-super-heroes) initial_feature_count 2 8.00 8.50 -0.50 0.72 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Input Tokens 2 836455.00 84300.00 +752155.00 0.09 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Latency 2 270.59 156.28 +114.31 0.26 No -
sigevents: KI feature exclusion (quarkus-super-heroes) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Output Tokens 2 47166.00 34296.50 +12869.50 0.41 No -
sigevents: KI feature exclusion (quarkus-super-heroes) reported_input_tokens 2 909569.50 84222.00 +825347.50 0.06 No -
sigevents: KI feature exclusion (quarkus-super-heroes) reported_output_tokens 2 47992.00 34233.50 +13758.50 0.12 No -
sigevents: KI query generation: empty datastream expected_generation_outcome 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::significant-events::openai-gpt-5.2 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::significant-events::openai-gpt-5.2
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
10 significant difference(s) detected out of 89 comparisons.

Note: 2 unpaired (ran in only one experiment), 62 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) Input Tokens 3 300294.67 300487.67 -193.00 0.01 Yes Improvement
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Latency 2 60.75 36.08 +24.67 0.04 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) Cached Tokens 2 156224.00 40192.00 +116032.00 0.01 Yes -
sigevents: KI feature exclusion (bank-of-anthos) Chat Calls 2 14.00 4.00 +10.00 0.04 Yes -
sigevents: KI feature exclusion (bank-of-anthos) Input Tokens 2 229110.00 62524.00 +166586.00 0.05 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) reported_input_tokens 2 263257.00 62627.00 +200630.00 0.00 Yes Regression
sigevents: KI feature exclusion (otel-demo) Chat Calls 2 14.00 3.50 +10.50 0.02 Yes -
sigevents: KI feature exclusion (otel-demo) Input Tokens 2 227106.50 53025.00 +174081.50 0.02 Yes Regression
sigevents: KI feature exclusion (otel-demo) reported_input_tokens 2 261243.50 60568.00 +200675.50 0.00 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) initial_feature_count 2 2.50 8.00 -5.50 0.04 Yes Regression
No significant changes (79 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) Cached Tokens 3 223872.00 224128.00 -256.00 0.13 No -
sigevents: Discovery (bank-of-anthos) confidence_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) confirmation_alignment 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_collection 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_description_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) execute_esql_grounding 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) grouping_correctness 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) Latency 3 7.21 10.35 -3.14 0.24 No -
sigevents: Discovery (bank-of-anthos) narrative_fields_quality 3 0.89 0.89 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Output Tokens 3 1562.00 1710.33 -148.33 0.25 No -
sigevents: Discovery (bank-of-anthos) scenario_criteria 3 0.35 0.35 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_exact 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) signal_evidence_consistency 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) status_correctness 3 0.33 0.33 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Tool Calls 3 7.67 7.33 +0.33 0.35 No -
sigevents: Discovery (bank-of-anthos) topology_correctness 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) trajectory 3 0.83 0.83 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Cached Tokens 2 433920.00 515456.00 -81536.00 0.44 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Input Tokens 2 518939.50 537245.00 -18305.50 0.90 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Latency 2 50.00 63.49 -13.49 0.71 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Output Tokens 2 2066.50 2193.00 -126.50 0.91 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Tool Calls 2 12.50 13.00 -0.50 0.39 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) trajectory 2 0.00 0.38 -0.38 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Cached Tokens 2 504576.00 363008.00 +141568.00 0.13 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Input Tokens 2 590337.50 447918.00 +142419.50 0.31 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Output Tokens 2 2517.50 1472.00 +1045.50 0.35 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Tool Calls 2 12.50 12.50 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) trajectory 2 0.19 0.50 -0.31 0.39 No -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Cached Tokens 1 512768.00 469632.00 +43136.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Input Tokens 1 590297.00 554827.00 +35470.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Latency 1 55.10 75.85 -20.75 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Output Tokens 1 2326.00 2375.00 -49.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Tool Calls 1 14.00 12.00 +2.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) trajectory 1 0.38 0.00 +0.38 - n/a -
sigevents: Discovery memory (bank-of-anthos) Cached Tokens 1 124544.00 188160.00 -63616.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confidence_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confirmation_alignment 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_collection 1 1.00 0.00 +1.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_description_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Input Tokens 1 218995.00 291411.00 -72416.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Latency 1 4.83 9.34 -4.51 - n/a -
sigevents: Discovery memory (bank-of-anthos) narrative_fields_quality 1 0.83 1.00 -0.17 - n/a -
sigevents: Discovery memory (bank-of-anthos) Output Tokens 1 475.00 1077.00 -602.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) scenario_criteria 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_exact 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) signal_evidence_consistency 1 1.00 0.50 +0.50 - n/a -
sigevents: Discovery memory (bank-of-anthos) status_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Tool Calls 1 7.00 7.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) topology_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) trajectory 1 0.75 0.00 +0.75 - n/a -
sigevents: KI feature deduplication context contracts dedup_context_contract 2 0.50 0.50 0.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_retained_count 2 3.50 8.00 -4.50 0.15 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_returned_count 2 5.50 9.00 -3.50 0.06 No -
sigevents: KI feature exclusion (bank-of-anthos) initial_feature_count 2 6.50 8.50 -2.00 0.39 No -
sigevents: KI feature exclusion (bank-of-anthos) Latency 2 77.89 75.18 +2.70 0.87 No -
sigevents: KI feature exclusion (bank-of-anthos) llm_exclude_compliance 2 0.67 0.83 -0.17 0.39 No -
sigevents: KI feature exclusion (bank-of-anthos) Output Tokens 2 4347.50 7467.00 -3119.50 0.17 No -
sigevents: KI feature exclusion (bank-of-anthos) reported_output_tokens 2 5460.00 8531.50 -3071.50 0.07 No -
sigevents: KI feature exclusion (otel-demo) Cached Tokens 2 96128.00 24320.00 +71808.00 0.33 No -
sigevents: KI feature exclusion (otel-demo) follow_up_retained_count 2 7.00 10.50 -3.50 0.19 No -
sigevents: KI feature exclusion (otel-demo) follow_up_returned_count 2 7.00 10.50 -3.50 0.19 No -
sigevents: KI feature exclusion (otel-demo) initial_feature_count 2 8.00 11.50 -3.50 0.19 No -
sigevents: KI feature exclusion (otel-demo) Latency 2 84.12 74.54 +9.58 0.67 No -
sigevents: KI feature exclusion (otel-demo) llm_exclude_compliance 2 0.96 0.83 +0.13 0.55 No -
sigevents: KI feature exclusion (otel-demo) Output Tokens 2 5183.50 8003.00 -2819.50 0.29 No -
sigevents: KI feature exclusion (otel-demo) reported_output_tokens 2 6674.00 9548.00 -2874.00 0.25 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Cached Tokens 2 92864.00 31744.00 +61120.00 0.39 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Chat Calls 2 8.00 3.00 +5.00 0.45 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_retained_count 1 4.00 5.00 -1.00 - n/a -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_returned_count 1 4.00 5.00 -1.00 - n/a -
sigevents: KI feature exclusion (quarkus-super-heroes) Input Tokens 2 135179.50 50084.00 +85095.50 0.45 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Latency 2 37.74 43.12 -5.38 0.83 No -
sigevents: KI feature exclusion (quarkus-super-heroes) llm_exclude_compliance 1 1.00 1.00 0.00 - n/a -
sigevents: KI feature exclusion (quarkus-super-heroes) Output Tokens 2 1130.00 3894.50 -2764.50 0.11 No -
sigevents: KI feature exclusion (quarkus-super-heroes) reported_input_tokens 2 170332.00 66827.50 +103504.50 0.39 No -
sigevents: KI feature exclusion (quarkus-super-heroes) reported_output_tokens 2 1858.00 4932.50 -3074.50 0.14 No -
sigevents: KI query generation: empty datastream expected_generation_outcome 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::significant-events::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::significant-events::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
15 significant difference(s) detected out of 95 comparisons.

Note: 50 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Latency 2 25.86 41.65 -15.79 0.04 Yes Improvement
sigevents: KI feature exclusion (bank-of-anthos) Cached Tokens 2 141184.00 19328.00 +121856.00 0.01 Yes -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_retained_count 2 2.50 8.00 -5.50 0.04 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) follow_up_returned_count 2 2.50 8.00 -5.50 0.04 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) Input Tokens 2 228799.00 46906.00 +181893.00 0.00 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) reported_input_tokens 2 263011.50 62596.00 +200415.50 0.00 Yes Regression
sigevents: KI feature exclusion (otel-demo) Chat Calls 2 14.50 3.00 +11.50 0.02 Yes -
sigevents: KI feature exclusion (otel-demo) Input Tokens 2 237086.50 45401.00 +191685.50 0.02 Yes Regression
sigevents: KI feature exclusion (otel-demo) Output Tokens 2 6358.50 8718.00 -2359.50 0.02 Yes Improvement
sigevents: KI feature exclusion (otel-demo) reported_input_tokens 2 272615.00 60579.50 +212035.50 0.02 Yes Regression
sigevents: KI feature exclusion (otel-demo) reported_output_tokens 2 8181.50 11104.00 -2922.50 0.01 Yes Improvement
sigevents: KI feature exclusion (quarkus-super-heroes) Cached Tokens 2 170624.00 21184.00 +149440.00 0.02 Yes -
sigevents: KI feature exclusion (quarkus-super-heroes) Chat Calls 2 13.50 2.50 +11.00 0.04 Yes -
sigevents: KI feature exclusion (quarkus-super-heroes) Input Tokens 2 232384.50 41714.50 +190670.00 0.04 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) reported_input_tokens 2 276951.50 66769.50 +210182.00 0.00 Yes Regression
No significant changes (80 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) Cached Tokens 3 158933.33 115498.67 +43434.67 0.14 No -
sigevents: Discovery (bank-of-anthos) confidence_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) confirmation_alignment 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_collection 3 1.00 0.67 +0.33 0.35 No -
sigevents: Discovery (bank-of-anthos) evidence_description_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) execute_esql_grounding 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) grouping_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) Input Tokens 3 192291.00 167394.33 +24896.67 0.34 No -
sigevents: Discovery (bank-of-anthos) Latency 3 9.73 4.21 +5.52 0.19 No -
sigevents: Discovery (bank-of-anthos) narrative_fields_quality 3 0.89 0.89 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Output Tokens 3 1629.67 689.67 +940.00 0.31 No -
sigevents: Discovery (bank-of-anthos) scenario_criteria 3 0.39 0.33 +0.06 0.35 No -
sigevents: Discovery (bank-of-anthos) severity_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_exact 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) signal_evidence_consistency 3 1.00 0.83 +0.17 0.35 No -
sigevents: Discovery (bank-of-anthos) status_correctness 3 0.33 0.33 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Tool Calls 3 7.00 5.33 +1.67 0.15 No -
sigevents: Discovery (bank-of-anthos) topology_correctness 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) trajectory 3 0.75 0.50 +0.25 0.17 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Cached Tokens 2 321856.00 277376.00 +44480.00 0.17 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_routing 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_stability 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Input Tokens 2 342903.00 343295.50 -392.50 0.99 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Output Tokens 2 1593.00 1451.00 +142.00 0.45 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Tool Calls 2 13.00 11.50 +1.50 0.15 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) trajectory 2 0.56 0.69 -0.13 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Cached Tokens 2 275968.00 258496.00 +17472.00 0.73 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_routing 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_stability 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Input Tokens 2 360323.50 378955.00 -18631.50 0.38 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Latency 2 30.37 26.94 +3.43 0.31 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Output Tokens 2 1723.00 1894.50 -171.50 0.42 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Tool Calls 2 12.50 13.00 -0.50 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) trajectory 2 0.50 0.69 -0.19 0.39 No -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Cached Tokens 1 294144.00 322560.00 -28416.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_routing 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_stability 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Input Tokens 1 378696.00 342981.00 +35715.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Latency 1 29.90 29.27 +0.63 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Output Tokens 1 1658.00 1439.00 +219.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Tool Calls 1 12.00 10.00 +2.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) trajectory 1 0.38 0.38 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Cached Tokens 1 152576.00 45440.00 +107136.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confidence_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confirmation_alignment 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_collection 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_description_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Input Tokens 1 183872.00 147835.00 +36037.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Latency 1 9.23 4.75 +4.49 - n/a -
sigevents: Discovery memory (bank-of-anthos) narrative_fields_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Output Tokens 1 825.00 521.00 +304.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) scenario_criteria 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_exact 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) signal_evidence_consistency 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) status_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Tool Calls 1 8.00 5.00 +3.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) topology_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) trajectory 1 0.75 0.50 +0.25 - n/a -
sigevents: KI feature deduplication context contracts dedup_context_contract 2 0.00 0.00 0.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) Chat Calls 2 14.00 3.00 +11.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) initial_feature_count 2 7.00 10.00 -3.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) Latency 2 68.50 76.37 -7.88 0.76 No -
sigevents: KI feature exclusion (bank-of-anthos) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) Output Tokens 2 3348.00 5351.00 -2003.00 0.05 No -
sigevents: KI feature exclusion (bank-of-anthos) reported_output_tokens 2 4250.00 7132.50 -2882.50 0.10 No -
sigevents: KI feature exclusion (otel-demo) Cached Tokens 2 125120.00 21952.00 +103168.00 0.24 No -
sigevents: KI feature exclusion (otel-demo) follow_up_retained_count 2 8.00 12.00 -4.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) follow_up_returned_count 2 8.00 12.00 -4.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) initial_feature_count 2 10.50 15.50 -5.00 0.09 No -
sigevents: KI feature exclusion (otel-demo) Latency 2 80.64 69.29 +11.35 0.30 No -
sigevents: KI feature exclusion (otel-demo) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_retained_count 2 2.50 5.00 -2.50 0.09 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_returned_count 2 2.50 5.00 -2.50 0.09 No -
sigevents: KI feature exclusion (quarkus-super-heroes) initial_feature_count 2 5.00 7.50 -2.50 0.09 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Latency 2 61.84 30.12 +31.72 0.22 No -
sigevents: KI feature exclusion (quarkus-super-heroes) llm_exclude_compliance 2 0.96 1.00 -0.04 0.39 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Output Tokens 2 3106.00 3119.00 -13.00 0.99 No -
sigevents: KI feature exclusion (quarkus-super-heroes) reported_output_tokens 2 3702.00 4559.00 -857.00 0.37 No -
sigevents: KI query generation: empty datastream expected_generation_outcome 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::significant-events::openai-gpt-5.4-mini | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::significant-events::openai-gpt-5.4-mini
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
7 significant difference(s) detected out of 89 comparisons.

Note: 2 unpaired (ran in only one experiment), 62 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: KI feature exclusion (bank-of-anthos) Cached Tokens 2 121600.00 29952.00 +91648.00 0.02 Yes -
sigevents: KI feature exclusion (bank-of-anthos) Input Tokens 2 231537.00 47011.50 +184525.50 0.00 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) reported_input_tokens 2 267053.50 62788.50 +204265.00 0.00 Yes Regression
sigevents: KI feature exclusion (otel-demo) Chat Calls 2 14.00 3.50 +10.50 0.02 Yes -
sigevents: KI feature exclusion (otel-demo) Input Tokens 2 230941.00 53152.50 +177788.50 0.02 Yes Regression
sigevents: KI feature exclusion (otel-demo) reported_input_tokens 2 266306.00 60902.50 +205403.50 0.00 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) reported_input_tokens 2 275671.50 66841.00 +208830.50 0.01 Yes Regression
No significant changes (82 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) Cached Tokens 3 37376.00 119466.67 -82090.67 0.27 No -
sigevents: Discovery (bank-of-anthos) confidence_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) confirmation_alignment 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_collection 3 0.33 0.67 -0.33 0.60 No -
sigevents: Discovery (bank-of-anthos) evidence_description_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) execute_esql_grounding 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) grouping_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) Input Tokens 3 112273.00 171291.33 -59018.33 0.49 No -
sigevents: Discovery (bank-of-anthos) Latency 3 4.52 3.10 +1.42 0.25 No -
sigevents: Discovery (bank-of-anthos) narrative_fields_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Output Tokens 3 555.33 1257.67 -702.33 0.42 No -
sigevents: Discovery (bank-of-anthos) scenario_criteria 3 0.00 0.05 -0.05 0.35 No -
sigevents: Discovery (bank-of-anthos) severity_calibration 3 1.00 0.95 +0.05 0.35 No -
sigevents: Discovery (bank-of-anthos) severity_exact 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) signal_evidence_consistency 3 0.83 0.50 +0.33 0.45 No -
sigevents: Discovery (bank-of-anthos) status_correctness 3 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Tool Calls 3 3.67 3.67 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) topology_correctness 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) trajectory 3 0.17 0.17 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Cached Tokens 2 203008.00 226816.00 -23808.00 0.37 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Input Tokens 2 239761.00 239970.00 -209.00 0.40 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Latency 2 17.22 20.13 -2.91 0.20 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Output Tokens 2 781.50 885.00 -103.50 0.52 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Tool Calls 2 7.00 7.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) trajectory 2 0.00 0.13 -0.13 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Cached Tokens 2 176128.00 145664.00 +30464.00 0.41 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Input Tokens 2 289047.50 222029.50 +67018.00 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Latency 2 17.13 16.42 +0.70 0.79 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Output Tokens 2 870.00 678.00 +192.00 0.49 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Tool Calls 2 8.50 7.00 +1.50 0.55 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) trajectory 2 0.19 0.00 +0.19 0.39 No -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Cached Tokens 1 193024.00 151040.00 +41984.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Input Tokens 1 271841.00 207753.00 +64088.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Latency 1 23.88 14.47 +9.41 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Output Tokens 1 973.00 509.00 +464.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Tool Calls 1 8.00 6.00 +2.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) trajectory 1 0.25 0.00 +0.25 - n/a -
sigevents: Discovery memory (bank-of-anthos) Cached Tokens 1 24576.00 152064.00 -127488.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confidence_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confirmation_alignment 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_collection 1 0.00 1.00 -1.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_description_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Input Tokens 1 111792.00 183201.00 -71409.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Latency 1 3.62 10.64 -7.02 - n/a -
sigevents: Discovery memory (bank-of-anthos) narrative_fields_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Output Tokens 1 375.00 627.00 -252.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) scenario_criteria 1 0.00 0.25 -0.25 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_exact 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) signal_evidence_consistency 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) status_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Tool Calls 1 4.00 6.00 -2.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) topology_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) trajectory 1 0.00 0.75 -0.75 - n/a -
sigevents: KI feature deduplication context contracts dedup_context_contract 2 0.50 0.50 0.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) Chat Calls 2 14.00 3.00 +11.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_retained_count 2 8.50 6.00 +2.50 0.26 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_returned_count 2 10.00 8.50 +1.50 0.15 No -
sigevents: KI feature exclusion (bank-of-anthos) initial_feature_count 2 11.00 9.00 +2.00 0.22 No -
sigevents: KI feature exclusion (bank-of-anthos) Latency 2 79.41 53.34 +26.08 0.37 No -
sigevents: KI feature exclusion (bank-of-anthos) llm_exclude_compliance 2 0.46 0.00 +0.46 0.20 No -
sigevents: KI feature exclusion (bank-of-anthos) Output Tokens 2 6636.00 5428.50 +1207.50 0.06 No -
sigevents: KI feature exclusion (bank-of-anthos) reported_output_tokens 2 9038.00 7997.50 +1040.50 0.06 No -
sigevents: KI feature exclusion (otel-demo) Cached Tokens 2 99584.00 36608.00 +62976.00 0.16 No -
sigevents: KI feature exclusion (otel-demo) follow_up_retained_count 2 7.00 8.00 -1.00 0.39 No -
sigevents: KI feature exclusion (otel-demo) follow_up_returned_count 2 9.50 10.50 -1.00 0.39 No -
sigevents: KI feature exclusion (otel-demo) initial_feature_count 2 9.00 10.50 -1.50 0.15 No -
sigevents: KI feature exclusion (otel-demo) Latency 2 88.94 48.15 +40.80 0.11 No -
sigevents: KI feature exclusion (otel-demo) llm_exclude_compliance 2 0.04 0.00 +0.04 0.39 No -
sigevents: KI feature exclusion (otel-demo) Output Tokens 2 7894.50 8715.50 -821.00 0.48 No -
sigevents: KI feature exclusion (otel-demo) reported_output_tokens 2 10014.50 11740.00 -1725.50 0.06 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Cached Tokens 2 130816.00 12800.00 +118016.00 0.20 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Chat Calls 2 13.50 2.00 +11.50 0.06 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_retained_count 2 1.50 4.00 -2.50 0.09 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_returned_count 2 4.00 5.50 -1.50 0.39 No -
sigevents: KI feature exclusion (quarkus-super-heroes) initial_feature_count 2 5.50 7.50 -2.00 1.00 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Input Tokens 2 232213.00 33331.00 +198882.00 0.06 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Latency 2 49.73 19.95 +29.79 0.08 No -
sigevents: KI feature exclusion (quarkus-super-heroes) llm_exclude_compliance 2 0.04 0.33 -0.29 0.39 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Output Tokens 2 4148.50 3314.00 +834.50 0.62 No -
sigevents: KI feature exclusion (quarkus-super-heroes) reported_output_tokens 2 4523.50 6034.50 -1511.00 0.31 No -
sigevents: KI query generation: empty datastream expected_generation_outcome 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::significant-events::openai-gpt-5.4-nano | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::significant-events::openai-gpt-5.4-nano
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
9 significant difference(s) detected out of 91 comparisons.

Note: 58 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: KI feature exclusion (bank-of-anthos) Chat Calls 2 24.00 4.00 +20.00 0.02 Yes -
sigevents: KI feature exclusion (bank-of-anthos) Input Tokens 2 402631.00 62647.00 +339984.00 0.03 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) reported_input_tokens 2 404744.50 62757.50 +341987.00 0.01 Yes Regression
sigevents: KI feature exclusion (otel-demo) Chat Calls 2 25.50 3.00 +22.50 0.03 Yes -
sigevents: KI feature exclusion (otel-demo) Output Tokens 2 13252.50 6216.50 +7036.00 0.02 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) Cached Tokens 2 335104.00 23296.00 +311808.00 0.02 Yes -
sigevents: KI feature exclusion (quarkus-super-heroes) Chat Calls 2 23.00 2.50 +20.50 0.01 Yes -
sigevents: KI feature exclusion (quarkus-super-heroes) Input Tokens 2 470442.50 41703.50 +428739.00 0.03 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) Latency 2 134.27 32.88 +101.40 0.02 Yes Regression
No significant changes (82 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) Cached Tokens 3 47274.67 68522.67 -21248.00 0.34 No -
sigevents: Discovery (bank-of-anthos) confidence_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) confirmation_alignment 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_collection 3 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_description_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) execute_esql_grounding 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) grouping_correctness 3 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Input Tokens 3 96410.00 120013.67 -23603.67 0.35 No -
sigevents: Discovery (bank-of-anthos) Latency 3 6.67 6.03 +0.64 0.36 No -
sigevents: Discovery (bank-of-anthos) narrative_fields_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Output Tokens 3 495.67 390.67 +105.00 0.68 No -
sigevents: Discovery (bank-of-anthos) scenario_criteria 3 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_exact 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) signal_evidence_consistency 3 0.50 0.67 -0.17 0.60 No -
sigevents: Discovery (bank-of-anthos) status_correctness 3 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Tool Calls 3 3.67 4.00 -0.33 0.69 No -
sigevents: Discovery (bank-of-anthos) topology_correctness 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) trajectory 3 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Cached Tokens 2 253696.00 195200.00 +58496.00 0.18 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Input Tokens 2 271476.50 239548.50 +31928.00 0.06 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Latency 2 22.20 23.79 -1.59 0.62 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Output Tokens 2 841.50 768.50 +73.00 0.15 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Tool Calls 2 8.00 7.00 +1.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) trajectory 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Cached Tokens 2 204544.00 222848.00 -18304.00 0.67 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Input Tokens 2 253409.00 271210.00 -17801.00 0.72 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Latency 2 20.87 24.38 -3.50 0.36 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Output Tokens 2 725.50 835.00 -109.50 0.46 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Tool Calls 2 7.50 8.00 -0.50 0.72 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) trajectory 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Cached Tokens 1 257280.00 124160.00 +133120.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Input Tokens 1 271355.00 207808.00 +63547.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Latency 1 22.80 21.66 +1.15 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Output Tokens 1 779.00 780.00 -1.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Tool Calls 1 8.00 6.00 +2.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) trajectory 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Cached Tokens 1 88832.00 44800.00 +44032.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confidence_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confirmation_alignment 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_collection 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_description_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) execute_esql_grounding 1 1.00 0.00 +1.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) grouping_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Input Tokens 1 183454.00 118228.00 +65226.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Latency 1 7.17 4.16 +3.01 - n/a -
sigevents: Discovery memory (bank-of-anthos) narrative_fields_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Output Tokens 1 471.00 379.00 +92.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) scenario_criteria 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_exact 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) signal_evidence_consistency 1 0.50 1.00 -0.50 - n/a -
sigevents: Discovery memory (bank-of-anthos) status_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Tool Calls 1 6.00 4.00 +2.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) topology_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) trajectory 1 0.00 0.00 0.00 - n/a -
sigevents: KI feature deduplication context contracts dedup_context_contract 2 0.00 0.50 -0.50 0.39 No -
sigevents: KI feature exclusion (bank-of-anthos) Cached Tokens 2 287104.00 45440.00 +241664.00 0.13 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_retained_count 2 6.50 4.00 +2.50 0.09 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_returned_count 2 8.00 6.50 +1.50 0.15 No -
sigevents: KI feature exclusion (bank-of-anthos) initial_feature_count 2 12.00 5.50 +6.50 0.23 No -
sigevents: KI feature exclusion (bank-of-anthos) Latency 2 172.35 65.00 +107.35 0.22 No -
sigevents: KI feature exclusion (bank-of-anthos) llm_exclude_compliance 2 0.58 0.17 +0.42 0.09 No -
sigevents: KI feature exclusion (bank-of-anthos) Output Tokens 2 13466.00 6474.50 +6991.50 0.27 No -
sigevents: KI feature exclusion (bank-of-anthos) reported_output_tokens 2 12154.50 6462.50 +5692.00 0.18 No -
sigevents: KI feature exclusion (otel-demo) Cached Tokens 2 265088.00 24064.00 +241024.00 0.08 No -
sigevents: KI feature exclusion (otel-demo) follow_up_retained_count 2 8.50 6.50 +2.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) follow_up_returned_count 2 9.00 8.00 +1.00 0.39 No -
sigevents: KI feature exclusion (otel-demo) initial_feature_count 2 10.00 9.50 +0.50 0.72 No -
sigevents: KI feature exclusion (otel-demo) Input Tokens 2 461227.00 45519.00 +415708.00 0.07 No -
sigevents: KI feature exclusion (otel-demo) Latency 2 183.25 63.47 +119.78 0.11 No -
sigevents: KI feature exclusion (otel-demo) llm_exclude_compliance 2 0.71 0.63 +0.08 0.39 No -
sigevents: KI feature exclusion (otel-demo) reported_input_tokens 2 476070.00 60759.50 +415310.50 0.08 No -
sigevents: KI feature exclusion (otel-demo) reported_output_tokens 2 11666.50 8145.50 +3521.00 0.13 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_retained_count 2 5.00 2.00 +3.00 1.00 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_returned_count 2 6.00 3.50 +2.50 0.09 No -
sigevents: KI feature exclusion (quarkus-super-heroes) initial_feature_count 2 6.00 5.50 +0.50 0.72 No -
sigevents: KI feature exclusion (quarkus-super-heroes) llm_exclude_compliance 2 0.67 0.17 +0.50 0.39 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Output Tokens 2 10826.50 2627.00 +8199.50 0.13 No -
sigevents: KI feature exclusion (quarkus-super-heroes) reported_input_tokens 2 464135.50 66917.50 +397218.00 0.05 No -
sigevents: KI feature exclusion (quarkus-super-heroes) reported_output_tokens 2 8710.00 4024.50 +4685.50 0.05 No -
sigevents: KI query generation: empty datastream expected_generation_outcome 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::significant-events::openai-gpt-5.6-luna | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::significant-events::openai-gpt-5.6-luna
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
15 significant difference(s) detected out of 85 comparisons.

Note: 1 unpaired (ran in only one experiment), 70 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: KI feature exclusion (bank-of-anthos) Input Tokens 2 223228.50 39068.50 +184160.00 0.00 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) Latency 2 60.64 27.65 +32.99 0.01 Yes Regression
sigevents: KI feature exclusion (bank-of-anthos) reported_input_tokens 2 266658.50 62642.50 +204016.00 0.00 Yes Regression
sigevents: KI feature exclusion (otel-demo) Chat Calls 2 14.50 4.50 +10.00 0.04 Yes -
sigevents: KI feature exclusion (otel-demo) follow_up_retained_count 2 7.50 12.00 -4.50 0.05 Yes Regression
sigevents: KI feature exclusion (otel-demo) follow_up_returned_count 2 8.00 12.50 -4.50 0.05 Yes Regression
sigevents: KI feature exclusion (otel-demo) initial_feature_count 2 9.00 15.50 -6.50 0.03 Yes Regression
sigevents: KI feature exclusion (otel-demo) Input Tokens 2 238852.50 68125.00 +170727.50 0.04 Yes Regression
sigevents: KI feature exclusion (otel-demo) Latency 2 64.94 54.58 +10.36 0.04 Yes Regression
sigevents: KI feature exclusion (otel-demo) reported_input_tokens 2 265514.50 60530.50 +204984.00 0.00 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) Chat Calls 2 13.50 2.00 +11.50 0.02 Yes -
sigevents: KI feature exclusion (quarkus-super-heroes) Input Tokens 2 232329.50 33306.00 +199023.50 0.02 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) llm_exclude_compliance 2 0.96 0.00 +0.96 0.02 Yes Improvement
sigevents: KI feature exclusion (quarkus-super-heroes) reported_input_tokens 2 277235.00 66758.00 +210477.00 0.00 Yes Regression
sigevents: KI feature exclusion (quarkus-super-heroes) reported_output_tokens 2 4194.50 6293.50 -2099.00 0.04 Yes Improvement
No significant changes (70 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) Cached Tokens 1 171576.00 135044.00 +36532.00 - n/a -
sigevents: Discovery (bank-of-anthos) confidence_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) confirmation_alignment 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_collection 3 0.33 0.33 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_description_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) execute_esql_grounding 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) grouping_correctness 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Input Tokens 3 124762.67 156495.33 -31732.67 0.53 No -
sigevents: Discovery (bank-of-anthos) Latency 3 6.36 6.67 -0.32 0.94 No -
sigevents: Discovery (bank-of-anthos) narrative_fields_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Output Tokens 3 470.00 729.33 -259.33 0.52 No -
sigevents: Discovery (bank-of-anthos) scenario_criteria 3 0.33 0.14 +0.19 0.50 No -
sigevents: Discovery (bank-of-anthos) severity_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_exact 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) signal_evidence_consistency 3 0.67 0.50 +0.17 0.69 No -
sigevents: Discovery (bank-of-anthos) status_correctness 3 0.33 0.00 +0.33 0.35 No -
sigevents: Discovery (bank-of-anthos) Tool Calls 3 4.33 5.00 -0.67 0.69 No -
sigevents: Discovery (bank-of-anthos) topology_correctness 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) trajectory 3 0.33 0.25 +0.08 0.86 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Cached Tokens 2 144752.50 98890.50 +45862.00 0.17 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Input Tokens 2 324546.00 257564.00 +66982.00 0.21 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Latency 2 20.39 21.09 -0.70 0.85 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Output Tokens 2 1079.50 732.00 +347.50 0.24 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Tool Calls 2 10.00 8.00 +2.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) trajectory 2 0.31 0.00 +0.31 0.09 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Cached Tokens 2 151980.50 183650.00 -31669.50 0.63 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Input Tokens 2 310984.00 360291.00 -49307.00 0.44 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Latency 2 36.53 25.43 +11.10 0.13 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Output Tokens 2 1063.50 1135.00 -71.50 0.82 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Tool Calls 2 9.00 10.50 -1.50 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) trajectory 2 0.31 0.31 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Cached Tokens 1 162735.00 127211.00 +35524.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Input Tokens 1 342718.00 307004.00 +35714.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Latency 1 22.78 24.20 -1.41 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Output Tokens 1 1270.00 1106.00 +164.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Tool Calls 1 10.00 9.00 +1.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) trajectory 1 0.38 0.25 +0.13 - n/a -
sigevents: Discovery memory (bank-of-anthos) confidence_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confirmation_alignment 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_collection 1 1.00 0.00 +1.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_description_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Input Tokens 1 219887.00 111957.00 +107930.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Latency 1 3.76 5.20 -1.44 - n/a -
sigevents: Discovery memory (bank-of-anthos) narrative_fields_quality 1 0.83 1.00 -0.17 - n/a -
sigevents: Discovery memory (bank-of-anthos) Output Tokens 1 810.00 327.00 +483.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) scenario_criteria 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_exact 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) signal_evidence_consistency 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) status_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Tool Calls 1 8.00 4.00 +4.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) topology_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) trajectory 1 0.75 0.00 +0.75 - n/a -
sigevents: KI feature deduplication context contracts dedup_context_contract 2 0.50 0.50 0.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) Chat Calls 2 13.50 2.50 +11.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_retained_count 2 6.00 8.50 -2.50 0.09 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_returned_count 2 7.50 11.00 -3.50 0.06 No -
sigevents: KI feature exclusion (bank-of-anthos) initial_feature_count 2 8.50 10.00 -1.50 0.39 No -
sigevents: KI feature exclusion (bank-of-anthos) llm_exclude_compliance 2 0.67 0.17 +0.50 0.15 No -
sigevents: KI feature exclusion (bank-of-anthos) Output Tokens 2 5655.00 5181.50 +473.50 0.27 No -
sigevents: KI feature exclusion (bank-of-anthos) reported_output_tokens 2 7066.50 8111.50 -1045.00 0.20 No -
sigevents: KI feature exclusion (otel-demo) llm_exclude_compliance 2 0.92 0.79 +0.13 0.55 No -
sigevents: KI feature exclusion (otel-demo) Output Tokens 2 7642.50 10645.00 -3002.50 0.18 No -
sigevents: KI feature exclusion (otel-demo) reported_output_tokens 2 8254.00 10213.50 -1959.50 0.23 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_retained_count 2 3.50 4.50 -1.00 0.39 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_returned_count 2 3.50 7.00 -3.50 0.06 No -
sigevents: KI feature exclusion (quarkus-super-heroes) initial_feature_count 2 5.50 8.50 -3.00 0.15 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Latency 2 45.82 22.25 +23.58 0.07 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Output Tokens 2 3402.50 3358.00 +44.50 0.83 No -
sigevents: KI query generation: empty datastream expected_generation_outcome 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::significant-events::openai-gpt-5.6-terra | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::significant-events::openai-gpt-5.6-terra
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
6 significant difference(s) detected out of 95 comparisons.

Note: 2 unpaired (ran in only one experiment), 53 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) Cached Tokens 3 136653.67 100300.67 +36353.00 0.00 Yes -
sigevents: Discovery (bank-of-anthos) Input Tokens 3 228740.67 192139.33 +36601.33 0.00 Yes Regression
sigevents: KI feature exclusion (otel-demo) Cached Tokens 2 57009.00 29659.50 +27349.50 0.05 Yes -
sigevents: KI feature exclusion (otel-demo) Chat Calls 2 18.50 3.00 +15.50 0.01 Yes -
sigevents: KI feature exclusion (otel-demo) Input Tokens 2 301853.00 45309.00 +256544.00 0.01 Yes Regression
sigevents: KI feature exclusion (otel-demo) reported_input_tokens 2 346620.00 60455.00 +286165.00 0.03 Yes Regression
No significant changes (89 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) confidence_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) confirmation_alignment 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_collection 3 0.67 1.00 -0.33 0.35 No -
sigevents: Discovery (bank-of-anthos) evidence_description_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) execute_esql_grounding 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) grouping_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) Latency 3 5.70 4.42 +1.28 0.32 No -
sigevents: Discovery (bank-of-anthos) narrative_fields_quality 3 0.89 1.00 -0.11 0.35 No -
sigevents: Discovery (bank-of-anthos) Output Tokens 3 1114.67 895.00 +219.67 0.26 No -
sigevents: Discovery (bank-of-anthos) scenario_criteria 3 0.39 0.39 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_exact 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) signal_evidence_consistency 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) status_correctness 3 0.33 0.33 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Tool Calls 3 7.33 7.00 +0.33 0.60 No -
sigevents: Discovery (bank-of-anthos) topology_correctness 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) trajectory 3 0.83 0.83 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Cached Tokens 2 201737.00 233317.50 -31580.50 0.39 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_routing 2 1.00 1.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) continuation_stability 2 1.00 1.00 0.00 1.00 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Input Tokens 2 396213.00 418379.50 -22166.50 0.39 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Latency 2 28.63 31.66 -3.03 0.53 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Output Tokens 2 1475.00 1555.50 -80.50 0.44 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Tool Calls 2 12.00 12.50 -0.50 0.39 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) trajectory 2 0.75 0.75 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Cached Tokens 2 251175.50 250836.50 +339.00 0.99 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_routing 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) continuation_stability 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Input Tokens 2 431847.00 431182.00 +665.00 0.98 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Latency 2 29.86 27.33 +2.53 0.56 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Output Tokens 2 1763.50 1526.00 +237.50 0.24 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Tool Calls 2 13.50 13.00 +0.50 0.39 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) trajectory 2 0.75 0.50 +0.25 0.22 No -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Cached Tokens 1 268937.00 233633.00 +35304.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_routing 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) continuation_stability 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Input Tokens 1 449656.00 414071.00 +35585.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Latency 1 27.50 28.53 -1.02 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Output Tokens 1 1754.00 1502.00 +252.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Tool Calls 1 14.00 12.00 +2.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) trajectory 1 0.38 0.38 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Cached Tokens 1 100748.00 64744.00 +36004.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confidence_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confirmation_alignment 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_collection 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_description_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Input Tokens 1 219853.00 183608.00 +36245.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Latency 1 5.36 4.00 +1.36 - n/a -
sigevents: Discovery memory (bank-of-anthos) narrative_fields_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Output Tokens 1 812.00 489.00 +323.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) scenario_criteria 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_exact 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) signal_evidence_consistency 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) status_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Tool Calls 1 8.00 6.00 +2.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) topology_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) trajectory 1 0.75 0.75 0.00 - n/a -
sigevents: KI feature deduplication context contracts dedup_context_contract 2 0.50 0.50 0.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) Cached Tokens 1 32913.00 35803.00 -2890.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) Chat Calls 2 9.00 3.00 +6.00 0.48 No -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_retained_count 1 3.00 9.00 -6.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) follow_up_returned_count 1 3.00 9.00 -6.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) initial_feature_count 2 3.50 9.50 -6.00 1.00 No -
sigevents: KI feature exclusion (bank-of-anthos) Input Tokens 2 146153.50 46900.00 +99253.50 0.48 No -
sigevents: KI feature exclusion (bank-of-anthos) Latency 2 35.64 33.94 +1.70 0.95 No -
sigevents: KI feature exclusion (bank-of-anthos) llm_exclude_compliance 1 1.00 1.00 0.00 - n/a -
sigevents: KI feature exclusion (bank-of-anthos) Output Tokens 2 1403.50 4693.00 -3289.50 0.15 No -
sigevents: KI feature exclusion (bank-of-anthos) reported_input_tokens 2 188244.50 62579.00 +125665.50 0.39 No -
sigevents: KI feature exclusion (bank-of-anthos) reported_output_tokens 2 2141.00 6318.00 -4177.00 0.10 No -
sigevents: KI feature exclusion (otel-demo) follow_up_retained_count 2 5.50 9.00 -3.50 0.06 No -
sigevents: KI feature exclusion (otel-demo) follow_up_returned_count 2 5.50 9.00 -3.50 0.06 No -
sigevents: KI feature exclusion (otel-demo) initial_feature_count 2 7.50 13.00 -5.50 0.27 No -
sigevents: KI feature exclusion (otel-demo) Latency 2 77.24 45.66 +31.57 0.07 No -
sigevents: KI feature exclusion (otel-demo) llm_exclude_compliance 2 1.00 1.00 0.00 1.00 No -
sigevents: KI feature exclusion (otel-demo) Output Tokens 2 5269.00 6584.50 -1315.50 0.06 No -
sigevents: KI feature exclusion (otel-demo) reported_output_tokens 2 6567.50 8583.00 -2015.50 0.10 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Cached Tokens 1 51993.00 9346.00 +42647.00 - n/a -
sigevents: KI feature exclusion (quarkus-super-heroes) Chat Calls 2 14.50 2.00 +12.50 0.09 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_retained_count 2 2.00 3.50 -1.50 0.15 No -
sigevents: KI feature exclusion (quarkus-super-heroes) follow_up_returned_count 2 2.00 3.50 -1.50 0.15 No -
sigevents: KI feature exclusion (quarkus-super-heroes) initial_feature_count 2 5.50 7.00 -1.50 0.39 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Input Tokens 2 249159.50 33302.50 +215857.00 0.09 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Latency 2 59.40 20.68 +38.72 0.16 No -
sigevents: KI feature exclusion (quarkus-super-heroes) llm_exclude_compliance 2 0.50 1.00 -0.50 0.39 No -
sigevents: KI feature exclusion (quarkus-super-heroes) Output Tokens 2 2804.50 2422.50 +382.00 0.72 No -
sigevents: KI feature exclusion (quarkus-super-heroes) reported_input_tokens 2 309797.50 66755.50 +243042.00 0.07 No -
sigevents: KI feature exclusion (quarkus-super-heroes) reported_output_tokens 2 3271.50 4066.50 -795.00 0.42 No -
sigevents: KI query generation: empty datastream expected_generation_outcome 1 1.00 1.00 0.00 - n/a -

PR run: bk-01a0951d-95bd-4b60-b1da-e440f89a96f4::significant-events::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::significant-events::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
5 significant difference(s) detected out of 50 comparisons.

Note: 10 unpaired (ran in only one experiment), 73 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

Significant changes

Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) Tool Calls 3 0.00 4.67 -4.67 0.04 Yes -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Tool Calls 2 0.00 5.50 -5.50 0.04 Yes -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Input Tokens 2 59495.50 278708.00 -219212.50 0.01 Yes Improvement
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Output Tokens 2 4388.00 2635.00 +1753.00 0.02 Yes Regression
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Tool Calls 2 0.00 8.50 -8.50 0.03 Yes -
No significant changes (45 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff p-value Sig Outcome
sigevents: Discovery (bank-of-anthos) confidence_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) confirmation_alignment 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) evidence_collection 3 0.00 0.33 -0.33 0.35 No -
sigevents: Discovery (bank-of-anthos) evidence_description_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) grouping_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery (bank-of-anthos) Input Tokens 3 17782.33 195355.33 -177573.00 0.17 No -
sigevents: Discovery (bank-of-anthos) Latency 3 9.65 8.03 +1.62 0.81 No -
sigevents: Discovery (bank-of-anthos) narrative_fields_quality 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) Output Tokens 3 1877.67 3385.33 -1507.67 0.65 No -
sigevents: Discovery (bank-of-anthos) scenario_criteria 3 0.00 0.38 -0.38 0.28 No -
sigevents: Discovery (bank-of-anthos) severity_calibration 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) severity_exact 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) signal_evidence_consistency 3 1.00 1.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) status_correctness 3 0.00 0.33 -0.33 0.35 No -
sigevents: Discovery (bank-of-anthos) topology_correctness 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery (bank-of-anthos) trajectory 3 0.00 0.25 -0.25 0.35 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Input Tokens 2 72034.50 179806.00 -107771.50 0.08 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Latency 2 29.54 19.39 +10.16 0.25 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) Output Tokens 2 5037.50 2080.00 +2957.50 0.10 No -
sigevents: Discovery agent continuation - closed significant event (bank-of-anthos) trajectory 2 0.00 0.00 0.00 1.00 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) Latency 2 20.81 39.30 -18.48 0.35 No -
sigevents: Discovery agent continuation - open significant event with same rules (bank-of-anthos) trajectory 2 0.00 0.31 -0.31 0.09 No -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Cached Tokens 1 65789.00 122.00 +65667.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Input Tokens 1 74538.00 126176.00 -51638.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Latency 1 15.38 48.77 -33.38 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Output Tokens 1 4308.00 6920.00 -2612.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) Tool Calls 1 9.00 3.00 +6.00 - n/a -
sigevents: Discovery agent continuation - open significant events with topology-related rules (bank-of-anthos) trajectory 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confidence_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) confirmation_alignment 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_collection 1 0.00 1.00 -1.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) evidence_description_quality 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Input Tokens 1 25460.00 243266.00 -217806.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Latency 1 9.57 9.20 +0.37 - n/a -
sigevents: Discovery memory (bank-of-anthos) narrative_fields_quality 1 1.00 0.50 +0.50 - n/a -
sigevents: Discovery memory (bank-of-anthos) Output Tokens 1 968.00 3461.00 -2493.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) scenario_criteria 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_calibration 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) severity_exact 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) signal_evidence_consistency 1 1.00 1.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) status_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) Tool Calls 1 0.00 7.00 -7.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) topology_correctness 1 0.00 0.00 0.00 - n/a -
sigevents: Discovery memory (bank-of-anthos) trajectory 1 0.00 0.75 -0.75 - n/a -
sigevents: KI query generation: empty datastream expected_generation_outcome 1 0.00 1.00 -1.00 - n/a -

@kibanamachine

kibanamachine commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

🧪 LLM eval (PR) — Significant Events (significant-events) failed.

Failing models:

  • eis-anthropic-claude-4-6-opus
  • eis-anthropic-claude-4-6-sonnet
  • eis-anthropic-claude-4-7-opus
  • eis-anthropic-claude-4-8-opus
  • eis-google-gemini-3-0-flash
  • eis-google-gemini-3-1-flash-lite
  • eis-google-gemini-3-5-flash
  • eis-openai-gpt-5-2
  • eis-openai-gpt-5-4
  • eis-openai-gpt-5-4-mini
  • eis-openai-gpt-5-4-nano
  • eis-openai-gpt-5-6-luna
  • eis-openai-gpt-5-6-terra
  • eis-openai-gpt-oss-120b

View build

Triage summary:

ERROR Error: Playwright exited with code 1

x-pack/platform/packages/shared/kbn-evals-suite-significant-events/evals/ki_feature_extraction/ki_feature_extraction.spec.ts:102:36 — eis-anthropic-claude-4-6-opus, eis-anthropic-claude-4-6-sonnet, eis-anthropic-claude-4-7-opus, eis-anthropic-claude-4-8-opus, eis-google-gemini-3-0-flash, eis-google-gemini-3-1-flash-lite, eis-google-gemini-3-5-flash, eis-openai-gpt-5-2, eis-openai-gpt-5-4, eis-openai-gpt-5-4-mini, eis-openai-gpt-5-4-nano, eis-openai-gpt-5-6-luna, eis-openai-gpt-5-6-terra, eis-openai-gpt-oss-120b
Root cause: Playwright test execution failed during KI feature extraction. Check the Playwright execution logs and test output for the underlying assertion or runtime errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@viduni94 viduni94 viduni94 approved these changes

Assignees

Couldn't load assignees.