[kbn-evals] Scaffold the Nightshift investigations eval suite

[viduni94]

Closes https://github.com/elastic/nightshift-program/issues/1402

Summary

Adds @kbn/evals-suite-nightshift-investigations with a single smoke eval, so the suite's wiring is proven before real evaluator work starts. The eval restores a synthetic snapshot from GCS, reports what landed in the eval cluster, and scores it with two CODE evaluators. No model takes part, so a red score points at seed data loading, score ingestion or the golden-cluster export rather than at investigation quality.

Real evaluators against the investigation engine land on top of this in a follow-up, so the layout is built for that: one eval per folder under evals/, with src/ holding only what every eval shares. Adding an eval means copying evals/smoke/ and editing five files, with nothing shared to edit. The README also separates the two things that both get called "dataset" - seed data (the documents an eval runs against) from eval datasets (the examples scored) - and keeps them independent, so an eval dataset and its evaluators can later be pointed at a different target under test.

Beyond the suite itself:

• node scripts/publish_nightshift_eval_snapshot.js creates the synthetic snapshot. It refuses to overwrite an existing snapshot unless --replace is passed, because snapshots are found by name and labelled ground truth is not recoverable.
• NIGHTSHIFT_DATASETS runs a subset of eval datasets, failing on unknown ids rather than silently running fewer.
• CI registration in evals.suites.json (label evals:nightshift-investigations, Slack #nightshift-alerts) plus a weekly step in llm_evals.yml. The suite pins a cheap connector through defaultModelGroups, so no models:* label is needed - no model affects the score.

Testing

Local run

Checklist

• Unit or functional tests were updated or added to match the most common scenarios
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
• Review the backport guidelines and apply applicable backport:* labels.

[infra-vault-gh-plugin-prod]

Pinging @elastic/nightshift-context-and-research-team (Team:nightshift-context-and-research)

[kibanamachine]

PR size reminder

This PR has 1693 added lines of reviewable code, which is above the 500-line guideline for Nightshift PRs.

Large PRs get significantly less review engagement and take longer to merge. Consider splitting this into smaller, focused PRs before requesting review.

[Copilot]

Note

Copilot was unable to run its full agentic suite in this review.

Pull request overview

Adds a new Nightshift investigations eval suite package with an initial “smoke” eval to validate suite wiring (seed snapshot restore → task execution → CODE evaluator scoring → score ingestion), plus tooling/CI plumbing to publish and run the suite.

Changes:

• Introduces @kbn/evals-suite-nightshift-investigations package with shared dataset/seed-data helpers and a smoke eval.
• Adds a developer CLI (scripts/publish_nightshift_eval_snapshot.js) to publish the synthetic GCS-backed snapshot used by the smoke eval.
• Registers the suite in Buildkite (suite registry + weekly run step) and wires workspace/TS pathing.

Reviewed changes

Copilot reviewed 36 out of 37 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/tsconfig.json	TypeScript config for the new eval suite package.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/src/seed_data/with_seed_data.ts	Adds a helper to seed/cleanup seed data via suite-wide hooks.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/src/seed_data/types.ts	Defines seed source contracts and seeding dependency/result types.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/src/seed_data/sources.ts	Declares the synthetic smoke seed snapshot source and constants.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/src/seed_data/seed.ts	Implements seeding dispatch + teardown cleanup for seeded indices.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/src/seed_data/publish_es_snapshot.ts	Adds snapshot publishing to GCS with overwrite protection.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/src/seed_data/index.ts	Exposes seed-data utilities/types to eval-side consumers.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/src/seed_data/es_snapshot.ts	Implements snapshot replay into eval cluster + stale temp index cleanup.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/src/evaluate.ts	Centralizes evaluate import for future suite-wide fixtures.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/src/datasets/types.ts	Defines suite-level dataset type combining eval examples and seed source.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/src/datasets/to_evaluation_dataset.ts	Converts suite dataset shape to framework EvaluationDataset.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/src/datasets/select_datasets.ts	Implements NIGHTSHIFT_DATASETS filtering with validation.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/src/datasets/select_datasets.test.ts	Adds unit coverage for dataset selection behavior.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/src/datasets/index.ts	Exports dataset utilities/types.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/scripts/publish_synthetic_snapshot/publish_synthetic_snapshot.ts	Implements synthetic doc seeding + snapshot publishing CLI logic.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/scripts/publish_synthetic_snapshot/index.ts	Registers the CLI runner entrypoint and flags/help text.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/scripts/lib/es_client.ts	Adds helper to create ES client (direct ES URL or via Kibana proxy).
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/playwright.config.ts	Adds Playwright config for running eval specs under evals/.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/package.json	Declares the new private workspace package.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/moon.yml	Registers the project with Moon (generated).
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/kibana.jsonc	Declares Kibana project metadata/ownership for the new suite package.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/jest.config.js	Adds Jest config for unit tests in the suite package.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/evals/smoke/types.ts	Defines smoke eval example/expectation and evaluator types.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/evals/smoke/task.ts	Adds smoke eval task that summarizes seeded data in ES.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/evals/smoke/smoke.spec.ts	Adds Playwright spec wiring datasets → seeding → task → evaluators.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/evals/smoke/evaluators.ts	Adds CODE evaluators for doc count + timestamp replay checks.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/evals/smoke/evaluators.test.ts	Adds unit tests for smoke evaluators.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/evals/smoke/datasets.ts	Declares the synthetic-smoke eval dataset and filters via env var.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/README.md	Documents suite purpose, layout, datasets vs seed data, and CI usage.
x-pack/platform/packages/shared/kbn-evals-suite-nightshift-investigations/.gitignore	Ignores Playwright test results directory.
tsconfig.base.json	Adds TS path aliases for the new suite package.
scripts/publish_nightshift_eval_snapshot.js	Adds repo-root script entrypoint for publishing the synthetic snapshot.
pnpm-workspace.yaml	Adds the new suite package to the pnpm workspace.
package.json	Adds the new package as a workspace dependency.
.buildkite/pipelines/evals/llm_evals.yml	Adds weekly Buildkite job to run the Nightshift investigations suite.
.buildkite/pipelines/evals/evals.suites.json	Registers the suite (id, config path, labels, default model group, etc.).

Files not reviewed (1)

• pnpm-lock.yaml: Generated file

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[MinasCham]

Hey @viduni94 , code review only and focused on the evals mostly, one blocker inline in my eyes but otherwise LGTM!

[milistu]

LGTM 🦋 Great work!

[MinasCham]

Thanks for addressing this @viduni94 , 🚢 it

[kibanamachine]

PR run: bk-01a09590-5dc0-4e13-9b28-233bfc015595::smoke-tests::anthropic-claude-4.5-haiku | Baseline (main): bk-01a086a9-0774-4273-8efa-e2b095b5e434::smoke-tests::anthropic-claude-4.5-haiku
Baseline: commit 144b3c5, 2 days ago
Significance threshold: p < 0.05

Summary
No significant regressions detected (5 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (5 rows)

Dataset	Evaluator	N	Mean (PR)	Mean (main)	Diff	p-value	Sig	Outcome
smoke tests: es-snapshot-loader	SnapshotRestored	1	1.00	1.00	0.00	-	n/a	-
smoke tests: llm-judge	Criteria	1	1.00	1.00	0.00	-	n/a	-
smoke tests: score ingestion and code evaluator	ContainsKibana	1	1.00	1.00	0.00	-	n/a	-
smoke tests: trace-retrieval	Input Tokens	1	12.00	12.00	0.00	-	n/a	-
smoke tests: trace-retrieval	Output Tokens	1	4.00	5.00	-1.00	-	n/a	-

[kibanamachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 0a1c11f
• Build duration: 52 mins
• Storybooks Preview

Failed CI Steps

• Jest Tests #10
• FTR Configs #12
• FTR Configs #75
• Serverless Rule Management - Prebuilt Rules Upgrade - Security Solution Cypress Tests #1

Metrics [docs]

✅ unchanged

Test Failures

• [job] [logs] FTR Configs #75 / alerting api integration security and spaces enabled - Group 4 Alerts alerts alerts space_1_all at space1 should throttle alerts when appropriate
• [job] [logs] Jest Tests #10 / McpClientForm supports adding and removing remote redirect URLs
• [job] [logs] FTR Configs #12 / serverless security UI Security ML Notifications page displays only notification types for security projects

History

• 💔 Build #500672 failed bcde53b
• 💔 Build #500578 failed 95c5a43
• 💔 Build #500540 failed 381b973
• 💔 Build #500492 failed c1bae15

cc @viduni94