Merge branch 'main' into rk/ab-font-sizes

Author: ryankeairns

.agents/skills/flaky-test-investigator/SKILL.md

@@ -45,7 +45,7 @@ For every failure, try to retrieve:
 
 Things to specifically check in the artifacts before forming a root-cause hypothesis:
 
-- **Did the expected element render at all?** If yes and the selector missed it → flaky selector (Tier 2 fix territory). If no → real rendering / race / data issue (Tier 1 territory).
+- **Did the expected element render at all?** If yes and the selector missed it → flaky selector (Tier 2 fix territory). If no, distinguish **not yet** (a missing test-side wait) from **never** (the awaited state is unreachable — e.g. the component doesn't re-render when its async data arrives): read the component that renders the element, or find trace evidence of it appearing later. A missing wait in the test does not by itself prove the element would eventually have rendered.
 - **Is there an error visible in the UI** (toast, banner, console error in the HTML report)? If yes → product side, not test side.
 - **Is the page in an unexpected state** (different URL, different user's data, different space)? → cleanup or isolation issue, often points at `afterEach` / `afterAll`.
 - **Does the screenshot timestamp match the failure timestamp**? Stale artifacts from a prior step can mislead.
@@ -114,6 +114,7 @@ Watch out for these pitfalls when investigating the failure:
 - **Reducing coverage surface**: don't recommend stripping tags to skip the test in certain environments (e.g. Cloud) or project types (e.g. serverless Security) unless you have a real reason it shouldn't run there. "It's flaky here" is not a real reason.
 - **Trusting flaky-test-runner alone**: a green 30/30 or 60/60 run does not prove a fix held. The runner runs tests in isolation, which isn't always the case (Scout test runs share the same test servers for multiple test configs).
 - **Assuming "fix the test, not the product"**: always ask first whether the product could be at fault. Test-only fixes are meaningfully less durable than fixes that change production code.
+- **Reading fault from the throwing stack frame**: a waiting-side timeout always throws from the waiter (FTR/Playwright service code), so the frame tells you who threw, not whose fault it is. It is not evidence against a product bug.
 - **Reporting false certainty**: "I don't know, here are the two plausible explanations and what would distinguish them" is more useful to the owning team than a confident wrong answer.
 
 ### Is a fix worth it?

.buildkite/pipeline-utils/affected-packages/README.md

@@ -134,6 +134,10 @@ const filteredFiles = filterFilesByPackages(
 
 On pull request builds, Jest unit and integration test groups are narrowed to configs under affected packages (see `pick_test_group_run_order` in CI stats). Add the GitHub label `ci:prevent-selective-testing` to run the full Jest suite instead. Touching files listed in `CRITICAL_FILES_JEST_*` in `const.ts` also skips filtering for the relevant test type.
 
+### Always-run integration configs
+
+Some integration suites boot a full Kibana and snapshot a *global registry* (rule-type params, connector types, task types, …) populated at runtime by downstream publishers that sit **upstream** of the suite's own package. `includeDownstream` expansion never reaches them, so a publisher-only change can silently skip the snapshot. Configs listed in `ALWAYS_RUN_JEST_INTEGRATION_CONFIGS` (`const.ts`) are re-added after affected-filtering so they run on every PR regardless of the graph. Keep the list tiny — it is a deliberate escape hatch.
+
 ## Scout selective testing: git -> Moon (shadow mode)
 
 `resolve_selective_testing.ts` still uses **git** as the authoritative strategy (written to `.scout/code_changes.json`, no behavior change). In parallel, it runs the **Moon** strategy above for observation only, and writes the result plus a diff of `affectedModules` to `.scout/code_changes.moon_shadow.json` (uploaded as a Buildkite artifact). Mismatches are logged as a warning via `ToolingLog`; a Moon failure is swallowed and logged, never fails the build.

.buildkite/pipeline-utils/affected-packages/const.ts

@@ -47,3 +47,12 @@ export const CRITICAL_FILES_JEST_INTEGRATION_TESTS = [
   '.buildkite/pipeline-utils/affected-packages/**/*.{ts,js,sh}',
   '.buildkite/pipeline-utils/ci-stats/**/*.{ts,js}',
 ];
+
+// Integration configs that snapshot a global registry (rule-type params, connector types, task
+// types) fed by downstream plugins. Those publishers sit upstream of these configs, so
+// includeDownstream never marks them affected — they must run regardless of the graph. Keep tiny.
+export const ALWAYS_RUN_JEST_INTEGRATION_CONFIGS = [
+  'x-pack/platform/plugins/shared/alerting/jest.integration.config.js',
+  'x-pack/platform/plugins/shared/actions/jest.integration.config.js',
+  'x-pack/platform/plugins/shared/task_manager/jest.integration.config.js',
+];

.buildkite/pipeline-utils/ci-stats/pick_test_group_run_order/selective_testing.test.ts

@@ -0,0 +1,97 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the "Elastic License
+ * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
+ * Public License v 1"; you may not use this file except in compliance with, at
+ * your election, the "Elastic License 2.0", the "GNU Affero General Public
+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+jest.mock('../../affected-packages', () => ({
+  ALWAYS_RUN_JEST_INTEGRATION_CONFIGS: ['always/jest.integration.config.js'],
+  CRITICAL_FILES_JEST_INTEGRATION_TESTS: ['CRITICAL_INT'],
+  CRITICAL_FILES_JEST_UNIT_TESTS: ['CRITICAL_UNIT'],
+  getAffectedPackages: jest.fn(),
+  listChangedFiles: jest.fn(),
+  filterFilesByPackages: (files: string[], pkgs: Set<string>) =>
+    files.filter((f) => [...pkgs].some((pkg) => f.startsWith(pkg))),
+  touchedCriticalFiles: (files: string[], critical: string[]) =>
+    files.some((f) => critical.includes(f)),
+}));
+
+jest.mock('./jest_configs', () => ({ SHARD_ANNOTATION_SEP: '||shard=' }));
+
+import type { SelectiveTestingContext } from './selective_testing';
+import {
+  filterJestIntegrationConfigsByAffected,
+  filterJestUnitConfigsByAffected,
+} from './selective_testing';
+
+const context = (
+  affected: string[],
+  changed: string[] = ['irrelevant.ts']
+): SelectiveTestingContext => ({
+  affectedPackages: new Set(affected),
+  prChangedFiles: changed,
+});
+
+describe('filterJestIntegrationConfigsByAffected', () => {
+  it('drops unaffected configs but re-adds always-run configs', () => {
+    const configs = [
+      'always/jest.integration.config.js',
+      'other/jest.integration.config.js',
+      'affected/jest.integration.config.js',
+    ];
+
+    const result = filterJestIntegrationConfigsByAffected(configs, context(['affected/']));
+
+    expect(result).toEqual(
+      expect.arrayContaining([
+        'always/jest.integration.config.js',
+        'affected/jest.integration.config.js',
+      ])
+    );
+    expect(result).not.toContain('other/jest.integration.config.js');
+  });
+
+  it('re-adds an always-run config even when no package is affected', () => {
+    const configs = ['always/jest.integration.config.js', 'other/jest.integration.config.js'];
+
+    const result = filterJestIntegrationConfigsByAffected(configs, context([]));
+
+    expect(result).toEqual(['always/jest.integration.config.js']);
+  });
+
+  it('restores every shard of an always-run config', () => {
+    const configs = [
+      'always/jest.integration.config.js||shard=1/2',
+      'always/jest.integration.config.js||shard=2/2',
+      'other/jest.integration.config.js',
+    ];
+
+    const result = filterJestIntegrationConfigsByAffected(configs, context(['nothing/']));
+
+    expect(result).toEqual([
+      'always/jest.integration.config.js||shard=1/2',
+      'always/jest.integration.config.js||shard=2/2',
+    ]);
+  });
+
+  it('returns all configs unchanged when a critical file changed', () => {
+    const configs = ['other/jest.integration.config.js'];
+
+    const result = filterJestIntegrationConfigsByAffected(configs, context([], ['CRITICAL_INT']));
+
+    expect(result).toEqual(configs);
+  });
+});
+
+describe('filterJestUnitConfigsByAffected', () => {
+  it('does not force always-run integration configs into unit runs', () => {
+    const configs = ['always/jest.integration.config.js', 'affected/jest.config.js'];
+
+    const result = filterJestUnitConfigsByAffected(configs, context(['affected/']));
+
+    expect(result).toEqual(['affected/jest.config.js']);
+  });
+});

.buildkite/pipeline-utils/ci-stats/pick_test_group_run_order/selective_testing.ts

@@ -8,6 +8,7 @@
  */
 
 import {
+  ALWAYS_RUN_JEST_INTEGRATION_CONFIGS,
   CRITICAL_FILES_JEST_INTEGRATION_TESTS,
   CRITICAL_FILES_JEST_UNIT_TESTS,
   filterFilesByPackages,
@@ -17,6 +18,7 @@ import {
 } from '../../affected-packages';
 
 import { expandJestImplicitConsumers } from './jest_implicit_consumers';
+import { SHARD_ANNOTATION_SEP } from './jest_configs';
 
 /**
  * The shared inputs both per-variant filters need: which packages the PR
@@ -73,7 +75,7 @@ export function filterJestUnitConfigsByAffected(
   });
 }
 
-/** Narrow Jest integration configs to those owned by affected packages, unless a critical file changed. */
+/** Like the unit filter, but always re-adds ALWAYS_RUN_JEST_INTEGRATION_CONFIGS. */
 export function filterJestIntegrationConfigsByAffected(
   jestIntegrationConfigs: string[],
   context: SelectiveTestingContext
@@ -82,6 +84,7 @@ export function filterJestIntegrationConfigsByAffected(
     label: 'integration',
     configs: jestIntegrationConfigs,
     criticalFiles: CRITICAL_FILES_JEST_INTEGRATION_TESTS,
+    alwaysRun: ALWAYS_RUN_JEST_INTEGRATION_CONFIGS,
     context,
   });
 }
@@ -90,16 +93,44 @@ function filterByAffected(args: {
   label: 'unit' | 'integration';
   configs: string[];
   criticalFiles: string[];
+  alwaysRun?: readonly string[];
   context: SelectiveTestingContext;
 }): string[] {
-  const { label, configs, criticalFiles, context } = args;
+  const { label, configs, criticalFiles, alwaysRun = [], context } = args;
 
   if (touchedCriticalFiles(context.prChangedFiles, criticalFiles)) {
     console.log(`Not filtering Jest ${label} tests because critical files changed`);
     return configs;
   }
 
   const filtered = filterFilesByPackages(configs, context.affectedPackages);
-  console.log(`Filtering Jest ${label} tests: ${configs.length} -> ${filtered.length}`);
-  return filtered;
+  const withAlwaysRun = addAlwaysRunConfigs(filtered, configs, alwaysRun);
+  console.log(`Filtering Jest ${label} tests: ${configs.length} -> ${withAlwaysRun.length}`);
+  return withAlwaysRun;
+}
+
+// Matches on the base path so every shard of an always-run config is restored.
+function addAlwaysRunConfigs(
+  filtered: string[],
+  allConfigs: string[],
+  alwaysRun: readonly string[]
+): string[] {
+  if (alwaysRun.length === 0) {
+    return filtered;
+  }
+
+  const alwaysRunSet = new Set(alwaysRun);
+  const result = new Set(filtered);
+  for (const config of allConfigs) {
+    if (alwaysRunSet.has(baseConfigPath(config)) && !result.has(config)) {
+      result.add(config);
+      console.log(`Always-run Jest integration config re-added: ${config}`);
+    }
+  }
+  return [...result];
+}
+
+function baseConfigPath(config: string): string {
+  const idx = config.indexOf(SHARD_ANNOTATION_SEP);
+  return idx === -1 ? config : config.slice(0, idx);
 }

.buildkite/scripts/steps/security/third_party_packages.txt

@@ -1,3 +1,4 @@
+eslint-plugin-n
 @xterm/xterm
 knip
 safe-stable-stringify
@@ -58,4 +59,5 @@ redux-thunk-v2
 redux-toolkit-v1
 redux-v4
 reselect-v4
+@swc/plugin-emotion
 graphql

.eslintrc.js

@@ -1523,7 +1523,7 @@ module.exports = {
         'x-pack/solutions/security/packages/data-stream-adapter/**/*.{js,mjs,ts,tsx}',
         'src/platform/packages/shared/kbn-cell-actions/**/*.{js,mjs,ts,tsx}',
       ],
-      plugins: ['eslint-plugin-node', 'react'],
+      plugins: ['eslint-plugin-n', 'react'],
       env: {
         jest: true,
       },
@@ -1532,7 +1532,7 @@ module.exports = {
         'array-callback-return': 'error',
         'no-array-constructor': 'error',
         complexity: 'warn',
-        'node/no-deprecated-api': 'error',
+        'n/no-deprecated-api': 'error',
         'no-bitwise': 'error',
         'no-continue': 'error',
         'no-dupe-keys': 'error',
@@ -1824,7 +1824,7 @@ module.exports = {
     {
       // typescript and javascript for front and back
       files: ['x-pack/solutions/security/plugins/lists/**/*.{js,mjs,ts,tsx}'],
-      plugins: ['eslint-plugin-node'],
+      plugins: ['eslint-plugin-n'],
       env: {
         jest: true,
       },
@@ -1848,7 +1848,7 @@ module.exports = {
             ignoreDeclarationSort: true,
           },
         ],
-        'node/no-deprecated-api': 'error',
+        'n/no-deprecated-api': 'error',
         'no-bitwise': 'error',
         'no-continue': 'error',
         'no-dupe-keys': 'error',
@@ -2540,7 +2540,7 @@ module.exports = {
         'src/platform/packages/shared/kbn-workflows/**/*.{js,mjs,ts,tsx}',
         'src/platform/packages/shared/kbn-workflows-ui/**/*.{js,mjs,ts,tsx}',
       ],
-      plugins: ['eslint-plugin-node', 'react'],
+      plugins: ['eslint-plugin-n', 'react'],
       env: {
         jest: true,
       },
@@ -2549,7 +2549,7 @@ module.exports = {
         'array-callback-return': 'error',
         'no-array-constructor': 'error',
         complexity: 'warn',
-        'node/no-deprecated-api': 'error',
+        'n/no-deprecated-api': 'error',
         'no-bitwise': 'error',
         'no-continue': 'error',
         'no-dupe-keys': 'error',