[Security Solution] defend policy management skill by joeypoon · Pull Request #289633 · elastic/kibana

joeypoon · 2026-09-07T14:26:54Z

Summary

Introduces a read-only Elastic Defend package policy management skill.
Capabilities include:

listing policies and explaining what a setting does
OS-specific tuning guidance grounded in KB docs
comparing policies for drift, duplicates, and unused policies
reporting assigned-versus-applied counts
assessing the impact of a proposed change
recommending staged rollout steps

Feature gated behind a new elasticDefendPolicyManagementSkill feature flag.

High-level supporting changes

Policy analysis and rollout foundations

Adds normalized policy snapshots, configuration hashing and diffing, field-reference metadata, and policy-change expansion and eligibility checks.
Adds request-scoped read-only access for policy, assignment, endpoint-count, and rollout-status data.
Bounds and trims tool responses to keep model context predictable.

Shared Elastic Defend policy semantics

Consolidates policy configuration helpers and schemas so the UI, server tools, and evaluations use consistent setting behavior.
Reuses shared latest-policy-response and global-manifest logic for rollout reporting and troubleshooting.
Clarifies the routing boundary between policy decisions and endpoint operational troubleshooting.
Existing withEndpointAuthz is intentionally not unified as to not risk modifying existing authz inside of this large unrelated PR.

CI evals intentionally skipped as these will not pass since they lack the required KB docs to be introduced in elastic/endpoint-package#796.

Local eval results w/ KB docs manually populated:

Model: anthropic-claude-5-sonnet (Claude/Elastic)
Evaluator Model: openai-gpt-5.5 (Claude/Elastic)
[scout-worker]
═══ EVALUATION RESULTS ═══
╔═══════════════════════════════════════════════════════╤═══════╤═══════════╤════════════════╤══════════════════════════════╤══════════════╤══════════════════╗
║ Dataset                                               │     # │  Criteria │        Latency │ Policy Management Tool Usage │   Tool Calls │           Tokens ║
╟───────────────────────────────────────────────────────┼───────┼───────────┼────────────────┼──────────────────────────────┼──────────────┼──────────────────╢
║ security: policy-management-detect-to-prevent-rollout │ 3 x 1 │   mean: 1 │    mean: 7.93s │                      mean: 1 │    mean: 7.3 │     Input Tokens ║
║                                                       │       │ median: 1 │  median: 6.42s │                    median: 1 │    median: 8 │     mean: 116206 ║
║                                                       │       │    std: 0 │     std: 3.00s │                       std: 0 │       min: 6 │   median: 116204 ║
║                                                       │       │    min: 1 │     min: 5.25s │                       min: 1 │       max: 8 │         std: 4.3 ║
║                                                       │       │    max: 1 │    max: 12.12s │                       max: 1 │              │      min: 116202 ║
║                                                       │       │           │                │                              │              │      max: 116212 ║
║                                                       │       │           │                │                              │              │ ──────────────── ║
║                                                       │       │           │                │                              │              │    Output Tokens ║
║                                                       │       │           │                │                              │              │     mean: 1659.3 ║
║                                                       │       │           │                │                              │              │     median: 1292 ║
║                                                       │       │           │                │                              │              │       std: 596.0 ║
║                                                       │       │           │                │                              │              │        min: 1186 ║
║                                                       │       │           │                │                              │              │        max: 2500 ║
╟───────────────────────────────────────────────────────┼───────┼───────────┼────────────────┼──────────────────────────────┼──────────────┼──────────────────╢
║ security: policy-management-assigned-vs-applied       │ 3 x 1 │   mean: 1 │    mean: 2.97s │                      mean: 1 │      mean: 4 │     Input Tokens ║
║                                                       │       │ median: 1 │  median: 2.99s │                    median: 1 │    median: 4 │     mean: 116206 ║
║                                                       │       │    std: 0 │     std: 0.05s │                       std: 0 │       min: 4 │   median: 116212 ║
║                                                       │       │    min: 1 │     min: 2.89s │                       min: 1 │       max: 4 │         std: 8.5 ║
║                                                       │       │    max: 1 │     max: 3.02s │                       max: 1 │              │      min: 116194 ║
║                                                       │       │           │                │                              │              │      max: 116212 ║
║                                                       │       │           │                │                              │              │ ──────────────── ║
║                                                       │       │           │                │                              │              │    Output Tokens ║
║                                                       │       │           │                │                              │              │      mean: 542.7 ║
║                                                       │       │           │                │                              │              │      median: 518 ║
║                                                       │       │           │                │                              │              │        std: 40.7 ║
║                                                       │       │           │                │                              │              │         min: 510 ║
║                                                       │       │           │                │                              │              │         max: 600 ║
╟───────────────────────────────────────────────────────┼───────┼───────────┼────────────────┼──────────────────────────────┼──────────────┼──────────────────╢
║ security: policy-management-estate-cleanup            │ 3 x 1 │   mean: 1 │   mean: 20.85s │                      mean: 1 │     mean: 18 │     Input Tokens ║
║                                                       │       │ median: 1 │ median: 19.98s │                    median: 1 │   median: 18 │   mean: 237255.3 ║
║                                                       │       │    std: 0 │     std: 2.41s │                       std: 0 │      min: 16 │   median: 260754 ║
║                                                       │       │    min: 1 │    min: 18.44s │                       min: 1 │      max: 20 │     std: 37000.7 ║
║                                                       │       │    max: 1 │    max: 24.14s │                       max: 1 │              │      min: 185016 ║
║                                                       │       │           │                │                              │              │      max: 265996 ║
║                                                       │       │           │                │                              │              │ ──────────────── ║
║                                                       │       │           │                │                              │              │    Output Tokens ║
║                                                       │       │           │                │                              │              │       mean: 6700 ║
║                                                       │       │           │                │                              │              │     median: 6106 ║
║                                                       │       │           │                │                              │              │       std: 874.2 ║
║                                                       │       │           │                │                              │              │        min: 6058 ║
║                                                       │       │           │                │                              │              │        max: 7936 ║
╟───────────────────────────────────────────────────────┼───────┼───────────┼────────────────┼──────────────────────────────┼──────────────┼──────────────────╢
║ security: policy-management-os-tuning                 │ 3 x 1 │   mean: 1 │   mean: 18.77s │                      mean: 1 │   mean: 19.3 │     Input Tokens ║
║                                                       │       │ median: 1 │  median: 5.23s │                    median: 1 │   median: 20 │     mean: 303855 ║
║                                                       │       │    std: 0 │    std: 19.47s │                       std: 0 │      min: 18 │   median: 254342 ║
║                                                       │       │    min: 1 │     min: 4.77s │                       min: 1 │      max: 20 │     std: 70055.9 ║
║                                                       │       │    max: 1 │    max: 46.30s │                       max: 1 │              │      min: 254294 ║
║                                                       │       │           │                │                              │              │      max: 402929 ║
║                                                       │       │           │                │                              │              │ ──────────────── ║
║                                                       │       │           │                │                              │              │    Output Tokens ║
║                                                       │       │           │                │                              │              │     mean: 2836.7 ║
║                                                       │       │           │                │                              │              │     median: 2160 ║
║                                                       │       │           │                │                              │              │      std: 1180.0 ║
║                                                       │       │           │                │                              │              │        min: 1854 ║
║                                                       │       │           │                │                              │              │        max: 4496 ║
║                                                       │       │           │                │                              │              │ ──────────────── ║
║                                                       │       │           │                │                              │              │    Cached Tokens ║
║                                                       │       │           │                │                              │              │      mean: 44811 ║
║                                                       │       │           │                │                              │              │    median: 44811 ║
║                                                       │       │           │                │                              │              │           std: 0 ║
║                                                       │       │           │                │                              │              │       min: 44811 ║
║                                                       │       │           │                │                              │              │       max: 44811 ║
╟───────────────────────────────────────────────────────┼───────┼───────────┼────────────────┼──────────────────────────────┼──────────────┼──────────────────╢
║ Overall                                               │ 3 x 4 │   mean: 1 │   mean: 12.63s │                      mean: 1 │   mean: 12.2 │     Input Tokens ║
║                                                       │       │ median: 1 │ median: 12.63s │                    median: 1 │ median: 12.2 │   mean: 193380.6 ║
║                                                       │       │    std: 0 │    std: 11.48s │                       std: 0 │       min: 4 │ median: 193380.6 ║
║                                                       │       │    min: 1 │     min: 2.89s │                       min: 1 │      max: 20 │     std: 90793.5 ║
║                                                       │       │    max: 1 │    max: 46.30s │                       max: 1 │              │      min: 116194 ║
║                                                       │       │           │                │                              │              │      max: 402929 ║
║                                                       │       │           │                │                              │              │ ──────────────── ║
║                                                       │       │           │                │                              │              │    Output Tokens ║
║                                                       │       │           │                │                              │              │     mean: 2934.7 ║
║                                                       │       │           │                │                              │              │   median: 2934.7 ║
║                                                       │       │           │                │                              │              │      std: 2516.0 ║
║                                                       │       │           │                │                              │              │         min: 510 ║
║                                                       │       │           │                │                              │              │        max: 7936 ║
║                                                       │       │           │                │                              │              │ ──────────────── ║
║                                                       │       │           │                │                              │              │    Cached Tokens ║
║                                                       │       │           │                │                              │              │      mean: 44811 ║
║                                                       │       │           │                │                              │              │    median: 44811 ║
║                                                       │       │           │                │                              │              │           std: 0 ║
║                                                       │       │           │                │                              │              │           min: 0 ║
║                                                       │       │           │                │                              │              │       max: 44811 ║
╚═══════════════════════════════════════════════════════╧═══════╧═══════════╧════════════════╧══════════════════════════════╧══════════════╧══════════════════╝

Model: anthropic-claude-5-sonnet (Claude/Elastic)
[scout-worker]
═══ EVALUATION RESULTS ═══
╔═════════════════════════════════════════════════════════════════════════╤════════╤═════════════════╗
║ Dataset                                                                 │      # │ Skill Selection ║
╟─────────────────────────────────────────────────────────────────────────┼────────┼─────────────────╢
║ skill-selection-benchmark: elastic-defend-policy-management             │  3 x 6 │         mean: 1 ║
║                                                                         │        │       median: 1 ║
║                                                                         │        │          std: 0 ║
║                                                                         │        │          min: 1 ║
║                                                                         │        │          max: 1 ║
╟─────────────────────────────────────────────────────────────────────────┼────────┼─────────────────╢
║ skill-selection-benchmark: elastic-defend-configuration-troubleshooting │  3 x 5 │         mean: 1 ║
║                                                                         │        │       median: 1 ║
║                                                                         │        │          std: 0 ║
║                                                                         │        │          min: 1 ║
║                                                                         │        │          max: 1 ║
╟─────────────────────────────────────────────────────────────────────────┼────────┼─────────────────╢
║ Overall                                                                 │ 3 x 11 │         mean: 1 ║
║                                                                         │        │       median: 1 ║
║                                                                         │        │          std: 0 ║
║                                                                         │        │          min: 1 ║
║                                                                         │        │          max: 1 ║
╚═════════════════════════════════════════════════════════════════════════╧════════╧═════════════════╝

Example:

Checklist

Check the PR satisfies following conditions.

Unit or functional tests were updated or added to match the most common scenarios

infra-vault-gh-plugin-prod · 2026-09-07T14:27:09Z

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

Click to trigger kibana-pull-request for this PR!
Click to trigger kibana-deploy-project-from-pr for this PR!
Click to trigger kibana-deploy-cloud-from-pr for this PR!
Click to trigger kibana-entity-store-performance-from-pr for this PR!
Click to trigger kibana-storybooks-from-pr for this PR!

Introduces a read-only Elastic Defend package policy management skill. Capabilities include: - listing policies and explaining what a setting does - OS-specific tuning guidance grounded in KB docs - comparing policies for drift, duplicates, and unused policies - reporting assigned-versus-applied counts - assessing the impact of a proposed change - recommending staged rollout steps

kibanamachine · 2026-09-12T18:54:07Z

💔 Build Failed

Buildkite Build
Commit: e76a01a
Build duration: 21 mins

Failed CI Steps

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`shared-plugins`	3875	3876	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	1016.0KB	1013.8KB	-2.3KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`shared-plugins`	12.1MB	12.1MB	+1.9KB

Unknown metric groups

ESLint disabled in files

id	before	after	diff
`securitySolution`	104	105	+1

shared chunks total size

id	before	after	diff
`all`	7.0MB	7.0MB	-2.3KB

Total ESLint disabled count

id	before	after	diff
`securitySolution`	869	870	+1

total optimizer output size

id	before	after	diff
`all`	64.0MB	64.0MB	-2.6KB

History

💛 Build #500702 was flaky c5130e5
💔 Build #500678 failed 2f10acb
💔 Build #500673 failed 865a589

joeypoon added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting Team:Defend Workflows “EDR Workflows” sub-team of Security Solution labels Sep 7, 2026

joeypoon force-pushed the feature/policy-management-skill branch 4 times, most recently from 2f10acb to c5130e5 Compare September 12, 2026 03:04

joeypoon force-pushed the feature/policy-management-skill branch from c5130e5 to e76a01a Compare September 12, 2026 18:40

Changes from node scripts/eslint_all_files --no-cache --fix

6af29b6

Navigation Menu

Uh oh!

[Security Solution] defend policy management skill - #289633

[Security Solution] defend policy management skill#289633
joeypoon wants to merge 2 commits into
elastic:mainfrom
joeypoon:feature/policy-management-skill

joeypoon commented Sep 7, 2026 •

edited

Loading

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 7, 2026 •

edited

Loading

Uh oh!

kibanamachine commented Sep 12, 2026 •

edited

Loading

ESLint disabled in files

shared chunks total size

Total ESLint disabled count

total optimizer output size

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

joeypoon commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

High-level supporting changes

Policy analysis and rollout foundations

Shared Elastic Defend policy semantics

Checklist

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kibanamachine commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Metrics [docs]

Module Count

Async chunks

Page load bundle

ESLint disabled in files

shared chunks total size

Total ESLint disabled count

total optimizer output size

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joeypoon commented Sep 7, 2026 •

edited

Loading

infra-vault-gh-plugin-prod Bot commented Sep 7, 2026 •

edited

Loading

kibanamachine commented Sep 12, 2026 •

edited

Loading