[Security Solution] Strip tools from alert analysis classification agent

[stephmilovic]

Summary

This PR hard-strips the alert analysis classification agent's tool and skill belt by setting configuration_overrides on runAgent_step (enable_elastic_capabilities: false, empty tools and skill_ids).

configuration_overrides was merged into the agent builder steps weeks after the alert analysis workflow shipped. Because the agent's context is pre-built by earlier steps, this agent should never have been able to invoke tools. The previous protection was a prompt-only instruction ("Don't use any tools") which was an unnecessary artifact that introduced possible model variance. This PR drops that redundant prompt line and enforces the restriction at the framework level, shrinking the attack surface on untrusted alert fields.

consts.workflow_version is bumped to v0.0.3. The managed definition version is unchanged (the yaml content hash still auto-upgrades installs).

Local eval results

Suite security-alert-analysis-workflow, 5 reps × 8 examples, on the current batched workflow. Judge: Gemini 3.1 Pro.

Model	Phase	ClassificationAccuracy	ValidVerdict	criteria	inputTokens median
Sonnet 4.6 (MUT)	before	1.00	1.00	0.73	23415
Sonnet 4.6 (MUT)	after	0.97	0.97	0.73	23783
Gemini 3.1 Pro	before	1.00	1.00	0.72	19343
Gemini 3.1 Pro	after	1.00	1.00	0.68	11388

Eval notes:

• Sonnet accuracy: Not a wrong TP/FP. One run returned a corrupted alert id (995f instead of 95f), scoring no verdict. The same example was correct on the other 4 reps.
• Tokens: Gemini median input dropped ~41%. Sonnet median stayed roughly flat (the remaining framework prompt dominates).
• Tool usage: Baseline tracing confirmed the model already was not executing tools, so this only removes unused surface area.

To test

1. Install or upgrade managed system-security-alert-analysis so it picks up the new yaml hash.
2. Confirm installed yaml has configuration_overrides on runAgent_step and workflow_version: "v0.0.3".
3. Run a classification on a sample alert and confirm a normal TP/FP verdict note still lands.

Screenshots: not applicable (no UI change).

PR developed with Cursor + Gemini 3.1 Pro

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-solution (Team: SecuritySolution)

[jonwalstedt]

Nice!

[stephmilovic]

@elasticmachine merge upstream

[kibanamachine]

PR run: bk-01a09293-6c96-4589-9975-4fc1edd61a38::security-alert-analysis-workflow::anthropic-claude-4.6-opus | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-analysis-workflow::anthropic-claude-4.6-opus
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

Note: 16 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)

Dataset	Evaluator	N	Mean (PR)	Mean (main)	Diff	p-value	Sig	Outcome
security: alert-analysis-workflow-classification	ClassificationAccuracy	8	1.00	1.00	0.00	1.00	No	-
security: alert-analysis-workflow-classification	criteria	8	0.67	0.67	0.00	1.00	No	-
security: alert-analysis-workflow-classification	ValidVerdict	8	1.00	1.00	0.00	1.00	No	-

PR run: bk-01a09293-6c96-4589-9975-4fc1edd61a38::security-alert-analysis-workflow::anthropic-claude-4.6-sonnet | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-analysis-workflow::anthropic-claude-4.6-sonnet
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

Note: 16 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)

Dataset	Evaluator	N	Mean (PR)	Mean (main)	Diff	p-value	Sig	Outcome
security: alert-analysis-workflow-classification	ClassificationAccuracy	8	1.00	1.00	0.00	1.00	No	-
security: alert-analysis-workflow-classification	criteria	8	0.71	0.67	+0.04	0.32	No	-
security: alert-analysis-workflow-classification	ValidVerdict	8	1.00	1.00	0.00	1.00	No	-

PR run: bk-01a09293-6c96-4589-9975-4fc1edd61a38::security-alert-analysis-workflow::google-gemini-3.0-flash | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-analysis-workflow::google-gemini-3.0-flash
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

Note: 16 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)

Dataset	Evaluator	N	Mean (PR)	Mean (main)	Diff	p-value	Sig	Outcome
security: alert-analysis-workflow-classification	ClassificationAccuracy	8	1.00	1.00	0.00	1.00	No	-
security: alert-analysis-workflow-classification	criteria	8	0.75	0.71	+0.04	0.32	No	-
security: alert-analysis-workflow-classification	ValidVerdict	8	1.00	1.00	0.00	1.00	No	-

PR run: bk-01a09293-6c96-4589-9975-4fc1edd61a38::security-alert-analysis-workflow::google-gemini-3.1-pro | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-analysis-workflow::google-gemini-3.1-pro
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

Note: 16 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)

Dataset	Evaluator	N	Mean (PR)	Mean (main)	Diff	p-value	Sig	Outcome
security: alert-analysis-workflow-classification	ClassificationAccuracy	8	1.00	1.00	0.00	1.00	No	-
security: alert-analysis-workflow-classification	criteria	8	0.67	0.63	+0.04	0.32	No	-
security: alert-analysis-workflow-classification	ValidVerdict	8	1.00	1.00	0.00	1.00	No	-

PR run: bk-01a09293-6c96-4589-9975-4fc1edd61a38::security-alert-analysis-workflow::openai-gpt-5.4 | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-analysis-workflow::openai-gpt-5.4
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

Note: 16 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)

Dataset	Evaluator	N	Mean (PR)	Mean (main)	Diff	p-value	Sig	Outcome
security: alert-analysis-workflow-classification	ClassificationAccuracy	8	1.00	1.00	0.00	1.00	No	-
security: alert-analysis-workflow-classification	criteria	8	0.71	0.79	-0.08	0.32	No	-
security: alert-analysis-workflow-classification	ValidVerdict	8	1.00	1.00	0.00	1.00	No	-

PR run: bk-01a09293-6c96-4589-9975-4fc1edd61a38::security-alert-analysis-workflow::openai-gpt-oss-120b | Baseline (main): bk-01a07328-2f89-4201-93fc-e763ec8c96e7::security-alert-analysis-workflow::openai-gpt-oss-120b
Baseline: commit 0f9d1e1, 6 days ago

Warning: Baseline is 6 days old. Results may not reflect current main.
Significance threshold: p < 0.05

Summary
No significant regressions detected (3 evaluator comparisons).

Note: 16 null scores skipped. Comparison may be incomplete — the baseline could be a partial run.

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (3 rows)

Dataset	Evaluator	N	Mean (PR)	Mean (main)	Diff	p-value	Sig	Outcome
security: alert-analysis-workflow-classification	ClassificationAccuracy	8	1.00	1.00	0.00	1.00	No	-
security: alert-analysis-workflow-classification	criteria	8	0.63	0.67	-0.04	0.32	No	-
security: alert-analysis-workflow-classification	ValidVerdict	8	1.00	1.00	0.00	1.00	No	-

[kibanamachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: b6ce330
• Build duration: 69 mins

Failed CI Steps

• FTR Configs #72
• FTR Configs #141

Metrics [docs]

✅ unchanged

Test Failures

• [job] [logs] FTR Configs #72 / X-Pack Accessibility Tests - Group 2 Lens Accessibility lens XY chart with annotations layer
• [job] [logs] FTR Configs #72 / X-Pack Accessibility Tests - Group 2 Lens Accessibility lens XY chart with multiple layers
• [job] [logs] FTR Configs #72 / X-Pack Accessibility Tests - Group 2 Lens Accessibility lens XY chart with reference line layer
• [job] [logs] FTR Configs #72 / X-Pack Accessibility Tests - Group 2 Lens Accessibility saves lens chart

History

• 💛 Build #499368 was flaky bb190e8