Skip to content

fix(kbn-evals): count load_skill invocations in the skill gate - #289968

Draft
patrykkopycinski wants to merge 1 commit into
elastic:mainfrom
patrykkopycinski:fix/skill-invocation-load-skill
Draft

fix(kbn-evals): count load_skill invocations in the skill gate#289968
patrykkopycinski wants to merge 1 commit into
elastic:mainfrom
patrykkopycinski:fix/skill-invocation-load-skill

Conversation

@patrykkopycinski

@patrykkopycinski patrykkopycinski commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Summary

This is the same fix as #284661, applied to the parameterised evaluator that was missed.

#284661 (merged 2026-08-12) fixed the bare SkillInvoked evaluator in evaluate_dataset.ts:

-  attributes.gen_ai.tool.name == "filestore.read" AND (${skillPredicate}),
+  attributes.gen_ai.tool.name IN ("load_skill", "filestore.read") AND (${skillPredicate}),

The parameterised Skill Invoked (<name>) evaluator in skill_invocation.ts carries the same defect and was not covered by that PR. This applies the same predicate change there.

Evidence

#284661 measured the tool rename directly against the golden cluster over a 7-day window:

tool name spans (7d)
filestore.read 0
load_skill 7,991

Agent Builder gained the dedicated load_skill tool in #270159 (2026-05-22); filestore.read is retired. Because total_tool_spans is still non-zero on these traces, the "no tool spans → unavailable" guard does not trip, so the evaluator returns a confident score: 0 rather than null.

The score history matches the retirement, and shows the bare evaluator recovering after August while the parameterised one does not:

Month Observations Scored 1 Pass rate
2026-06 26,731 9,547 40.9%
2026-07 36,064 12,112 45.6%
2026-08 9,751 1,541 15.8%
2026-09 1,840 56 3.0%

In security-persona-matrix the parameterised check passes 4 times in 3,418 observations (0.1%), all 4 dated July. Six of its seven instances have never scored 1:

Skill Invoked (alert-analysis)          n=489  1s=4
Skill Invoked (detection-rule-creation) n=489  1s=0
Skill Invoked (entity-analytics)        n=488  1s=0
Skill Invoked (security-multi-step)     n=488  1s=0
Skill Invoked (security-tools)          n=488  1s=0
Skill Invoked (threat-intel-hunt)       n=488  1s=0
Skill Invoked (workflow-authoring)      n=488  1s=0

The bare evaluator in that same suite scores 2,146 ones out of 5,316. Two evaluators, same traces, opposite results — the difference is the predicate, not the models.

As #284661 put it: the reported zero is a measurement artifact, not a model capability result. It silently drags every model's aggregate down and reads as a model failure.

Fix

Align the parameterised query with the bare one — accept both load_skill and filestore.read, and both argument shapes.

Testing

  • New test detects a skill loaded through load_skill — verified RED before the fix.
  • One existing test asserted the buggy filestore.read-only behaviour; updated to assert correct behaviour.
  • 13/13 tests pass in skill_invocation.test.ts, eslint clean, tsc exit 0.
  • Mutation-tested: 3/3 mutations caught.

What is not verified here

The unit tests prove the predicate matches the fixtures in this PR; they do not re-prove the span shape. That evidence comes from #284661's golden-cluster span counts above. The golden cluster currently holds scores (2,073,547 docs) but no trace indices (traces-* returns count: 0, shards: 0), so an end-to-end re-confirmation needs a live eval run with tool-detail tracing enabled.

Branch is off clean upstream/main, verified there rather than on a feature branch, since that is where it has to land.

`createSkillInvocationEvaluator` only matched `filestore.read` calls whose
arguments contained `/<skill>/SKILL.md`. Agent Builder activates a skill by
calling `load_skill` with a `{"skill":"<name>"}` argument, so the gate scored
0 for every model on every run.

Golden shows the effect across 415 observations of the parameterised
`Skill Invoked (...)` instances: not one scored 1. The same models score 1 on
the persona-matrix `SkillInvoked` evaluator, which already matches both tool
names and both argument shapes -- claude-5-sonnet scores 1 on all 9 bare
checks and 0 on the parameterised one in the same run.

Match what that evaluator matches: tool name IN ("load_skill",
"filestore.read") AND either argument shape.
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown
🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

  • Click to trigger kibana-pull-request for this PR!
  • Click to trigger kibana-deploy-project-from-pr for this PR!
  • Click to trigger kibana-deploy-cloud-from-pr for this PR!
  • Click to trigger kibana-entity-store-performance-from-pr for this PR!
  • Click to trigger kibana-storybooks-from-pr for this PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant