fix(kbn-evals): count load_skill invocations in the skill gate - #289968
Draft
patrykkopycinski wants to merge 1 commit into
Draft
fix(kbn-evals): count load_skill invocations in the skill gate#289968patrykkopycinski wants to merge 1 commit into
patrykkopycinski wants to merge 1 commit into
Conversation
`createSkillInvocationEvaluator` only matched `filestore.read` calls whose
arguments contained `/<skill>/SKILL.md`. Agent Builder activates a skill by
calling `load_skill` with a `{"skill":"<name>"}` argument, so the gate scored
0 for every model on every run.
Golden shows the effect across 415 observations of the parameterised
`Skill Invoked (...)` instances: not one scored 1. The same models score 1 on
the persona-matrix `SkillInvoked` evaluator, which already matches both tool
names and both argument shapes -- claude-5-sonnet scores 1 on all 9 bare
checks and 0 on the parameterised one in the same run.
Match what that evaluator matches: tool name IN ("load_skill",
"filestore.read") AND either argument shape.
|
🤖 Jobs for this PR can be triggered through checkboxes. 🚧
ℹ️ To trigger the CI, please tick the checkbox below 👇
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is the same fix as #284661, applied to the parameterised evaluator that was missed.
#284661 (merged 2026-08-12) fixed the bare
SkillInvokedevaluator inevaluate_dataset.ts:The parameterised
Skill Invoked (<name>)evaluator inskill_invocation.tscarries the same defect and was not covered by that PR. This applies the same predicate change there.Evidence
#284661 measured the tool rename directly against the golden cluster over a 7-day window:
filestore.readload_skillAgent Builder gained the dedicated
load_skilltool in #270159 (2026-05-22);filestore.readis retired. Becausetotal_tool_spansis still non-zero on these traces, the "no tool spans →unavailable" guard does not trip, so the evaluator returns a confidentscore: 0rather thannull.The score history matches the retirement, and shows the bare evaluator recovering after August while the parameterised one does not:
In
security-persona-matrixthe parameterised check passes 4 times in 3,418 observations (0.1%), all 4 dated July. Six of its seven instances have never scored 1:The bare evaluator in that same suite scores 2,146 ones out of 5,316. Two evaluators, same traces, opposite results — the difference is the predicate, not the models.
As #284661 put it: the reported zero is a measurement artifact, not a model capability result. It silently drags every model's aggregate down and reads as a model failure.
Fix
Align the parameterised query with the bare one — accept both
load_skillandfilestore.read, and both argument shapes.Testing
detects a skill loaded through load_skill— verified RED before the fix.filestore.read-only behaviour; updated to assert correct behaviour.13/13tests pass inskill_invocation.test.ts, eslint clean,tscexit 0.What is not verified here
The unit tests prove the predicate matches the fixtures in this PR; they do not re-prove the span shape. That evidence comes from #284661's golden-cluster span counts above. The golden cluster currently holds scores (2,073,547 docs) but no trace indices (
traces-*returnscount: 0,shards: 0), so an end-to-end re-confirmation needs a live eval run with tool-detail tracing enabled.Branch is off clean
upstream/main, verified there rather than on a feature branch, since that is where it has to land.