[Feature]: skills_filter step is opaque — filtered_skill_ids records which skills survived but not why any were dropped #54

Problem

Summary

The filter_skills pipeline step (src/opensquilla/engine/steps/skills_filter.py) silently drops skills at four different gates plus a retrieval cut, but only emits a single flat list of survivors. Operators have no way to tell why a skill is missing — eligibility check? missing required tool? lost the retrieval
ranking? — short of reading the source and re-running with extra logging. The field name (filtered_skill_ids) is also ambiguous: it sounds like "filtered out" but actually means "kept after filtering".

Repro

Any turn writes a row like this to /logs/decisions-YYYYMMDD.jsonl:

"pipeline_steps": [
...,
{
"step_name": "filter_skills",
"applied": true,
"filtered_skill_ids": [
"coding-agent","cron","deep-research","docx","html-to-pdf",
"memory","multi-search-engine","pdf-toolkit","pptx",
"skill-creator","summarize","tmux","weather","xlsx","agent-browser"
],
...
}
]

Questions a user cannot answer from this log:

1. How many skills did the loader return originally? (only the survivor count is recoverable, from skill_count).
2. Of the skills not listed, which were rejected by _deterministic_gate vs. dropped by the retriever's top_k?
3. For a deterministic drop, which of the four reasons applied:
   - disable_model_invocation
   - check_eligibility failed
   - requires_tools not satisfied (and which tool was missing)
   - fallback_for_toolsets superseded by a real tool
4. For a retrieval drop, what was the skill's score / rank, and which strategy (lexical / semantic / hybrid) produced it?

Why this matters

filter_skills is one of the headline token-saving mechanisms of OpenSquilla. When a user reports "skill X didn't fire", today the only debugging path is:

• Set log level to DEBUG and re-run (loses the original run state),
• Or open skills_filter.py and add print statements.

This also blocks legitimate operational work: there is no way to compute a "skill activation rate" or detect a misconfigured requires_tools declaration causing a popular skill to be silently shadowed across every turn.

Source pointers

src/opensquilla/engine/steps/skills_filter.py:57-73 — _deterministic_gate drops with no record:

for s in skills:
if s.disable_model_invocation:
continue
if not check_eligibility(s, _elig_ctx):
continue
if s.requires_tools and not all(t in available_tools for t in s.requires_tools):
continue
if s.fallback_for_toolsets and any(t in available_tools for t in s.fallback_for_toolsets):
continue
gated.append(s)

src/opensquilla/engine/steps/skills_filter.py:112-119 — retriever drops with no per-skill score returned:

if filter_enabled:
top_k = getattr(skills_cfg, "filter_top_k", 5)
retriever = _get_retriever(skills_cfg)
filtered = retriever.retrieve(filterable, semantic_message, top_k=top_k)

src/opensquilla/observability/decision_log.py:94 — schema only carries the survivor list:

filtered_skill_ids: list[str] | None = None

Proposed behavior

Replace filtered_skill_ids: list[str] with a structured per-skill record, and rename it for clarity. Backwards-compat can be preserved by keeping the old field as a derived view for one minor version.

observability/decision_log.py
class SkillDecision(BaseModel):
id: str
kept: bool
reason: Literal[
"kept_pinned", # skill.always == True
"kept_retrieved", # survived retriever top_k
"kept_no_filter", # filter_enabled == False
"dropped_model_invocation_disabled",
"dropped_ineligible",
"dropped_missing_required_tool",
"dropped_superseded_by_toolset",
"dropped_below_top_k",
]
missing_tools: list[str] | None = None # for dropped_missing_required_tool
score: float | None = None # for retriever decisions
rank: int | None = None

class PipelineStepRecord(BaseModel):
...
skill_decisions: list[SkillDecision] | None = None
# Kept temporarily; derive from skill_decisions where kept=True:
filtered_skill_ids: list[str] | None = None

In skills_filter.py, collect a decision per input skill instead of just appending to gated/filtered. The retriever needs a small API change to return scores (retrieve(...) -> list[tuple[SkillSpec, float]]), which is also useful for tuning filter_top_k.

Also add a run-level summary log line at session end:

skills_filter.run_summary kept=23 dropped_ineligible=4 dropped_missing_tool=11 dropped_below_top_k=37

Bonus: rename

The current name filtered_skill_ids reads as "the IDs that were filtered out" but actually means "the IDs that survived the filter". Suggest kept_skill_ids (or active_skill_ids) for the new derived field; keep filtered_skill_ids as a deprecated alias for one release.

Area

CLI

Alternatives considered

No response