Skip to content

Commit f4b6d22

Browse files
authored
feat: implement deterministic scorers → SARIF 2.1.0 (F17) with skillet score command (#7)
Add `skillet score <path> [--format tty|json|sarif]` so produced text can be checked with free, model-free deterministic scorers emitting standard SARIF 2.1.0, closing the "no static signal" gap where skill outputs bypass triage/editor/CI integration. - `ScoreKit` implements S001–S007 (slop-vocabulary, puffery, em-dash, rule-of-three, knowledge-cutoff, not-x-but-y experimental, sarif-validity) Co-authored-by: csjones <637026+csjones@users.noreply.github.com>
1 parent 0444274 commit f4b6d22

30 files changed

Lines changed: 2278 additions & 85 deletions

AGENTS.md

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,16 @@ structured evidence, and ship a `SKILL.md` edit only after a previously-failing
88
> **Phase 2 IN PROGRESS — F3 (`skillet doctor`, the $0 preflight) and F14 (`skillet run --axis
99
> trigger`, the deterministic description axis) shipped 2026-07-04; F15 (`skillet run --ab`, the
1010
> provably skill-free baseline arm with paired Δ) shipped 2026-07-07; F16 (`skillet run --judge
11-
> grounded-judge`, the file-contents grader) shipped 2026-07-08**
11+
> grounded-judge`, the file-contents grader) shipped 2026-07-08; F17 (`skillet score`, free
12+
> deterministic scorers over produced text → SARIF 2.1.0) shipped 2026-07-09**
1213
> ([Specs/008](Specs/008-doctor-preflight/plan.md), [Specs/009](Specs/009-trigger-axis/plan.md),
13-
> [Specs/010](Specs/010-ab-baseline/plan.md), [Specs/011](Specs/011-grounded-judge/plan.md)).
14+
> [Specs/010](Specs/010-ab-baseline/plan.md), [Specs/011](Specs/011-grounded-judge/plan.md),
15+
> [Specs/012](Specs/012-deterministic-scorers/plan.md)).
1416
> `Package.swift` plus `EDDCore`, `TraceKit`,
15-
> `ProjectKit`, `RenderKit`, `HarnessKit`, `LintKit`, `JudgeKit`, `RunKit`, `ConfigYAML`, and the
17+
> `ProjectKit`, `RenderKit`, `HarnessKit`, `LintKit`, `ScoreKit`, `JudgeKit`, `RunKit`, `ConfigYAML`, and the
1618
> `skillet` executable exist — the executable owns the full ArgumentParser command tree (no
17-
> `skilletCLI` library; `ProjectKit` is the discovery/config-IO home) — with 356 tests green
18-
> (`swift build && swift test`). Shipped commands: `init` · `doctor` · `lint` · `run` · `harness
19+
> `skilletCLI` library; `ProjectKit` is the discovery/config-IO home) — with 390 tests green
20+
> (`swift build && swift test`). Shipped commands: `init` · `doctor` · `lint` · `score` · `run` · `harness
1921
> list`/`info` — see *Commands* below. **Per-feature records live nearest the artifact**: each
2022
> feature's detailed capsule is its [`Specs/NNN` plan](Specs/README.md) status, with design-only
2123
> changes in [`skillet-design-changelog.md`](skillet-design-changelog.md); this banner stays a
@@ -44,11 +46,11 @@ Contributing, security disclosure, and code of conduct are handled at the org le
4446
## Commands (true now — Phase 1 / F1, F2, F4–F8)
4547

4648
- `swift build` — build the package (resolves `swift-argument-parser`, `swift-subprocess`, `swift-system`, `swift-yaml`).
47-
- `swift test` — run the unit + integration suites (356 tests, all green). The integration suite drives
49+
- `swift test` — run the unit + integration suites (390 tests, all green). The integration suite drives
4850
the built binary, which `swift test` builds first; filter with tags, e.g. `swift test --skip slow`.
4951
- `.build/debug/skillet` — the CLI: try `skillet`, `skillet --json`, `skillet -C <dir>`, `skillet init`,
5052
`skillet init --json`, `skillet doctor [<skill>...] [--json]` (free $0 preflight; exit 3 + remedy on failure),
51-
`skillet lint [--json]`, `skillet harness list`, `skillet harness info [--json]`,
53+
`skillet lint [--json]`, `skillet score <path> [--format tty|json|sarif]` (free, model-free deterministic scorers over produced text → SARIF 2.1.0; a reporter, not a gate — exit 0 even with findings, exit 4 on a corrupt config), `skillet harness list`, `skillet harness info [--json]`,
5254
`skillet run [<skill>] [--axis behavior|trigger|all] [--ab] [--judge text-judge|grounded-judge] [--runs <k>] [--dry-run] [--yes] [--no-input] [--keep-workspace]` (paid: shells `claude`, resolved via `SKILLET_CLAUDE_CODE_BIN` env → `harness.claude-code.path` in `skillet.yaml``PATH` — behavioral trials add the judge; trigger trials are judge-free single calls; `--ab` doubles behavioral trials with a provably skill-free baseline arm, both arms judged; `--judge grounded-judge` reads produced-file contents to catch created-but-wrong, larger grading requests — gated by the combined spend estimate),
5355
`skillet --help`, `skillet --version`. Hidden test seams on `run`: `--replay` (offline adapter+judge),
5456
`--replay-map <json>` (with-arm canned verdicts), `--replay-baseline-map <json>` (baseline-arm canned

Package.swift

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,11 +52,15 @@ let package = Package(
5252
.target(name: "RenderKit", dependencies: ["EDDCore"]),
5353
// Free static-lint rule engine — pure (rules run over a parsed SkillSource), no I/O, no YAML.
5454
.target(name: "LintKit", dependencies: ["EDDCore"]),
55+
// Free, model-free deterministic output scorers → SARIF (F17). Reads produced text via ProjectKit's
56+
// SafeFile; emits EDDCore's SarifDocument / ScoreReport. No network, no subprocess.
57+
.target(name: "ScoreKit", dependencies: ["EDDCore", "ProjectKit"]),
5558
.target(
5659
name: "HarnessKit",
5760
dependencies: [
5861
"TraceKit",
5962
"EDDCore",
63+
"ProjectKit", // SafeFile — shared confinement/hidden predicates (F17)
6064
// Sole sanctioned launcher — for the real ProcessLauncher (probe; F7's run).
6165
.product(name: "Subprocess", package: "swift-subprocess"),
6266
.product(name: "SystemPackage", package: "swift-system")
@@ -68,7 +72,7 @@ let package = Package(
6872
.target(name: "JudgeKit", dependencies: ["EDDCore", "TraceKit"]),
6973
// The runner (F7): the per-trial sandbox lifecycle + the run/judge/aggregate loop. The
7074
// orchestrator that wires the harness adapter + judge over the pure pass^k core.
71-
.target(name: "RunKit", dependencies: ["EDDCore", "TraceKit", "HarnessKit", "JudgeKit"]),
75+
.target(name: "RunKit", dependencies: ["EDDCore", "TraceKit", "HarnessKit", "JudgeKit", "ProjectKit"]),
7276

7377
// Isolated YAML config seam: swift-yaml + C++ interop live ONLY here. It exposes a pure-Swift
7478
// API (decodes into EDDCore's pure SkilletConfig), so consumers stay interop-free.
@@ -90,6 +94,7 @@ let package = Package(
9094
"ProjectKit",
9195
"RenderKit",
9296
"LintKit",
97+
"ScoreKit",
9398
"TraceKit",
9499
"HarnessKit",
95100
"JudgeKit",
@@ -109,7 +114,8 @@ let package = Package(
109114
.testTarget(name: "LintKitTests", dependencies: ["LintKit"]),
110115
.testTarget(name: "HarnessKitTests", dependencies: ["HarnessKit"]),
111116
.testTarget(name: "JudgeKitTests", dependencies: ["JudgeKit"]),
112-
.testTarget(name: "RunKitTests", dependencies: ["RunKit"]),
117+
.testTarget(name: "RunKitTests", dependencies: ["RunKit", "ProjectKit"]),
118+
.testTarget(name: "ScoreKitTests", dependencies: ["ScoreKit"]),
113119
// Consumer of ConfigYAML — must also enable C++ interop (it's viral to direct importers).
114120
.testTarget(name: "ConfigYAMLTests", dependencies: ["ConfigYAML"], swiftSettings: [.interoperabilityMode(.Cxx)]),
115121

README.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,9 @@ skillet drafts and proves — **a human lands every write**.
1313
> F7 (`skillet run` — the neutral runner with `pass^k`) have landed, and Phase 2 is underway:
1414
> F3 (`skillet doctor` — the free $0 preflight), F14 (`skillet run --axis trigger` — the
1515
> description axis: does the skill *fire*?), F15 (`skillet run --ab` — the provably
16-
> skill-free baseline arm with paired Δ), and F16 (`skillet run --judge grounded-judge` — the
17-
> file-contents grader: did it write the *right* file?) shipped. The rest
16+
> skill-free baseline arm with paired Δ), F16 (`skillet run --judge grounded-judge` — the
17+
> file-contents grader: did it write the *right* file?), and F17 (`skillet score` — free,
18+
> model-free deterministic scorers over produced text → SARIF 2.1.0) shipped. The rest
1819
> of the loop lights up across later phases. See [ROADMAP.md](ROADMAP.md).
1920
2021
## How it works
@@ -69,6 +70,9 @@ skillet doctor [<skill>...] # free $0 preflight: config, harness, skill visibili
6970
skillet doctor --json # machine-readable check rows (schema: skillet.doctor/1)
7071
skillet lint # free static analysis of SKILL.md (exit 1 on error-tier findings)
7172
skillet lint --json # machine-readable findings (schema: skillet.lint/1)
73+
skillet score <path> # free, model-free scorers over produced text → SARIF 2.1.0 (reporter, not a gate; exit 0 with findings)
74+
skillet score <path> --format json # machine-readable findings (schema: skillet.score/1)
75+
skillet score <path> --format sarif # standard SARIF 2.1.0 on stdout
7276
skillet run <skill> # run the skill's evals k×, judge, report pass^k (paid; spend-gated)
7377
skillet run <skill> -n # dry-run: preview the trial-count estimate, spend nothing
7478
skillet run <skill> --axis trigger # description axis: did it fire? (deterministic, judge-free)

ROADMAP-changelog.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@ The roadmap's versioned change log, extracted from `ROADMAP.md` on 2026-07-04 (v
44
stays lean. Latest first; every version is a linkable heading; historical entries are never rewritten.
55
Companion: [`skillet-design-changelog.md`](skillet-design-changelog.md).
66

7+
## v1.17.0 — 2026-07-09
8+
9+
MINOR — **F17 SHIPPED: deterministic scorers → SARIF** ([Specs/012](Specs/012-deterministic-scorers/plan.md); design → v0.49). New `skillet score <path> [--format tty|json|sarif]` — free, model-free checks over produced text emitting standard SARIF 2.1.0; a reporter, not a gate. Adds `ScoreKit` (S001–S007 + S000), `scorers:` config block, `SarifDocument`/`ScoreReport` frozen boundary formats, RenderKit human table, and ProjectKit safe-read/confinement ownership. 390 tests green. Phase 2 remains IN PROGRESS.
10+
711
## v1.16.6 — 2026-07-08
812

913
PATCH — **F16 polish round 6** (4 optional-polish observations; 2 acted, 2 reaffirmed-as-tracked;

ROADMAP.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Product Roadmap — skillet
22

3-
**Version:** v1.16.6
4-
**Last Updated:** 2026-07-08
3+
**Version:** v1.17.0
4+
**Last Updated:** 2026-07-09
55

66
`skillet` is the SKILL.md Evaluation Toolkit — eval-driven development (EDD)
77
for agent skills, as a public, multi-harness Swift CLI. This roadmap is

Roadmap/phase-2-measurement-static-gates.md

Lines changed: 19 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -86,16 +86,24 @@ believable before the workflow layer starts acting on it.
8686
explicitly (default `text-judge`); auto-routing staged (design §14-21); a P9 cost note prints.
8787
Suite green — see the Specs/011 status row.
8888

89-
5. **[F17]** Deterministic scorers → SARIF (CLI: `skillet score`) — PLANNED · Ported
90-
- Purpose & user value: Free, model-free checks over outputs/bundles emitting
89+
5. **[F17]** Deterministic scorers → SARIF (CLI: `skillet score <path>`) — IMPLEMENTED
90+
- Purpose & user value: Free, model-free checks over produced text emitting
9191
standard SARIF 2.1.0 — the deterministic-first signal that feeds triage and
92-
editors/CI.
92+
editors/CI. A reporter, not a gate (exit 0 even with findings).
9393
- Northstar: deterministic-first; gap #1 input.
9494
- Success metrics:
95-
- `score <bundle>` emits valid SARIF 2.1.0 on stdout (schema-validated).
95+
- `score <path>` emits valid SARIF 2.1.0 on stdout (`--format sarif`; structural
96+
goldens). *(`<path>` input ships now; the saved-run `<bundle>` mode arrives with
97+
capture, Phase 3.)*
9698
- Runs with no model and no network.
97-
- Dependencies: Trace/bundle model.
98-
- Confidence: Medium — design §6.2, §11 (ScoreKit).
99+
- Checks: `SKILL-S001``S005` (slop-vocabulary, puffery, em-dash, rule-of-three,
100+
knowledge-cutoff), `S006` not-x-but-y (experimental, default-off), `S007`
101+
sarif-validity, `S000` file-unreadable. Also `--format tty|json`.
102+
- Dependencies: ProjectKit `SafeFile`; EDDCore `SarifDocument`/`ScoreReport`.
103+
- Known limitation: reads repo `skillet.yaml` + built-in defaults only; the
104+
user/`$XDG_CONFIG_HOME` config layer for `scorers` is deferred to **F24** (config
105+
precedence), a known gap vs. design §5.2.
106+
- Confidence: High — shipped; Specs/012-deterministic-scorers/plan.md, design §6.2, §11 (ScoreKit).
99107

100108
6. **[F18]** Flaky-eval hygiene, watchdog & infra-only retry — PLANNED · Net-new
101109
- Purpose & user value: Make reliability honest — flag flaky evals as hygiene
@@ -224,6 +232,11 @@ believable before the workflow layer starts acting on it.
224232

225233
## Phase Change Log
226234

235+
- 2026-07-09: **F17 IMPLEMENTED** — deterministic scorers → SARIF ([Specs/012](../Specs/012-deterministic-scorers/plan.md)):
236+
`skillet score <path> [--format tty|json|sarif]`, free model-free checks over produced text,
237+
`ScoreKit` (S001–S007 + S000), `scorers:` config block, EDDCore `SarifDocument`/`ScoreReport`
238+
frozen boundaries, RenderKit human table. A reporter, not a gate. Fifth Phase-2 feature.
239+
Roadmap → v1.17.0. Design → v0.49.
227240
- 2026-07-08: PATCH — F16 capture-hardening rounds 1–5 (17 reviewed points, incl. one correctness
228241
bug): grounded capture never opens special files (FIFO/socket/device — a hang past the harness
229242
timeout) or hard-linked/symlinked files (host-content leak), withholds non-UTF-8/unreadable files
Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
import Foundation
2+
3+
/// A discrete SARIF severity (`result.level`). Distinct from the continuous ``SarifResult/rank``:
4+
/// `level` buckets, `rank` (0–100) carries the fine-grained score. `none` is emitted for a pass-like
5+
/// result; the scorers use `note`/`warning`/`error` only.
6+
public enum SarifLevel: String, Codable, Sendable, Equatable, CaseIterable {
7+
case none, note, warning, error
8+
}
9+
10+
/// A **typed, emit-only** SARIF 2.1.0 log (F17). Companion to the tolerant reader ``SarifLog`` — this
11+
/// is the *writer*: skillet authors the shape, so there is no unknown-key round-tripping to preserve.
12+
/// Serialized by ``jsonString()`` with a **camelCase** encoder (the shared `SkilletJSON` is snake_case,
13+
/// which would emit `start_line`/`rule_id` and break GitHub/SonarQube ingestion). Frozen boundary
14+
/// format — golden-tested.
15+
public struct SarifDocument: Encodable, Sendable, Equatable {
16+
/// The advisory schema pointer. The community mirror (what the repo's reader tests already use);
17+
/// the OASIS canonical is the authoritative source:
18+
/// `https://docs.oasis-open.org/sarif/sarif/v2.1.0/errata01/os/schemas/sarif-schema-2.1.0.json`.
19+
public static let schemaURI = "https://json.schemastore.org/sarif-2.1.0.json"
20+
public static let version = "2.1.0"
21+
22+
public let runs: [SarifRun]
23+
public init(runs: [SarifRun]) { self.runs = runs }
24+
25+
private enum CodingKeys: String, CodingKey { case schema = "$schema"; case version; case runs }
26+
public func encode(to encoder: Encoder) throws {
27+
var c = encoder.container(keyedBy: CodingKeys.self)
28+
try c.encode(Self.schemaURI, forKey: .schema)
29+
try c.encode(Self.version, forKey: .version)
30+
try c.encode(runs, forKey: .runs)
31+
}
32+
33+
/// Serialize to camelCase SARIF 2.1.0 JSON (no trailing newline). Deterministic (`sortedKeys`), and
34+
/// **not** via `SkilletJSON` — the default key strategy keeps `startLine`/`ruleId`/`$schema` literal.
35+
public func jsonString() throws -> String {
36+
let encoder = JSONEncoder()
37+
encoder.outputFormatting = [.sortedKeys, .withoutEscapingSlashes]
38+
return String(decoding: try encoder.encode(self), as: UTF8.self)
39+
}
40+
}
41+
42+
public struct SarifRun: Encodable, Sendable, Equatable {
43+
public let tool: SarifTool
44+
public let results: [SarifResult]
45+
public init(tool: SarifTool, results: [SarifResult]) { self.tool = tool; self.results = results }
46+
}
47+
48+
public struct SarifTool: Encodable, Sendable, Equatable {
49+
public let driver: SarifDriver
50+
public init(driver: SarifDriver) { self.driver = driver }
51+
}
52+
53+
/// The tool descriptor. `rules` is a **static catalog** — every rule the tool can emit, listed whether
54+
/// or not it fired this run (the SARIF-idiomatic "advertise your catalog" model).
55+
public struct SarifDriver: Encodable, Sendable, Equatable {
56+
public let name: String
57+
public let version: String
58+
public let rules: [SarifRule]
59+
public init(name: String, version: String, rules: [SarifRule]) {
60+
self.name = name; self.version = version; self.rules = rules
61+
}
62+
}
63+
64+
public struct SarifRule: Encodable, Sendable, Equatable {
65+
public let id: String
66+
public let name: String
67+
public let shortDescription: SarifMessage
68+
public let properties: [String: JSONValue]?
69+
public init(id: String, name: String, shortDescription: String, properties: [String: JSONValue]? = nil) {
70+
self.id = id; self.name = name; self.shortDescription = SarifMessage(text: shortDescription)
71+
self.properties = (properties?.isEmpty ?? true) ? nil : properties
72+
}
73+
}
74+
75+
public struct SarifResult: Encodable, Sendable, Equatable {
76+
public let ruleId: String
77+
public let level: SarifLevel
78+
public let rank: Double?
79+
public let message: SarifMessage
80+
public let locations: [SarifLocation]
81+
public let properties: [String: JSONValue]?
82+
83+
public init(ruleId: String, level: SarifLevel, rank: Double?, message: String,
84+
locations: [SarifLocation], properties: [String: JSONValue]?) {
85+
self.ruleId = ruleId; self.level = level; self.rank = rank
86+
self.message = SarifMessage(text: message); self.locations = locations
87+
self.properties = (properties?.isEmpty ?? true) ? nil : properties
88+
}
89+
}
90+
91+
public struct SarifMessage: Encodable, Sendable, Equatable {
92+
public let text: String
93+
public init(text: String) { self.text = text }
94+
}
95+
96+
public struct SarifLocation: Encodable, Sendable, Equatable {
97+
public let physicalLocation: SarifPhysicalLocation
98+
public init(physicalLocation: SarifPhysicalLocation) { self.physicalLocation = physicalLocation }
99+
/// A file-level location (categorical rules): `artifactLocation` only, no `region`.
100+
public static func file(_ uri: String) -> SarifLocation {
101+
SarifLocation(physicalLocation: SarifPhysicalLocation(artifactLocation: SarifArtifactLocation(uri: uri), region: nil))
102+
}
103+
}
104+
105+
public struct SarifPhysicalLocation: Encodable, Sendable, Equatable {
106+
public let artifactLocation: SarifArtifactLocation
107+
public let region: SarifRegion?
108+
public init(artifactLocation: SarifArtifactLocation, region: SarifRegion?) {
109+
self.artifactLocation = artifactLocation; self.region = region
110+
}
111+
}
112+
113+
public struct SarifArtifactLocation: Encodable, Sendable, Equatable {
114+
public let uri: String
115+
public init(uri: String) { self.uri = uri }
116+
}
117+
118+
/// A region within an artifact. Line/column are **1-based**; `charOffset`/`charLength` are **0-based**,
119+
/// per the SARIF spec. All counts are **Unicode scalars** (§4.3). `endColumn` is the column immediately
120+
/// after the region.
121+
public struct SarifRegion: Encodable, Sendable, Equatable {
122+
public let startLine: Int
123+
public let startColumn: Int
124+
public let endLine: Int
125+
public let endColumn: Int
126+
public let charOffset: Int
127+
public let charLength: Int
128+
public init(startLine: Int, startColumn: Int, endLine: Int, endColumn: Int, charOffset: Int, charLength: Int) {
129+
self.startLine = startLine; self.startColumn = startColumn
130+
self.endLine = endLine; self.endColumn = endColumn
131+
self.charOffset = charOffset; self.charLength = charLength
132+
}
133+
}

0 commit comments

Comments
 (0)