docs(skill-eval): ailly-skill-eval authoring skill and guide

.gitignore

@@ -14,6 +14,8 @@
 e2e/**/.env
 !mise.toml
 !rust-toolchain.toml
+!skills
+!skills/**
 !src
 !src/**
 !tests

docs/developer/TASKS.md

@@ -2,9 +2,12 @@
 
 Initial development queue to reach MVP for the three e2e projects under `e2e/`: `insurance-claim`, `patterns-eval`, `delegate-52`. Ordered so each task delivers a running slice the next task builds on. Source of truth for schemas is [DESIGN.md](../../DESIGN.md); source of truth for e2e behaviour is each project's `README.md`.
 
-## Patterns-eval enablement
+- **eval-cmd review-and-refactor** — Once `knowledge/eval.rs eval-cmd` is fully implemented and green, run `developer:refactor` over `src/knowledge/eval.rs`, `src/cli/eval.rs`, the `EvaluationRepository` additions in `src/content/repository.rs`, and `tests/eval_cmd.rs`. Confirm the orchestrator/CLI split mirrors `cli/run.rs`, the four outcome buckets are accounted for at each report level, and the report-shape contract is locked in by the feature test.
+- **project-layout review-and-refactor** — Once the `Project` aggregate ([src/content/project.rs](../../src/content/project.rs)), the `Vfs<T>` adapter migration in [src/content/repository.rs](../../src/content/repository.rs), and the `RunTx` Unit of Work are fully implemented and green, run `developer:refactor` over `src/content/project.rs`, the migrated `src/content/repository.rs` adapters, `src/cli/{assemble,run,eval}.rs`, and `tests/project_layout.rs`. Confirm `ProjectPath` is the only path argument accepted by the read-side sub-handles, the `grep -r tempfile src/content/` lint guard is in place, and no `Fs<T>` adapter, `open_fs_repositories`, or `AssembleUnitOfWork` reference survives.
+- **patterns-eval-checkers review-and-refactor** — Once the three structural checkers are implemented and [tests/patterns_eval_checkers.rs](../../tests/patterns_eval_checkers.rs) is green (and the live falsification step in [e2e/patterns-eval/ci.sh](../../e2e/patterns-eval/ci.sh) asserts `improved > 0`), run `developer:refactor` over the three `check_*.py` checkers, the rewritten invocation synthetics in [tests/e2e_patterns_eval.rs](../../tests/e2e_patterns_eval.rs), and `ci.sh`. Confirm each checker's rules trace 1:1 to a SKILL.md "Common Mistakes" entry, the empty-stderr-on-Fail contract holds across all rule branches, and the noop suite totals stay unchanged (discovery 12/0/2/0, invocation 6/0/3/0).
+- **e2e-patterns-eval review-and-refactor** — Once the e2e-patterns-eval wiring slice is fully implemented and the feature test [tests/e2e_patterns_eval.rs](../../tests/e2e_patterns_eval.rs) is green, run `developer:refactor` over the new e2e project files ([e2e/patterns-eval/](../../e2e/patterns-eval/) — assemblies, prompts, eval YAMLs, Python checker stubs, `ci.sh`, AGENTS.md, `.gitignore`, the GitHub Actions workflow) and the feature test itself. Confirm the assemblies match `e2e/insurance-claim/` shape, the eval YAMLs match the README verbatim, `ci.sh` mirrors the insurance-claim driver with the documented two-suite differences, and the deferred-carry totals in the feature test match the design's Metrics section exactly (discovery: 12 passed / 2 deferred; invocation: 3 passed / 6 deferred).
+- **ailly-skill-eval review-and-refactor** — Once the `ailly-skill-eval` skill is fully written and the feature test [tests/skill_eval_guide.rs](../../tests/skill_eval_guide.rs) is green, run `developer:refactor` over `skills/ailly-skill-eval/SKILL.md`, `skills/ailly-skill-eval/references/method.md`, and the feature test itself. Confirm: no schema is duplicated from [DESIGN.md](../../DESIGN.md) (the guide links out); every section of `method.md` traces to what `e2e/patterns-eval/` actually builds (Fidelity rule — `baseline.yaml`, not `invocation-baseline.yaml`); the `SKILL.md` body has not grown a standalone "before you start, set up the project" section (the `general:writing-paired-skills` split trigger from the design's deferred decisions); and the final `name`/`description` wording was validated against the discovery-dogfooding metric, with the test's `SKILL_NAME` constant matching.
 
-- **e2e-patterns-eval** — Wire patterns-eval: resolve deferred items. Green end to end. Refactor step, then dedicated global refactor. Add CI step.
 
 ## Delegate-52 enablement
 
@@ -17,9 +20,16 @@ Initial development queue to reach MVP for the three e2e projects under `e2e/`:
 ## Follow-ups
 
 - **dotenvy README pointer** — Once `e2e-insurance-claim` and `e2e-delegate-52` are wired, add a one-line note to each project's README that contributors can drop a `<project>/.env` instead of exporting `ANTHROPIC_API_KEY` in the shell, with a forward pointer to [src/cli/env.rs](../../src/cli/env.rs). Not blocking; lands as a docs commit once the projects exist.
-- **eval-cmd review-and-refactor** — Once `knowledge/eval.rs eval-cmd` is fully implemented and green, run `developer:refactor` over `src/knowledge/eval.rs`, `src/cli/eval.rs`, the `EvaluationRepository` additions in `src/content/repository.rs`, and `tests/eval_cmd.rs`. Confirm the orchestrator/CLI split mirrors `cli/run.rs`, the four outcome buckets are accounted for at each report level, and the report-shape contract is locked in by the feature test.
-- **project-layout review-and-refactor** — Once the `Project` aggregate ([src/content/project.rs](../../src/content/project.rs)), the `Vfs<T>` adapter migration in [src/content/repository.rs](../../src/content/repository.rs), and the `RunTx` Unit of Work are fully implemented and green, run `developer:refactor` over `src/content/project.rs`, the migrated `src/content/repository.rs` adapters, `src/cli/{assemble,run,eval}.rs`, and `tests/project_layout.rs`. Confirm `ProjectPath` is the only path argument accepted by the read-side sub-handles, the `grep -r tempfile src/content/` lint guard is in place, and no `Fs<T>` adapter, `open_fs_repositories`, or `AssembleUnitOfWork` reference survives.
-- **e2e-patterns-eval review-and-refactor** — Once the e2e-patterns-eval wiring slice is fully implemented and the feature test [tests/e2e_patterns_eval.rs](../../tests/e2e_patterns_eval.rs) is green, run `developer:refactor` over the new e2e project files ([e2e/patterns-eval/](../../e2e/patterns-eval/) — assemblies, prompts, eval YAMLs, Python checker stubs, `ci.sh`, AGENTS.md, `.gitignore`, the GitHub Actions workflow) and the feature test itself. Confirm the assemblies match `e2e/insurance-claim/` shape, the eval YAMLs match the README verbatim, `ci.sh` mirrors the insurance-claim driver with the documented two-suite differences, and the deferred-carry totals in the feature test match the design's Metrics section exactly (discovery: 12 passed / 2 deferred; invocation: 3 passed / 6 deferred).
+- **patterns-eval no-key CI policy** — The live falsification step in [e2e/patterns-eval/ci.sh](../../e2e/patterns-eval/ci.sh) now hard-fails when `ANTHROPIC_API_KEY` is absent, against the insurance-claim convention of gracefully skipping the live half so keyless contributors still see the assemble half pass. The acceptance run uses the key in `e2e/patterns-eval/.env`; the broader policy for the no-key case is unresolved. **Trigger:** a contributor without API access needs the assemble half to pass in CI without the live half erroring.
+
+- **patterns-eval non-TypeScript checkers** — The three `check_*.py` checkers target TypeScript only, because the invocation prompts pin TS (via [e2e/patterns-eval/context/AGENTS.md](../../e2e/patterns-eval/context/AGENTS.md)). If a future sweep pins a different output language, each checker needs a language branch or a per-language sibling. **Trigger:** a patterns-eval sweep pins a non-TS output language.
+
+- **patterns-eval structural-checker fragility** — The live falsification gate in [e2e/patterns-eval/ci.sh](../../e2e/patterns-eval/ci.sh) passes but with a thin, variable margin: `improved` ranged 1–3 across live runs and the gate survives largely on the LLM-as-judge assertion. The regex/heuristic `check_*.py` checkers reliably discriminate only `emitting-logs`, and even there a capable model's idiom variation (a bare `name:` event field, `Brand<>` helpers, expression-bodied arrow constructors) intermittently fails the invocation arm. `newtype` is an intentional null result (both arms pass). Decide whether to (a) lean the structural assertion on the judge and demote the script checkers to advisory, (b) pin model temperature/seed so the gate is reproducible, or (c) accept occasional gate flake and document it. Current framing is in [e2e/patterns-eval/AGENTS.md](../../e2e/patterns-eval/AGENTS.md) "Reading the invocation comparison".
+
+- **patterns-eval README reconciliation** — The built falsification arm in [e2e/patterns-eval/](../../e2e/patterns-eval/) is `baseline.yaml`; the project README still calls it `invocation-baseline.yaml`. The [ailly-skill-eval guide](../../skills/ailly-skill-eval/references/method.md) follows the built artifact (Fidelity rule). **Trigger:** the patterns-eval README is next touched for any reason.
+
+- **ailly-skill-eval paired-skills split** — The [skills/ailly-skill-eval/SKILL.md](../../skills/ailly-skill-eval/SKILL.md) is intentionally holistic. If its body grows a standalone "before you start, set up the project" section that would repeat on every use, split into a bootstrap skill and a per-call-site skill under `general:writing-paired-skills`. **Trigger:** the SKILL.md body acquires a setup procedure section.
+
 
 ## Deferred
 

e2e/patterns-eval/.gitignore

@@ -1,2 +1,3 @@
 runs/
 evals/reports/
+evals/judges/

e2e/patterns-eval/AGENTS.md

@@ -10,3 +10,30 @@ Two axes are exercised: **discovery** asks the model to pick the right
 skill from its `description:` frontmatter alone, and **invocation** asks
 the model to produce code that structurally exhibits the named pattern
 once the skill is loaded.
+
+## Reading the invocation comparison
+
+The invocation axis is run as an A/B comparison: a **baseline** arm (no
+skill) against an **invocation** arm (skill loaded), over identical
+prompts. The falsification gate in `ci.sh` requires `improved > 0` and
+`regressed == 0` — the skill must help on at least one assertion and harm
+none.
+
+Not every skill produces improvement, and that is a feature, not a bug:
+
+- **emitting-logs** is the clear positive. Without the skill the model
+  interpolates values into the message body and skips `eventName`; with
+  it, the record is structured under semantic-convention keys. Both the
+  script checker and the judge flip from fail to pass.
+- **configuring-logging** improves on the judge: the skill produces a
+  fuller five-layer pipeline than the baseline's partial attempt.
+- **newtype** is a deliberate **null result**. A capable model already
+  reaches for brand types when asked for swap-proof ids, with or without
+  the skill, so both arms pass. newtype is retained precisely to show
+  what a skill that does *not* change a capable model's output looks like
+  in the report: `UnchangedPass`, contributing nothing to `improved`. The
+  gate does not depend on it.
+
+The lesson the harness encodes: a skill earns its place by changing
+behaviour a baseline model would otherwise get wrong. The eval reports
+both the skills that clear that bar and the ones that do not.

e2e/patterns-eval/assemblies/baseline.yaml

@@ -8,7 +8,8 @@ matrix:
     - emitting-logs
 
 prefix:
-  - { kind: file, path: ./AGENTS.md, cache: true }
+  - { kind: file, path: ./AGENTS.md,         cache: true }
+  - { kind: file, path: ./context/AGENTS.md, cache: true }
 
 conversation:
   - { role: user, path: "prompts/invocation/{{ skill }}.md" }

e2e/patterns-eval/assemblies/invocation.yaml

@@ -9,6 +9,7 @@ matrix:
 
 prefix:
   - { kind: file,   path: ./AGENTS.md,                                cache: true }
+  - { kind: file,   path: ./context/AGENTS.md,                        cache: true }
   - { kind: system, path: ./context/skills/using-patterns/SKILL.md,   cache: true }
   - kind: system
     path: "context/skills/{{ skill }}/SKILL.md"

e2e/patterns-eval/ci.sh

@@ -5,11 +5,11 @@
 # (discovery and invocation):
 #   1. `ailly assemble <suite>` -- always runs; asserts N conversation
 #      files land under runs/<id>/ for each suite.
-#   2. `ailly run runs/<id>/`   -- runs when ANTHROPIC_API_KEY is
-#      present; asserts every conversation file's trailing blank
-#      assistant slot has been filled. Skipped with a clear notice
-#      otherwise so contributors without API access still see the
-#      assemble half pass.
+#   2. `ailly run runs/<id>/`   -- requires a live model. Asserts every
+#      conversation file's trailing blank assistant slot has been
+#      filled. With neither ANTHROPIC_API_KEY nor a project .env the
+#      script hard-fails: there is no assemble-only success path, so the
+#      live half (and the falsification gate below) always runs.
 #   3. `ailly eval <suite> --over runs/<id>/` -- runs after `ailly run`;
 #      asserts the per-run report file landed at
 #      evals/reports/<run-id>.json and prints a deferred-tolerance
@@ -19,7 +19,6 @@
 # resolves its own location to find the project root.
 
 set -euo pipefail
-set -x
 
 project_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 repo_root="$(cd "${project_dir}/../.." && pwd)"
@@ -96,8 +95,9 @@ assemble_suite invocation
 # --- CUJ 2: run (both suites, gated on credentials) -------------------------
 
 if [[ -z "${ANTHROPIC_API_KEY:-}" && ! -f "${project_dir}/.env" ]]; then
-  echo "SKIP: ailly run requires ANTHROPIC_API_KEY in the shell or ${project_dir#"${repo_root}/"}/.env; assemble half passed."
-  exit 0
+  echo "FAIL: ailly run requires a live model. Set ANTHROPIC_API_KEY in the shell or drop a ${project_dir#"${repo_root}/"}/.env file." >&2
+  echo "      The live half exercises the model and the falsification gate; there is no assemble-only success path." >&2
+  exit 1
 fi
 
 # Asserts that every assembled conversation file under the suite's
@@ -235,6 +235,11 @@ report_comparison() {
     exit 1
   fi
 
+  # Falsification gate: the invocation arm (skill loaded) must pass
+  # assertions the baseline arm (no skill) fails -- improved > 0 -- and the
+  # skill must never make a passing baseline case fail -- regressed == 0.
+  # A checker too lenient to fail baseline output yields improved == 0 and
+  # fails CI here, turning the README's falsification property into a gate.
   python3 - "${run_id_a}" "${run_id_b}" "${comparison_json}" <<'PY'
 import json, sys
 id_a, id_b, path = sys.argv[1], sys.argv[2], sys.argv[3]
@@ -248,9 +253,25 @@ print(
     f"unchanged_pass={t['unchanged_pass']} "
     f"unchanged_fail={t['unchanged_fail']}"
 )
+errors = []
+if t["improved"] <= 0:
+    errors.append(
+        f"improved={t['improved']} (expected > 0): the invocation arm passed no "
+        "assertion the baseline arm failed; the checkers are too lenient to "
+        "falsify un-skilled output, so the falsification claim is vacuous"
+    )
+if t["regressed"] != 0:
+    errors.append(
+        f"regressed={t['regressed']} (expected 0): loading the skill made a "
+        "passing baseline case fail"
+    )
+if errors:
+    for e in errors:
+        print(f"FAIL: {e}", file=sys.stderr)
+    sys.exit(1)
 PY
 
-  echo "OK: ailly report wrote ${comparison_json#"${repo_root}/"} and ${comparison_md#"${repo_root}/"}"
+  echo "OK: ailly report wrote ${comparison_json#"${repo_root}/"} and ${comparison_md#"${repo_root}/"} (improved>0, regressed==0)"
 }
 
 report_discovery

e2e/patterns-eval/context/AGENTS.md

@@ -0,0 +1,11 @@
+# Candidate project: Hello World server
+
+This is the codebase you are editing. It is a small **TypeScript** HTTP
+server — a "Hello World" service that greets callers and places orders.
+It targets Node.js with `tsc` strict mode.
+
+All code you write is **TypeScript**. Answer every coding task with
+TypeScript source, never another language. Follow the idioms the loaded
+skills demonstrate in their primary examples (brand types, the subscriber
+registry, structured log records). Show the type or function definition
+and, where the task asks for one, a single example call site.

e2e/patterns-eval/evals/baseline.yaml

@@ -21,7 +21,9 @@ cases:
           resource attributes, installs the W3C `traceparent` propagator,
           and registers a shutdown flush with a hard timeout. No `init` is
           called from library code.
-      - { type: tokens, metric: total, op: "<", value: 8000 }
+      # Kept equal to invocation.yaml: see the note there. The budget must match
+      # across arms so the comparison's token outcome reflects a real size delta.
+      - { type: tokens, metric: total, op: "<", value: 14000 }
 
   - name: emitting-logs
     assertions: