Skip to content

TASKS.md

Latest commit

 

History

History
42 lines (28 loc) · 11 KB

TASKS.md

File metadata and controls

42 lines (28 loc) · 11 KB
Raw

TASKS

Initial development queue to reach MVP for the three e2e projects under e2e/: insurance-claim, patterns-eval, delegate-52. Ordered so each task delivers a running slice the next task builds on. Source of truth for schemas is DESIGN.md; source of truth for e2e behaviour is each project's README.md.

  • eval-cmd review-and-refactor — Once knowledge/eval.rs eval-cmd is fully implemented and green, run developer:refactor over src/knowledge/eval.rs, src/cli/eval.rs, the EvaluationRepository additions in src/content/repository.rs, and tests/eval_cmd.rs. Confirm the orchestrator/CLI split mirrors cli/run.rs, the four outcome buckets are accounted for at each report level, and the report-shape contract is locked in by the feature test.
  • project-layout review-and-refactor — Once the Project aggregate (src/content/project.rs), the Vfs<T> adapter migration in src/content/repository.rs, and the RunTx Unit of Work are fully implemented and green, run developer:refactor over src/content/project.rs, the migrated src/content/repository.rs adapters, src/cli/{assemble,run,eval}.rs, and tests/project_layout.rs. Confirm ProjectPath is the only path argument accepted by the read-side sub-handles, the grep -r tempfile src/content/ lint guard is in place, and no Fs<T> adapter, open_fs_repositories, or AssembleUnitOfWork reference survives.
  • patterns-eval-checkers review-and-refactor — Once the three structural checkers are implemented and tests/patterns_eval_checkers.rs is green (and the live falsification step in e2e/patterns-eval/ci.sh asserts improved > 0), run developer:refactor over the three check_*.py checkers, the rewritten invocation synthetics in tests/e2e_patterns_eval.rs, and ci.sh. Confirm each checker's rules trace 1:1 to a SKILL.md "Common Mistakes" entry, the empty-stderr-on-Fail contract holds across all rule branches, and the noop suite totals stay unchanged (discovery 12/0/2/0, invocation 6/0/3/0).
  • e2e-patterns-eval review-and-refactor — Once the e2e-patterns-eval wiring slice is fully implemented and the feature test tests/e2e_patterns_eval.rs is green, run developer:refactor over the new e2e project files (e2e/patterns-eval/ — assemblies, prompts, eval YAMLs, Python checker stubs, ci.sh, AGENTS.md, .gitignore, the GitHub Actions workflow) and the feature test itself. Confirm the assemblies match e2e/insurance-claim/ shape, the eval YAMLs match the README verbatim, ci.sh mirrors the insurance-claim driver with the documented two-suite differences, and the deferred-carry totals in the feature test match the design's Metrics section exactly (discovery: 12 passed / 2 deferred; invocation: 3 passed / 6 deferred).
  • ailly-skill-eval review-and-refactor — Once the ailly-skill-eval skill is fully written and the feature test tests/skill_eval_guide.rs is green, run developer:refactor over skills/ailly-skill-eval/SKILL.md, skills/ailly-skill-eval/references/method.md, and the feature test itself. Confirm: no schema is duplicated from DESIGN.md (the guide links out); every section of method.md traces to what e2e/patterns-eval/ actually builds (Fidelity rule — baseline.yaml, not invocation-baseline.yaml); the SKILL.md body has not grown a standalone "before you start, set up the project" section (the general:writing-paired-skills split trigger from the design's deferred decisions); and the final name/description wording was validated against the discovery-dogfooding metric, with the test's SKILL_NAME constant matching.

Delegate-52 enablement

  • multi-turn-skeletons — Assembly conversation: with multiple blank assistant turns; run resolves each against the cumulative transcript so far.
  • matrix-provider-axis — Matrix entries that are maps (e.g. { name: anthropic, model: claude-opus-4-7 }); per-binding model: override flows through to the engine call.
  • engine-multi-provider — Additional EngineProvider adapters for OpenAI and Google via Rig (gpt-5-turbo, gemini-3-pro per the e2e README).
  • eval-when-filter — when: subset match against meta.binding; no-filter fanout case running once per matched conversation; program_outputs plumbed into later cases for cross-binding rollups.
  • e2e-delegate-52 — Wire delegate-52: provider × domain × distractor_count matrix, six-turn workflow, per-domain scorers ported from microsoft/DELEGATE52, cross-provider judge rollup. Add scheduled CI step.

Follow-ups

  • dotenvy README pointer — Once e2e-insurance-claim and e2e-delegate-52 are wired, add a one-line note to each project's README that contributors can drop a <project>/.env instead of exporting ANTHROPIC_API_KEY in the shell, with a forward pointer to src/cli/env.rs. Not blocking; lands as a docs commit once the projects exist.

  • patterns-eval no-key CI policy — The live falsification step in e2e/patterns-eval/ci.sh now hard-fails when ANTHROPIC_API_KEY is absent, against the insurance-claim convention of gracefully skipping the live half so keyless contributors still see the assemble half pass. The acceptance run uses the key in e2e/patterns-eval/.env; the broader policy for the no-key case is unresolved. Trigger: a contributor without API access needs the assemble half to pass in CI without the live half erroring.

  • patterns-eval non-TypeScript checkers — The three check_*.py checkers target TypeScript only, because the invocation prompts pin TS (via e2e/patterns-eval/context/AGENTS.md). If a future sweep pins a different output language, each checker needs a language branch or a per-language sibling. Trigger: a patterns-eval sweep pins a non-TS output language.

  • patterns-eval structural-checker fragility — The live falsification gate in e2e/patterns-eval/ci.sh passes but with a thin, variable margin: improved ranged 1–3 across live runs and the gate survives largely on the LLM-as-judge assertion. The regex/heuristic check_*.py checkers reliably discriminate only emitting-logs, and even there a capable model's idiom variation (a bare name: event field, Brand<> helpers, expression-bodied arrow constructors) intermittently fails the invocation arm. newtype is an intentional null result (both arms pass). Decide whether to (a) lean the structural assertion on the judge and demote the script checkers to advisory, (b) pin model temperature/seed so the gate is reproducible, or (c) accept occasional gate flake and document it. Current framing is in e2e/patterns-eval/AGENTS.md "Reading the invocation comparison".

  • patterns-eval README reconciliation — The built falsification arm in e2e/patterns-eval/ is baseline.yaml; the project README still calls it invocation-baseline.yaml. The ailly-skill-eval guide follows the built artifact (Fidelity rule). Trigger: the patterns-eval README is next touched for any reason.

  • ailly-skill-eval paired-skills split — The skills/ailly-skill-eval/SKILL.md is intentionally holistic. If its body grows a standalone "before you start, set up the project" section that would repeat on every use, split into a bootstrap skill and a per-call-site skill under general:writing-paired-skills. Trigger: the SKILL.md body acquires a setup procedure section.

Deferred

  • content/conversation deferred decisions — Five revisit-after items carried over from the conversation design doc: typed ImageSource, narrowing tool_result.content, closing TraceEvent into a named enum, promoting Conversation to an aggregate root, and lifting blank/filled Message into type-states. See TASK-NOTES-conversation-deferred.md for trigger conditions per item.
  • content/evaluation deferred decisions — Three revisit-after items carried over from the evaluation design doc: ToolCallSpec extension, f64 on TextSemanticMatch.threshold (and the resulting PartialEq rather than Eq derivations), and strict ScriptBody exclusivity. (The fourth original item, semantic validation on assertion shapes, was resolved by eval-assertions-core in favor of executor-level Malformed outcomes.) See TASK-NOTES-evaluation-deferred.md for trigger conditions per item.
  • knowledge/assertions deferred decisions — Four revisit-after items carried over from the eval-assertions-core design doc: cache-hit token budgets, first-result vs all-results quantifier on json_path, reason-string format stability, and tool_call_order strict-contiguous variant. See TASK-NOTES-assertions-deferred.md for trigger conditions per item.
  • knowledge/eval-script deferred decisions — Eight revisit-after items carried over from the eval-script design doc: per-assertion timeout_ms override, suite-level pass_env defaults plus a denylist, hermetic Program resolution, runtime-version pinning via project config, stdout streaming into the report, an opt-in shared script library outside the project root, the Approach-2 ScriptContext bundle refactor, and full stderr surfacing. See TASK-NOTES-eval-script-deferred.md for trigger conditions per item.
  • engine deferred decisions — revisit-after items across the engine-provider and engine-rig design docs: keyed NoopEngine::from_table constructor, open_engine bootstrap helper, Conversation::request_at on the aggregate, promoting tokio from dev-dep to runtime dep, live wire-up for OpenAI / Gemini / Bedrock, Message.cache forwarding into the Rig request, streaming, tool-definition wiring on requests, and the RateLimited quota vs request-rate distinction. See TASK-NOTES-engine-deferred.md for trigger conditions per item.
  • eval-cmd deferred decisions — Five revisit-after items carried over from the eval-cmd design doc: per-class drill-down for script and program, report streaming, --report-dir override, exit code for deferred-only outcome, and run_id source. See TASK-NOTES-eval-cmd-deferred.md for trigger conditions per item.
  • project-layout deferred decisions — Five revisit-after items carried over from the project-layout design doc: recursive ** glob support, per-prompt cache strategy, RemoteAssemblyRepository, project schema validation at Project::open, and concurrent-writer atomicity on MemoryFS. See TASK-NOTES-project-layout-deferred.md for trigger conditions per item.