Implement: Backfill: per-route store swapping for the experiment switcher (#145)

[ealt]

Closes #145. Implements docs/plans/issue-145-per-route-store-swap.md.

Summary

• Makes the Phase 12c experiment switcher load-bearing: every per-experiment web-ui route now resolves the active experiment per-request (resolve_active_context) and operates against its store / config / integrator-repo, instead of the startup-bound app.state.store / experiment_id / experiment_config. "Select experiment Y" now actually changes the data on ideator / executor / evaluator / all /admin/* per-experiment pages.
• Reference-impl web-ui only — no spec / wire / JSON-schema / Pydantic / conformance change (Decision 10/11). Single-experiment / no-control-plane deployments are observably unchanged (the resolve fast path does zero validation work).
• New StoreFactory (per-(experiment_id, role) StoreClient views over one shared httpx.Client, JIT worker-credential bootstrap reusing eden_service_common.auth.bootstrap_worker_credential), a top-nav switcher dropdown, a form_experiment_id switch-mid-form guard, per-experiment config-dir + integrator-repo, and the four credential-bootstrap postures (§3.2).

Landed as waves W1–W6 (see commit history + CHANGELOG [Unreleased]).

What this does NOT cover

• GET /v0/experiments/{E}/config wire endpoint — per-experiment config is loaded from an on-disk --experiment-config-dir (Decision 6); the cleaner wire-read that removes the on-disk copy + its drift risk (Risk 12) is a normative chapter-7 amendment, out of scope. Tracked in Wire endpoint GET /v0/experiments/{E}/config (replace web-ui --experiment-config-dir) #259.
• Per-request resolution cache (Decision 8 5s TTL) — only the switcher's list_experiments cache shipped; the per-request seeded/unseeded classification is uncached (correct, but a latency cost on non-default admin pages, and the default-experiment credential-staleness refresh). Tracked in Web-ui: per-request active-experiment resolution cache (Decision 8 TTL) #260.
• Tab-scoped ?exp= permalink override + draft-survives-switch (v1 affordances). Tracked in Web-ui switcher v1 affordances: ?exp= permalink override + draft-survives-switch #261.
• form_experiment_id guard on admin mutating forms — shipped on the worker submit forms (the documented long-draft risk); admin forms fail safe (cross-experiment id → NotFound) but lack the explicit banner. Tracked in Web-ui: extend form_experiment_id switch-mid-form guard to admin mutating forms #262.
• Multi-experiment Compose smoke — the single-experiment smoke remains the golden path and stays green; the multi-experiment end-to-end smoke is the pre-existing Backfill: compose-smoke-multi-experiment CI job (Phase 12c deferral) #147. The Disambiguate user-facing names from system-generated ids (experiments, workers, groups) #128 / Operator identity as a registered worker (Model B): sign-up / login / profile across UI / CLI / skill #140 / Worker registry as deployment-level infrastructure; experiments opt in #141 retrofits are unchanged (§4.2).

Fresh-operator walkthrough

• Single-experiment surface (the unchanged-behavior guarantee) verified via the full web-ui suite including the real-subprocess e2e tests (test_e2e_real_subprocess, test_admin_e2e, test_executor_e2e, test_evaluator_e2e, test_admin_workers_e2e, test_admin_groups_e2e) — these spawn the actual python -m eden_web_ui against a real task-store-server and drive claim→draft→submit / admin-reclaim through real HTTP.
• Live multi-experiment switcher walkthrough NOT performed — this environment has no running control-plane + multi-experiment Compose stack. The switcher / resolution / posture-C-D credential paths are covered by test_admin_experiments_routes.py + test_resolve_active.py + test_store_factory.py (fakes + the real control-plane server over httpx.MockTransport), but a live operator click-through of "register 2 experiments → switch → observe data follows" is deferred to the multi-experiment Compose smoke (Backfill: compose-smoke-multi-experiment CI job (Phase 12c deferral) #147). Notes: single-experiment behavior passed cleanly; multi-experiment is unit/integration-covered but not live-walked.

Test plan

• uv run ruff check . — clean
• uv run pyright reference/services/web-ui/ — 0 errors
• python3 scripts/check-complexity.py — clean (0 blocking)
• python3 scripts/check-rename-discipline.py — clean
• python3 scripts/spec-xref-check.py — clean (no spec edits)
• markdownlint (CI command) — 0 errors
• uv run pytest -q reference/services/web-ui/tests/ — green (667 → 669 with new tests); the one full-suite test_e2e_real_subprocess flake and 2 eden-checkpoint failures both pass in isolation (e2e-under-load / test-ordering artifacts, not in this diff's packages)
• uv run pytest -q (full reference suite) — 1990 passed / 221 skipped / 2 (eden-checkpoint, pass in isolation)
• codex-review (implementation profile) — converged at round 2 (3 Bugs + 2 Risks round 0, all addressed; round 2 = no remaining Bug/Risk); record under docs/plans/review/issue-145/impl/
• bash reference/compose/healthcheck/smoke.sh / smoke-subprocess.sh — NOT run (no docker in this environment); the compose changes (--experiment-config-dir / --credentials-dir / --repo-root flags + bind-mounts + setup-experiment dirs) are single-experiment-additive and should be exercised by CI's compose-smoke jobs before merge.

Note: the wave + codex-review commits on this branch are unsigned — the 1Password SSH-signing agent was locked during most of the implementation. They can be re-signed (git rebase --exec 'git commit --amend --no-edit -S' main) before merge if signature verification is required.

Related issues

• Closes Backfill: per-route store swapping for the experiment switcher (Phase 12c §3.6 deferral) #145
• Follow-ups filed: Wire endpoint GET /v0/experiments/{E}/config (replace web-ui --experiment-config-dir) #259, Web-ui: per-request active-experiment resolution cache (Decision 8 TTL) #260, Web-ui switcher v1 affordances: ?exp= permalink override + draft-survives-switch #261, Web-ui: extend form_experiment_id switch-mid-form guard to admin mutating forms #262 (Backfill: compose-smoke-multi-experiment CI job (Phase 12c deferral) #147 / Disambiguate user-facing names from system-generated ids (experiments, workers, groups) #128 / Operator identity as a registered worker (Model B): sign-up / login / profile across UI / CLI / skill #140 / Worker registry as deployment-level infrastructure; experiments opt in #141 pre-existing)

[ealt]

codex-review: converged (implementation profile, 3 rounds; record under docs/plans/review/issue-145/impl/). Round 0 raised 3 Bugs + 2 Risks (config-fallback contract, Posture C/D 401 classification, per-experiment repo durability, credential-staleness eviction, Posture-D switcher visibility) — all addressed; round 1 confirmed 4/5 + 1 new startup-robustness Risk (control-plane bootstrap now degrades instead of crashing); round 2 found no remaining Bug/Risk. New follow-up filed during review: #270 (split executor.py, which crossed 800 SLOC once #137 + #145 both landed — carries a reviewed # slop-allow-file).

Merged origin/main (the branch was behind by #110/#137/#157/#168/#140-plan). Conflicts resolved in CHANGELOG.md, docs/roadmap.md, routes/_helpers.py (unified on main's StorageNotFound alias + urlencode), and routes/evaluator.py (kept both my config=active.config threading and #137's view/resolver/variant_id redesign). Re-validated: full web-ui suite 676 passed; ruff / pyright / complexity-gate / rename-discipline / markdownlint clean. PR is now MERGEABLE.

Note: commits are unsigned (the 1Password SSH-signing agent was locked during the work; pushes went through during unlock windows). Re-sign before merge if signature verification is required.

[ealt]

Fixed the compose-smoke regression (all 8 compose-smoke + compose-e2e jobs were red: "container eden-web-ui is unhealthy").

Root cause: the codex-round-1 Bug-1 fix added a "fail fast on default-config / config-dir mismatch" check in _resolve_default_config. Every compose smoke appends fields (ideation_policy / max_quiescent_iterations) to the mounted experiment-config.yaml after setup-experiment has already copied the pre-append config into the web-ui --experiment-config-dir. The check then saw the two differ and exited the web-ui (rc=1) → unhealthy → every stack-bringing smoke failed at up --wait. All 8 smokes append, so they shared this one root cause.

Fix (0e0275c): removed the mismatch fail-fast. It was over-strict — the deployment-default experiment always resolves its config from --experiment-config (active_config's default branch returns app.state.experiment_config and never reads <config-dir>/<default>.yaml), so a divergent default entry in the config-dir is harmless. The real Bug-1 concern (silent default→non-default config fallback) is still prevented by active_config raising ExperimentConfigMissing.

Verified locally: smoke.sh → PASS, smoke-subprocess.sh → PASS (web-ui healthy, quiescence reached, all assertions pass). codex-review re-confirmed (round 3): acceptable, does not reintroduce Bug 1 — converged across 4 rounds.

rename-discipline is green (the earlier failure was a stale read of the pre-merge head); the branch is current with main (no _helpers.py conflict — that too was a stale rebase artifact). CI is re-running the smokes now.