Implement: Backfill: per-route store swapping for the experiment switcher (#145)

CHANGELOG.md

@@ -8,6 +8,32 @@ Per-chunk entries preserve the full implementation record: contract amendments,
 
 ## [Unreleased]
 
+### Per-route store swapping for the experiment switcher (issue #145)
+
+Closes the Phase 12c §3.6 deferral: the cross-experiment switcher shipped in 12c (the `/admin/experiments/` dashboard, the `Session.selected_experiment_id` cookie field, and `POST /admin/experiments/{E}/select`) recorded the operator's selection but every per-experiment route still read the startup-bound `app.state.store` / `experiment_id` / `experiment_config`, so "select experiment Y" only relabelled the page. This chunk makes the selection load-bearing: every per-experiment web-ui route now resolves the active experiment per-request and operates against its store / config / repo. Reference-impl web-ui only — **no spec / wire / JSON-schema / Pydantic / conformance change** (Decision 10/11; per-route swapping is web-ui behavior with no observable signal at the chapter-9 §6 IUT contract).
+
+**Active-experiment resolution.** A new `resolve_active_context(request)` helper ([`routes/_helpers.py`](reference/services/web-ui/src/eden_web_ui/routes/_helpers.py)) is the single entry point each handler calls after its session guard: it reads `Session.selected_experiment_id`, falls back to the deployment default (`--experiment-id`), and returns either a ready `ActiveContext` (resolved `experiment_id` + per-experiment `store` / `admin_store` / `config`) or a `Response` the handler returns verbatim. With no control plane configured it always returns the deployment default with **zero** validation overhead — single-experiment deployments are observably unchanged. For a non-default selection in control-plane mode it (1) validates existence in the control plane (`StaleSelection` → dashboard redirect + cleared session field otherwise) and (2) classifies seeded vs registered-but-unseeded against the task-store-server (Decision 8's three-state model: an experiment registered on the dashboard but not yet bootstrapped by `setup-experiment` / checkpoint-import renders an "initialize me" page rather than being mis-classified as stale, Risk 11). `experiment_id` moved from a render-time Jinja global to a per-request template context processor so every page reflects the active experiment.
+
+**Per-experiment store vending.** [`store_factory.py`](reference/services/web-ui/src/eden_web_ui/store_factory.py): the live `StoreFactory` vends per-`(experiment_id, role)` `StoreClient` views against the one deployment-wide task-store URL (12c Decision 11 — no service discovery; only the `experiment_id` path segment varies), sharing one `httpx.Client` so connection-pooling is preserved. Worker-role views are JIT-credentialed on first access by `BearerCache`, which reuses [`eden_service_common.auth.bootstrap_worker_credential`](reference/services/_common/src/eden_service_common/auth.py) verbatim (the per-`worker_id` lock + idempotent-register-then-reissue + persisted-token `/whoami` verify) — never reimplementing those disciplines — under a per-experiment credential subtree `<credential-dir>/<experiment_id>/<worker_id>.token`. The auth-disabled posture (no admin token, no persisted credential) returns a `None` bearer, mirroring the prior `resolve_worker_bearer` posture 3. `StaticStoreFactory` vends one pre-built store for the single-experiment / test path. `make_app` now takes `store_factory` as its sole store dependency — the legacy `store=` / `admin_store=` kwargs and the `app.state.store` / `admin_store` attributes are gone (tests construct via the `conftest._one_experiment_factory` helper).
+
+**Credential plumbing (Posture B/C/D, §3.2).** [`credentials.py`](reference/services/web-ui/src/eden_web_ui/credentials.py) adds a deployment-scoped control-plane worker credential (`bootstrap_control_plane_credential`, persisted at `<credential-dir>/control-plane/<worker-id>.token`) so the switcher's control-plane reads (`list_experiments` / `read_experiment_metadata`, which accept any authenticated principal) keep working after the operator rotates the admin token out of the runtime env (Posture C). `resolve_credential_dir` resolves `--credential-dir` / `$EDEN_CREDENTIAL_DIR` → the common `--credentials-dir` / `$EDEN_WORKER_CREDENTIALS_DIR` → an XDG default, so the web-ui (itself a worker host) shares the established credentials volume by default. New CLI flags: `--credential-dir`, `--experiment-config-dir`, `--control-plane-worker-id`.
+
+**Per-experiment config + repo.** Each experiment's `ExperimentConfig` is loaded lazily from `<--experiment-config-dir>/<experiment_id>.yaml` (Decision 6; the deployment default still uses `--experiment-config`); `setup-experiment.sh` drops each experiment's YAML there. The executor module's local integrator clone is per-experiment via [`repo_factory.py`](reference/services/web-ui/src/eden_web_ui/repo_factory.py)'s `RepoMaterializer` (clones `<repo-path-parent>/<experiment_id>.git` from the substituted `--forgejo-url` org base, fetch-on-access); `repo_for` returns the startup clone for the default experiment.
+
+**Switcher + safety.** [`base.html`](reference/services/web-ui/src/eden_web_ui/templates/base.html) gains a no-JS top-nav switcher dropdown (CSRF-protected `select` POSTs; a 5s in-process `list_experiments` cache, §3.7; hidden without a control plane). Every worker submit form carries a hidden `form_experiment_id`; `form_experiment_guard` discards a submission whose form was rendered against a different experiment than the now-active one and redirects with a clear banner rather than writing to the wrong experiment (§3.6). The dashboard renders the resolution-failure banners (stale-selection / control-plane-unreachable / cannot-bootstrap-credential / task-store-unreachable / config-missing / config-invalid / switched-mid-form). The `AdminGateMiddleware` admins-group check now follows the active experiment (deployment-scoped `/admin/experiments` + `/admin/control` pages gate against the default and are exempt from per-experiment resolution — they are the redirect target, so resolving them per-experiment would loop).
+
+**Compose.** The web-ui service gains `--experiment-config-dir /var/lib/eden/web-ui-configs` (+ bind-mount) and an explicit `--credentials-dir /var/lib/eden/credentials` (the new resolver otherwise falls back to a non-persisted in-container XDG path); `setup-experiment.sh` creates the `web-ui-configs/` dir and copies each experiment's config in.
+
+**Tests.** New `test_store_factory.py`, `test_resolve_active.py`, `test_per_experiment_repo.py`; `test_admin_experiments_routes.py` extended (switcher render/highlight, resolution-failure banners, no-control-plane switcher absence). Full web-ui suite green (667), including the real-subprocess e2e tests; `ruff` + `pyright` + `complexity-gate` clean.
+
+**Deferred (tracked):**
+
+- *`GET /v0/experiments/{E}/config` wire endpoint* (Decision 6 alternative) — the cleaner long-term shape that removes the on-disk config-dir and its drift risk (Risk 12); a normative chapter-7 amendment, out of scope here. Filed as [#259](https://github.com/ealt/eden/issues/259).
+- *Per-request active-experiment resolution cache* (Decision 8 5s TTL) — only the switcher's `list_experiments` cache shipped; the per-request seeded/unseeded classification is uncached (correct, but a latency cost on non-default admin pages). Filed as [#260](https://github.com/ealt/eden/issues/260).
+- *Tab-scoped `?exp=` permalink override + draft-survives-switch* (§7.5 / §7.2, v1 affordances). Filed as [#261](https://github.com/ealt/eden/issues/261).
+- *`form_experiment_id` guard on admin mutating forms* — shipped on the worker submit forms; admin forms fail safe (cross-experiment id → `NotFound`) but lack the explicit guard banner. Filed as [#262](https://github.com/ealt/eden/issues/262).
+- *Multi-experiment Compose smoke* remains the existing [#147](https://github.com/ealt/eden/issues/147); the single-experiment smoke is the golden path and stays green through this chunk. The [#128](https://github.com/ealt/eden/issues/128) / [#140](https://github.com/ealt/eden/issues/140) / [#141](https://github.com/ealt/eden/issues/141) retrofits are unchanged (§4.2).
+
 ### Move deployment CLI flags into experiment-config fields (issue #157)
 
 Audits the orchestrator / worker-host CLI surface (issue #157) and moves five flags whose values two experiments sharing one deployment plausibly want different values for, from deployment-wide CLI flags / env vars into typed `experiment-config.yaml` fields — validated on both the JSON Schema and the Pydantic `eden-contracts` side per the repo's schema↔model parity discipline. Mirrors the [#133](https://github.com/ealt/eden/pull/215) `ideation_policy` template (discriminated-union YAML block + `build_*` factory + flag removal).

docs/glossary.md

@@ -247,6 +247,11 @@ EDEN maintains three branch namespaces in the experiment's git repo:
 | **`checkpoint:sha256:<hex>` URI** | Content-addressed scheme used only inside a portable-checkpoint archive: each `<hex>` is the lowercase SHA-256 of an artifact's bytes; the corresponding bytes live at `artifacts/sha256/<hex>` in the archive ([`spec/v0/10-checkpoints.md`](../spec/v0/10-checkpoints.md) §7). On import, the receiving Store rewrites each occurrence to its deployment-local URI (`file://`, `s3://`, …). Not a wire-resolvable scheme outside the archive. |
 | **import provenance** | The `Experiment.imported_from` field carrying `{checkpoint_exported_at, checkpoint_format_version}` set at import time; recovery-probe anchor for the lost-201 case in [`spec/v0/10-checkpoints.md`](../spec/v0/10-checkpoints.md) §10. Absent on natively-created experiments. |
 | **manifest** (in setup-experiment context) | A `.env` + `experiment-config.yaml` + forgejo credential helper produced for one experiment. Different from "evaluation manifest" above. |
+| **experiment switcher** | The reference web-ui affordance (cross-experiment dashboard + top-nav dropdown) by which an operator selects which registered experiment subsequent per-experiment pages operate against. Shipped in Phase 12c (selection recorded) and made load-bearing in issue [#145](https://github.com/ealt/eden/issues/145) (per-route data follows the selection). Reference-impl, not protocol. |
+| **selected experiment** | The value of `Session.selected_experiment_id` — the experiment the operator picked in the switcher, or `None` (no selection / no control plane). Distinct from the **active experiment** below. |
+| **active experiment / `active_experiment_id`** | The per-request *resolved* experiment a web-ui route operates against: the selected experiment when set and valid, else the deployment default (`--experiment-id`). Always a concrete id (never `None`). The verb is "resolve the active experiment" (`resolve_active_experiment`). Reference-impl term (issue #145). |
+| **`StoreFactory`** | The reference web-ui's per-process object that vends per-experiment `StoreClient` views against the one deployment-wide task-store URL, JIT-bootstrapping each experiment's worker credential and sharing one connection pool. Its `for_experiment(experiment_id, role)` returns the **active store** for a request. Reference-impl term (issue #145); not a protocol component. |
+| **active store** | The per-request `StoreClient` produced by `StoreFactory.for_experiment(active_experiment_id)`. A helper name, not a separate concept. |
 
 ## 9. Build / packaging vocabulary
 

docs/operations/README.md

@@ -20,6 +20,9 @@ the wire-observable end-state should look like.
   experiment (operator wire op + orchestrator policy-driven path),
   reference termination policies, drain semantics, idempotent
   re-terminate (Phase 12a-3).
+- [Web UI multi-experiment operation](web-ui-multi-experiment.md) — the
+  experiment switcher, the four credential-bootstrap postures, and the
+  per-experiment config / repo layout (issue #145).
 
 These docs assume the reference Compose deployment + the
 [`docs/glossary.md`](../glossary.md) vocabulary. For the underlying

docs/operations/web-ui-multi-experiment.md

@@ -0,0 +1,92 @@
+# Web UI: multi-experiment operation
+
+How the reference Web UI serves multiple experiments from one deployment
+once a control plane is configured (`--control-plane-url`). Shipped in
+issue [#145](https://github.com/ealt/eden/issues/145) (per-route store
+swapping for the 12c experiment switcher). Reference-impl behavior, not
+protocol.
+
+## What "select an experiment" does
+
+The cross-experiment dashboard (`/admin/experiments/`) and the top-nav
+switcher dropdown record the operator's choice in the session cookie
+(`Session.selected_experiment_id`). Every per-experiment page then
+resolves the **active experiment** per request — the selected one when
+set and valid, else the deployment default (`--experiment-id`) — and
+operates against that experiment's store, config, and (for the executor
+module) integrator repo. No control plane → the switcher is hidden and
+every page uses the deployment default with zero resolution overhead, so
+single-experiment deployments are unchanged.
+
+## Credentials: the four postures
+
+Workers are per-experiment-scoped (each experiment has its own worker
+registry). The web-ui's startup credential is registered only in its
+`--experiment-id` experiment; talking to another experiment needs a
+credential there. How that credential is obtained defines four postures
+(plan §3.2):
+
+| Posture | Admin token | Switcher works? |
+|---|---|---|
+| **A — no control plane** | optional | n/a (switcher hidden; single experiment) |
+| **B — control plane, admin token at runtime** | present | yes — per-experiment worker credentials are minted just-in-time on first switch |
+| **C — control plane, admin token bootstrap-only** | present at first boot, then rotated out | yes for experiments already credentialed on disk; new experiments redirect with `error=cannot-bootstrap-credential` |
+| **D — control plane, no admin token ever** | absent | dashboard read fails; a startup warning is logged; switcher effectively unavailable |
+
+Posture **B** is the default Compose path. Posture **C** is the
+production hardening (rotate the admin token out of the runtime env after
+first boot, matching the worker-host pattern): the web-ui persists a
+deployment-scoped control-plane worker credential at first boot so the
+switcher's control-plane reads keep working without the admin token, and
+per-experiment worker credentials persist on disk for reuse.
+
+If you switch to a new experiment in Posture C/D and see
+`cannot-bootstrap-credential`, the web-ui has no persisted credential for
+that experiment and no admin token to mint one. Either (a) provide the
+admin token at runtime (`--admin-token` / `$EDEN_ADMIN_TOKEN`), or
+(b) pre-provision the credential by running the web-ui once against that
+experiment with the admin token available.
+
+## Credential + config + repo layout
+
+Resolved by `--credential-dir` / `$EDEN_CREDENTIAL_DIR`, falling back to
+the common `--credentials-dir` / `$EDEN_WORKER_CREDENTIALS_DIR`, then an
+XDG default:
+
+```text
+<credential-dir>/
+  control-plane/<cp-worker-id>.token     # Posture B/C deployment-scoped
+  <experiment_id>/<worker-id>.token      # per-experiment worker (JIT)
+```
+
+Per-experiment **config** is loaded from
+`<--experiment-config-dir>/<experiment_id>.yaml` (the deployment default
+still uses `--experiment-config`); `setup-experiment.sh` drops each
+experiment's YAML there. Per-experiment **integrator repos** (executor
+module) live at `<repo-path-parent>/<experiment_id>.git`, cloned from the
+`--forgejo-url` org base with the experiment id substituted.
+
+> **Config drift (known limitation).** The on-disk config dir is a
+> separate source from the task-store-server's internal config text. If
+> you hand-edit one but not the other, the web-ui's and the worker hosts'
+> views of an experiment's objective / evaluation_schema can diverge.
+> Issue [#259](https://github.com/ealt/eden/issues/259) (a
+> `GET /v0/experiments/{E}/config` wire read) closes this by construction.
+
+## Compose
+
+The web-ui service passes `--experiment-config-dir
+/var/lib/eden/web-ui-configs` (bind-mounted from
+`${EDEN_EXPERIMENT_DATA_ROOT}/web-ui-configs`) and `--credentials-dir
+/var/lib/eden/credentials` (the existing credentials bind-mount).
+`setup-experiment.sh` creates `web-ui-configs/` and copies each
+experiment's config in. Register additional experiments via the
+dashboard's admin form, then run `setup-experiment.sh` per experiment to
+seed its task-store data + config YAML.
+
+A single-stack Compose deployment running multiple experiments through
+one control plane is the path of intent; running separate Compose
+projects per experiment (§12.2 of the user guide) remains valid for hard
+isolation. The end-to-end multi-experiment Compose smoke is tracked in
+[#147](https://github.com/ealt/eden/issues/147); the single-experiment
+smoke remains the golden path.