Skip to content

Commit 03614ef

Browse files
ealtclaude
andauthored
Fix #294: checkpoint exports bundle the Forgejo repo of record (#318)
* Fix #294: checkpoint exports bundle the Forgejo repo of record Under Compose every checkpoint archive carried an empty repo.bundle: the task-store-server ran without --repo-path, so the export route emitted the chapter 10 §6 zero-byte placeholder (structurally valid, non-resumable). Forgejo is the git remote of record with no canonical local bare repo, so the fix gives the task-store-server its own bare clone, synced lazily per export: - eden-git: extract ensure_local_clone (clone --bare if absent, fetch --prune if present) from the orchestrator's _ensure_repo; both the integrator startup and the new export path share it. - eden-storage: export_checkpoint gains repo_bundle_provider, invoked exactly once AFTER the store snapshot, outside the transaction — roles publish refs before committing rows, so snapshot-then-bundle yields a §12-permitted superset; the old bundle-then-snapshot order could produce an import-rejecting archive. - eden-wire: the export route refreshes + bundles via the provider; a failed remote sync maps to 503 eden://reference-error/checkpoint-repo-unavailable instead of a silently stale/empty bundle. - task-store-server: new --forgejo-url / --credential-helper flags (orchestrator contract) build the refresh callable; startup never touches the remote, so the checkpoint-import receiver posture (postgres + task-store-server, no forgejo) keeps working. - compose.yaml / setup-experiment.sh: task-store-repo bind-mount + credential-helper mount + flags; substrate-surface audit per #178 (smoke.sh existence assertion, compose README, durability doc). - smoke-checkpoint.sh now extracts the bundle, git-bundle-verifies it, and asserts main + >= variant.integrated-count variant/* refs; smoke-auto-checkpoint.sh asserts non-empty bundles in every archive (both previously asserted structural validity only — the gap that let #294 ship). Deferral: Helm-chart parity tracked as #306. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address codex round-0 findings on the #294 checkpoint-bundle fix - Export-side §12 self-validation: non-empty provider bundles are checked against the frozen snapshot with the importer's own cross-reference validator, closing the residual snapshot↔fetch ref race; exported_at is stamped at the snapshot instant (§10 anchor). - Bundle-creation failure with a remote of record configured now 503s (checkpoint-repo-unavailable) instead of silently emitting the zero-byte placeholder; the placeholder swallow survives only in the no-remote test-fixture posture. - §14.1 format_version query param enforced: 400 bad-request on unrecognized values. - Provider tests use real git bundles (self-validation rejects unparseable bytes by design); new coverage/failure-mode tests. - Round-0 review record under docs/plans/review/issue-294-checkpoint-bundle/impl/20260611T221618/; non-blocking findings filed as #312 / #313. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address codex round-1: condense export docstring under complexity gate, fix README auth - _checkpoint.py: condense the export_checkpoint docstring so the function clears the 100-line length gate (the round-0 §12 self-validation prose pushed it to 105); content preserved. - task-store-server/README.md: the Auth section documented the retired --shared-token / §12 scheme; replace with the normative §13 --admin-token bearer scheme (codex round-1 stale-doc finding). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Eric Alt <13019253+ealt@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 69c8531 commit 03614ef

28 files changed

Lines changed: 1129 additions & 84 deletions

File tree

CHANGELOG.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,24 @@ Planless chunk. Adds [`reference/scripts/setup-aws/setup-aws.sh`](reference/scri
3636

3737
**Codex-review provenance.** Impl-stage record at [`docs/plans/review/setup-aws/impl/20260611T222444/`](docs/plans/review/setup-aws/impl/20260611T222444/). Round 0: fix-then-ship — 1×P1 (EBS addon probe accepted `CREATE_FAILED`/`DEGRADED` as converged) + 6×P2 (interrupted-create convergence, non-atomic values write, dry-run secret leakage, unvalidated adopted policy, substring trust check, under-predicated ingress probe) — all addressed. Round 1: six verified fixed, policy check partial (substring) → made semantic. Round 2: verified, **ship**.
3838

39+
### Compose checkpoint exports carry the full git bundle (issue #294)
40+
41+
Planless chunk (priority-2 durability bug; no plan doc). Closes the inherited Phase 12b completeness gap that [#131](https://github.com/ealt/eden/issues/131) surfaced: under Compose, every checkpoint archive — manual and automatic alike — carried an **empty** `repo.bundle`, because the `task-store-server` ran without `--repo-path` and the export route emitted the chapter 10 §6 zero-byte placeholder. Wire state round-tripped; git history silently did not (the archive was structurally valid but non-resumable). Root cause confirmed by live repro before the fix: a quiesced fixture experiment exported a 0-byte `checkpoint/repo.bundle` while Forgejo held `main` + `work/*` + `variant/*` refs.
42+
43+
**The design choice (the issue's open question).** There is no canonical bare repo in the Compose stack — Forgejo is the remote of record and every service keeps a private clone — so the fix gives the task-store-server its **own bare clone, synced from Forgejo per export**, rather than bundling Forgejo's on-disk repo directly (which would couple the export to Forgejo's internal storage layout and race its writes). New task-store-server flags `--forgejo-url` + `--credential-helper` (same contract as the orchestrator's) build a refresh callable from the shared [`eden_git.ensure_local_clone`](reference/packages/eden-git/src/eden_git/repo.py) helper (extracted from the orchestrator's `_ensure_repo`): clone `--bare` on first export, `fetch --prune` thereafter. The sync is **lazy** — startup never touches the remote, so the checkpoint-import receiver posture (postgres + task-store-server only, no Forgejo; smoke-checkpoint phase 4) keeps working, and no `depends_on: forgejo` was added. A failed sync fails the export **loudly** with 503 `eden://reference-error/checkpoint-repo-unavailable` (new reference-only error) instead of emitting a stale/empty bundle — silent degradation was exactly the #294 failure mode. As a side benefit the synced clone deepens the chapter-3 §3.3 tree-identity resolver that `--repo-path` also wires (the resolver itself stays I/O-free; the export path is the "separate refresh path" its docstring anticipated).
44+
45+
**Snapshot-then-bundle ordering (chapter 10 §6).** The export route previously cut the bundle *before* `Store.export_checkpoint` took its transactional snapshot — a variant integrated in between would be referenced by the snapshot but missing from the bundle, which the receiver's §12 cross-reference validation rejects. `export_checkpoint` (Store protocol + `_checkpoint.py` + `StoreClient` parity) now accepts a `repo_bundle_provider` callable invoked exactly once, **after** the snapshot, outside the store transaction (a slow fetch never holds the write lock). Roles publish git refs before committing the corresponding store row (chapter 6 §3 step 2 → step 3), so a bundle captured at-or-after the snapshot instant is a superset of every commit the snapshot references — and §12 explicitly permits superset bundles. Codex round 0 pushed this further on the residual race (a ref deleted/force-moved between snapshot and fetch): non-empty provider bundles are now **self-validated against the frozen snapshot with the importer's own §12 check** (`_validate_bundle_covers_snapshot` reuses `_validate_bundle_cross_references`), so an export that would be rejected at import time instead fails at export time — while the source experiment still exists to retry against — surfacing on the wire as the same retryable 503. `exported_at` is stamped at the snapshot instant (it is the §10 recovery-probe anchor), not after the fetch.
46+
47+
**Compose / setup wiring.** `compose.yaml` passes `--repo-path /var/lib/eden/repo --forgejo-url ${FORGEJO_REMOTE_URL} --credential-helper /etc/eden/credential-helper.sh` and bind-mounts `${EDEN_EXPERIMENT_DATA_ROOT}/task-store-repo` (rw) + the existing credential-helper script (ro); `setup-experiment.sh` creates the new substrate subdir; `smoke.sh`'s substrate-existence assertion, the compose README table, and [`docs/operations/experiment-data-durability.md`](docs/operations/experiment-data-durability.md) all gained the `task-store-repo/` entry (the #178 same-PR substrate-surface audit).
48+
49+
**Smoke gap closed.** `smoke-checkpoint.sh` passed while #294 existed — it asserted structural validity only. It now extracts `checkpoint/repo.bundle`, requires it non-empty, `git bundle verify`s it against a scratch repo, and asserts the ref set carries `refs/heads/main` plus ≥ `variant.integrated`-count `refs/heads/variant/*` refs; `smoke-auto-checkpoint.sh` asserts a non-empty `repo.bundle` in every periodic + terminal archive (deep validation stays in smoke-checkpoint, which also round-trips an import — now exercising real §12 cross-reference validation since the bundle is no longer empty). [`docs/observability.md`](docs/observability.md) §2.9 swaps the "empty git bundle under Compose" operator gap for the new posture: full git history in every checkpoint; exports 503 while Forgejo is down (auto-checkpoint logs and retries at the next cadence boundary).
50+
51+
**Tests.** `ensure_local_clone` unit tests (clone / fetch / prune / transport-error) in eden-git; `repo_bundle_provider` supersedes-bytes / post-snapshot-ordering / failure-aborts-cleanly / bundle-must-cover-snapshot / covering-bundle-passes tests in eden-storage; export-route refresh-before-bundle / refresh-failure-503 / bundle-failure-503 / format-version / no-repo-path-skips-refresh tests in eden-wire; `--forgejo-url` CLI parse/validation/lazy-clone tests in the task-store-server suite.
52+
53+
**Codex-review provenance.** Planless-chunk impl-stage record at [`docs/plans/review/issue-294-checkpoint-bundle/impl/`](docs/plans/review/issue-294-checkpoint-bundle/impl/). Round 0: 3 blocking findings, all addressed — (1) the snapshot↔bundle race + late `exported_at` (closed by the §12 export-side self-validation + snapshot-instant stamping above); (2) bundle-creation failure under a configured remote still collapsed to the silent zero-byte placeholder (now 503; the placeholder swallow survives only in the no-remote test-fixture posture); (3) the §14.1 `format_version` query param was ignored (now 400 `eden://error/bad-request` on unrecognized values). Non-blocking findings filed as issues: smoke full-object round-trip comparison → [#312](https://github.com/ealt/eden/issues/312); streaming export for very large archives → [#313](https://github.com/ealt/eden/issues/313).
54+
55+
**Deferrals.** Helm parity — the chart's task-store-server Deployment still runs without `--repo-path`/`--forgejo-url`, so Kubernetes-substrate exports still carry the empty placeholder → [#306](https://github.com/ealt/eden/issues/306) (chart wiring only; the server-side machinery from this chunk is substrate-agnostic). Smoke object-level round-trip depth → [#312](https://github.com/ealt/eden/issues/312). Streaming (non-buffering) export materialization → [#313](https://github.com/ealt/eden/issues/313).
56+
3957
### Phase 13d — S3/GCS blob backend (issue #174)
4058

4159
Adds `S3Backend` (boto3; AWS S3 + any S3-compatible service, e.g. MinIO via `endpoint_url`) and `GcsBackend` (google-cloud-storage) implementations of the [#166](https://github.com/ealt/eden/issues/166) `ArtifactBackend` Protocol in [`eden_storage.artifact_backend`](reference/packages/eden-storage/src/eden_storage/artifact_backend.py), a `--blob-backend file|s3|gcs` selection surface on the task-store-server, and a `blob.*` values block in the 13a Helm chart. Plan: [`docs/plans/eden-phase-13d-blob-backend.md`](docs/plans/eden-phase-13d-blob-backend.md); operator runbook: [`docs/deployment/migrating-to-blob-backend.md`](docs/deployment/migrating-to-blob-backend.md).

docs/observability.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -290,7 +290,7 @@ When the experiment-config opts in with an [`auto_checkpoint`](user-guide.md#aut
290290

291291
- **Restoration stays manual.** There is no auto-restore on stack-up (too magical) — use the operator-driven `eden-experiment restore` flow.
292292
- **Orphan window (no terminal after orchestrator exit).** The orchestrator is not unconditionally long-running — it exits cleanly on quiescence, and the Compose deployment does not restart a clean exit. So if an admin terminates an experiment *after* the orchestrator has already exited, no terminal checkpoint fires; the most recent periodic archive is the safety net. Closing this needs a server-side on-terminate hook (deferred — see the CHANGELOG entry for the tracking issue). Keeping the orchestrator alive longer (a high `max_quiescent_iterations`) widens the window the cadence covers.
293-
- **Empty git bundle under Compose.** The Compose `task-store-server` carries no `--repo-path`, so the git bundle inside *every* checkpoint archive — manual and auto alike — is empty: task-store wire state round-trips, git history does not yet. This is an inherited Phase 12b completeness gap (deferred — see the CHANGELOG entry); auto-checkpoint protects the task-store state but is not yet a full git-history rollback under Compose.
293+
- **Exports need Forgejo reachable.** The Compose `task-store-server` syncs a bare clone from Forgejo on every export and bundles it into the archive (issue [#294](https://github.com/ealt/eden/issues/294)), so checkpoints — manual and auto alike — carry the full git history alongside the wire state. The flip side: an export attempted while Forgejo is down fails with 503 `eden://reference-error/checkpoint-repo-unavailable` rather than silently emitting a stale or empty bundle. Auto-checkpoint treats that like any other export failure (logged; retried at the next cadence boundary).
294294
- **Disk growth + admin-token lifetime.** Budget `retention_count × checkpoint_size` per experiment for the periodic ring (plus one terminal archive). And note that with `auto_checkpoint.enabled: true` the orchestrator holds the deployment admin token in memory for its whole run (the export endpoint is admin-gated per [`07-wire-protocol.md`](../spec/v0/07-wire-protocol.md) §14) — a modest, opt-in privilege-lifetime expansion over the startup-only use it otherwise makes of the token.
295295

296296
## 3. Bring-your-own admin UIs

docs/operations/experiment-data-durability.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@ $EDEN_EXPERIMENT_DATA_ROOT/
1616
├── executor-repo/ # executor-host's per-host bare clone
1717
├── evaluator-repo/ # evaluator-host's bare clone (subprocess mode)
1818
├── web-ui-repo/ # web-ui's per-host bare clone
19+
├── task-store-repo/ # task-store-server's bare clone, synced from
20+
│ # Forgejo per checkpoint export (issue #294);
21+
│ # DERIVED — re-cloned on the next export
1922
└── credentials/
2023
├── orchestrator/ # persisted per-worker registration token
2124
├── ideator/
Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Issue #294 impl-stage codex-review — round 0 (2026-06-11)
2+
3+
Codex (via the codex-companion runtime, codex-cli 0.130.0) reviewing
4+
the diff on `impl/issue-294-checkpoint-bundle` vs `main` at the
5+
reviewed commit `0a2d1c8` — the planless checkpoint-bundle chunk
6+
(issue #294). Verdict: 3 blocking + 2 non-blocking + 1
7+
deferred-needs-issue. All three blocking findings addressed in the
8+
same PR (fix commit follows the reviewed commit); the non-blocking
9+
and deferred items are filed as issues.
10+
11+
## [Blocking 1] Snapshot↔bundle race + late `exported_at` stamp
12+
13+
`eden_storage._checkpoint.export_checkpoint` read git state (the
14+
provider fetch) after the store transaction closed, and stamped
15+
`exported_at` after that read — chapter 10 §6 requires one logical
16+
snapshot instant, and §5 makes `exported_at` the §10 recovery-probe
17+
anchor for the snapshot. The superset argument (roles publish refs
18+
before committing rows, §12 permits superset bundles) covers the
19+
common path, but a ref **deleted or force-moved** between snapshot
20+
and fetch could still leave the bundle missing something the snapshot
21+
references.
22+
23+
**Resolution:** (a) `exported_at` is now stamped inside the snapshot
24+
transaction. (b) Non-empty provider bundles are self-validated
25+
against the frozen snapshot with the importer's own §12 check —
26+
`_validate_bundle_covers_snapshot` reuses
27+
`_validate_bundle_cross_references` over the snapshot's re-validated
28+
variant/idea rows. Mirroring the importer exactly means the check can
29+
never reject an archive that would have imported; it only moves the
30+
failure to export time, while the source still exists to retry
31+
against. The wire route maps the resulting `CheckpointInvalid` to the
32+
retryable 503 `eden://reference-error/checkpoint-repo-unavailable`.
33+
Regression tests: `test_repo_bundle_provider_bundle_must_cover_snapshot`
34+
(bundle missing a snapshot branch → raise, zero archive bytes) and
35+
`test_repo_bundle_provider_covering_bundle_passes` in
36+
`test_checkpoint_storage.py`.
37+
38+
## [Blocking 2] Bundle-creation failure still collapsed to the silent zero-byte placeholder
39+
40+
`_compose_repo_bundle` swallowed `CheckpointInvalid` from
41+
`create_bundle` into `b""` unconditionally — with a remote of record
42+
configured, that re-creates the #294 silent-non-resumable-archive
43+
failure mode (200 with an empty bundle), and the importer skips §12
44+
validation on zero-byte bundles, so the rows would import without git
45+
history.
46+
47+
**Resolution:** the swallow now survives only in the no-remote
48+
posture (a local test repo with no refs is a legitimate empty
49+
bundle). When `checkpoint_repo_refresh` is configured, any bundle
50+
failure after a successful sync raises `CheckpointRepoUnavailable`
51+
(503) — a healthy seeded remote always carries at least the seed ref.
52+
Regression tests: `test_export_bundle_failure_with_remote_maps_to_503`
53+
and `test_export_empty_local_repo_without_remote_keeps_placeholder`
54+
in `test_checkpoint_wire.py`.
55+
56+
## [Blocking 3] §14.1 `format_version` query param ignored
57+
58+
`spec/v0/07-wire-protocol.md` §14.1 defines the optional
59+
`format_version` query and mandates 400 for unrecognized values; the
60+
export handler ignored it entirely (pre-existing wave-4 gap, surfaced
61+
by this review).
62+
63+
**Resolution:** the route now parses `format_version` and rejects any
64+
value other than `CHECKPOINT_FORMAT_VERSION` with `BadRequest`
65+
(`eden://error/bad-request`). Tests:
66+
`test_export_accepts_current_format_version` /
67+
`test_export_rejects_unrecognized_format_version`.
68+
69+
## [Non-blocking 1] Smoke round-trip equality is counts/id-sets, not full objects
70+
71+
`smoke-checkpoint.sh` Phase 6 compares counts and sorted id sets;
72+
chapter 10 §9 promises field-level round-trip modulo the documented
73+
import stamps. Filed as
74+
[#312](https://github.com/ealt/eden/issues/312) (smoke-depth
75+
improvement; production path already exercised).
76+
77+
## [Non-blocking 2] Ordering test didn't prove bundle-satisfies-snapshot
78+
79+
Addressed by the Blocking-1 resolution's new coverage tests; the
80+
post-snapshot-mutation test now also documents that the §12
81+
self-validation runs against the frozen snapshot, not a re-read.
82+
83+
## [Deferred] Streaming export for very large archives
84+
85+
The route docstring deferred a streaming temp-file materialization
86+
model without a tracking issue. Filed as
87+
[#313](https://github.com/ealt/eden/issues/313) (chapter 10 §6
88+
explicitly permits the buffering model; this is scalability
89+
hardening, not a conformance gap).

reference/compose/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ and [`spec/v0/01-concepts.md`](../../spec/v0/01-concepts.md) §13.
5555
| `<DATA_ROOT>/executor-repo/` | `executor-host` | per-host bare clone of the Forgejo repo |
5656
| `<DATA_ROOT>/evaluator-repo/` | `evaluator-host` (subprocess overlay) | per-host bare clone for subprocess evaluator |
5757
| `<DATA_ROOT>/web-ui-repo/` | `web-ui` | per-host bare clone for the web-ui executor module |
58+
| `<DATA_ROOT>/task-store-repo/` | `task-store-server` | bare clone synced from Forgejo per checkpoint export (issue #294) |
5859
| `<DATA_ROOT>/artifacts/` | `web-ui`, `ideator-host`, `executor-host` (ro) | Artifact store / idea markdown |
5960
| `<DATA_ROOT>/credentials/{orchestrator,ideator,executor,evaluator,web-ui}/` | per-host services | Persisted per-worker registration tokens |
6061

reference/compose/compose.yaml

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,23 @@ services:
194194
- ${EDEN_ADMIN_TOKEN:?EDEN_ADMIN_TOKEN must be set (run setup-experiment)}
195195
- --artifacts-dir
196196
- /var/lib/eden/artifacts
197+
# Issue #294: the checkpoint export's repo.bundle source. The
198+
# task-store-server keeps its own bare clone of the Forgejo
199+
# repo of record at --repo-path and syncs it from --forgejo-url
200+
# per export (clone on first export, fetch --prune thereafter).
201+
# The sync is LAZY — nothing touches Forgejo at startup, which
202+
# keeps the checkpoint-import receiver posture working (the
203+
# smoke brings up only postgres + task-store-server, no
204+
# forgejo; see smoke-checkpoint.sh phase 4). NO depends_on on
205+
# forgejo for the same reason. As a side benefit the synced
206+
# clone deepens the §3.3 tree-identity resolver that --repo-path
207+
# also wires.
208+
- --repo-path
209+
- /var/lib/eden/repo
210+
- --forgejo-url
211+
- ${FORGEJO_REMOTE_URL:?}
212+
- --credential-helper
213+
- /etc/eden/credential-helper.sh
197214
- --host
198215
- 0.0.0.0
199216
- --port
@@ -211,6 +228,11 @@ services:
211228
# durability migration); the task-store-server gets ro
212229
# because it only reads.
213230
- ${EDEN_EXPERIMENT_DATA_ROOT:?}/artifacts:/var/lib/eden/artifacts:ro
231+
# Issue #294: the checkpoint repo clone (rw — the export's
232+
# clone/fetch writes here) + the Forgejo HTTP credential
233+
# helper, mirroring the orchestrator's mounts.
234+
- ${EDEN_EXPERIMENT_DATA_ROOT:?}/task-store-repo:/var/lib/eden/repo
235+
- ${EDEN_FORGEJO_CREDS_DIR_HOST:?}/credential-helper.sh:/etc/eden/credential-helper.sh:ro
214236
# Issue #109: per-service log bind-mount (writable). Survives
215237
# `compose down -v`. setup-experiment.sh creates the host-side
216238
# dir tree.

0 commit comments

Comments
 (0)