Caplin (gloas/ePBS): forkchoice head advances only per-epoch at tip — EL head lags ~1 epoch

[sudeepdino008]

Summary

On gloas/ePBS (glamsterdam-devnet-6), a synced erigon + embedded-Caplin node cannot reach true chain tip. It settles ~1 epoch behind: the forkchoice head advances only at epoch boundaries, so the EL is executed/committed once per epoch and JSON-RPC lags ~1 epoch (~6 min). On pre-gloas mainnet the same node follows one block per slot at tip.

Observed

• Blocks arrive every slot (OnBlock per ~12s), added to the fork graph.
• CL forkchoice head is pinned to epoch boundaries: head_slot = 242400, later 243456 — both exact multiples of 32 — each frozen for a full epoch while the current slot advances.
• EL follows per epoch: parallel executed blks=26–32, Timings: Forkchoice flush+commit exactly 384s (= 32 slots) apart, never blks=1. Steady ~1.5-epoch (~48-block) lag. is_syncing=false.

Root cause

1. The gloas head walk only advances through verified payloads. computeHeadGloas/getNodeChildren (cl/phase1/forkchoice/get_head.go, payload_vote.go) walks from the justified checkpoint; to step from block N to its child N+1 it must first reach the FULL(N) node, which is only offered when IsPayloadVerified(N) is true. If sub-epoch payloads aren't verified, the head stops at the justified block.

2. The per-slot path verifies only on a synchronous EL-Valid; a behind node never gets one. The per-slot gossip path on_execution_payload.go (validatePayloadWithEL) does mark the payload verified when NewPayload returns PayloadStatusValidated (line 306 → markPayloadVerifiedLocked) — so a caught-up node follows per slot. But when the node is behind, NewPayload returns PayloadStatusNone ("EL behind": the EL hasn't executed parent N-1 yet), and that branch defers — persists the envelope, queues it in pendingELPayloads, AddOptimisticCandidate — without marking verified. Only the batched chain_tip_sync path (drainPendingGloasPayloads, verifyUnverifiedGloasPayloads) later re-submits and marks verified.

3. A hard 1-block-per-cycle ceiling makes the gap un-closeable — it's a circular dependency, not just a slow drain. The integrated EL (ExecutionClientDirect.NewPayload) only validates a payload that is exactly EL_head + 1:

headHeader := cc.chainRW.CurrentHeader(ctx)
if headHeader == nil || header.Number > headHeader.Number+1 {
    return PayloadStatusNotValidated, nil   // more than 1 ahead → stored, not executed
}

and the EL's CurrentHeader only advances via ForkChoiceUpdate, which Caplin issues once per clstages cycle (stages/forkchoice.go) targeting the forkchoice head. So per cycle: drainPendingGloasPayloads drains the whole backlog synchronously, but only the single EL_head+1 payload validates; every deeper one returns NotValidated and requeues. That one verification lets the forkchoice head advance by one, the next cycle's FCU moves CurrentHeader by one, and the next drain verifies one more. Net: exactly one block verified per cycle ≈ chain rate, so a gap opened once (e.g. during the earlier PeerDAS stall / checkpoint sync) never closes — a stable ~1-epoch-behind equilibrium.

The circularity: verify(N) needs EL_head == N-1; EL_head advances by FCU to the forkchoice head; the forkchoice head advances only through verify(N). Each cycle breaks exactly one link forward. A caught-up node never hits it (every gossiped payload is exactly EL_head+1, validates immediately, head tracks per slot — the mainnet behavior); only a node carrying a gap does. (The maxGloasVerificationSweepPerCycle = 32 sweep is not the driver — it never runs here: zero GLOAS verification sweep log lines, swept=0.)

Corroborating symptom: PTC attestations for these blocks are rejected — bad attestation received err="PTC attestation requires verified payload envelope" — so the sub-epoch blocks are also denied payload weight.

Impact

• Node never reaches true tip on gloas; committed EL head / JSON-RPC stale by up to ~1 epoch.
• Stable ~1-epoch-behind equilibrium (drains at chain-rate each epoch, never closes the gap).

Spec + reference-client context

Spec (specs/gloas/fork-choice.md, v1.7.0-alpha.12): get_node_children emits the FULL(N) child only if is_payload_verified(store, N), and is_payload_verified(root) := root in store.payloads, written by on_execution_payload_envelope per slot. A synced node's head is expected to track within ~1 slot; per-epoch advancement is a symptom of not verifying sub-epoch payloads promptly. The spec's verify step requires a real EL validation, so the marking must follow an EL result — matching Caplin's line-306 behavior for the caught-up case.

Reference clients handle the behind case so it self-heals:

• Prysm (blockchain/receive_execution_payload_envelope.go → InsertPayload) accepts the payload optimistically when the EL returns SYNCING, letting the head advance and dragging the EL forward.
• Lighthouse gates its per-slot verify behind a synced EL and re-drives on the next tick.

Caplin has neither: on PayloadStatusNone it defers to the batched path and never re-engages the per-slot verify, so it can't climb back to per-slot cadence once behind.

Fix direction

The per-slot mark-on-Valid already exists and is correct. The missing piece is a catch-up drive that breaks the one-block-per-cycle ceiling when the verified head lags — walk the backlog in order and, for each block that is EL_head+1, NewPayload it, and once it validates, advance the EL (ForkChoiceUpdate) so the next block becomes EL_head+1, looping until the verified head reaches the seen tip. This keeps the spec invariant (mark verified only after a real EL validation) while letting a lagging node climb back to per-slot cadence in one recovery pass instead of one block per cycle.

Alternatives considered: re-entering ForwardSync — but it catches up via P2P range-download and doesn't re-drive blocks already in the graph, and its transition gate (MetaCatchingUp) keys off HighestSeen() (gossip-seen, always at tip) not the verified-head lag, so it never fires here anyway. Prysm-style optimistic acceptance of NotValidated/None would also advance the head but relaxes the spec's "verify only after real EL validation" invariant.

Environment

glamsterdam-devnet-6, erigon + embedded Caplin, recent main.

---

Update: devnet test of a catch-up fix — inert, and a second blocker surfaced

A draft fix for the ceiling (an FCU-interleaved catch-up drive in chain_tip_sync.go: after each per-slot payload validates, ForkChoiceUpdate the EL to it so the next block becomes EL_head+1 in the same pass) was built and run on glamsterdam-devnet-6. Result: inert — zero catch-up verifications. It could not engage, which exposed a distinct, dominant blocker:

• After a node restart, the block just above the EL head is missing its execution-payload envelope ([chainTipSync] envelope recovery: fetching missing envelopes). The catch-up walk only collects blocks that already have an envelope, so EL_head+1 isn't in the set, the oldest candidate is > EL_head+1, NewPayload returns NotValidated, and the drive stops immediately.
• When such an envelope is (re)fetched during forward sync, it sometimes fails EL validation with cannot derive rlp header: mismatching hash: <computed> != <payload.block_hash> and the payload is marked invalid. This is a per-payload hashing/pairing mismatch (independent of EL head or the catch-up FCU) — it looks like a forward-sync envelope re-fetch/pairing problem at the gloas boundary, and did not occur on the continuously-running baseline.

So there are two separate problems:

• (A) The FCU one-block-per-cycle ceiling (this issue's main body) — the steady-state cause of the ~1-epoch lag when envelopes are present, as seen on the original long run (blks=29 per-epoch bursts).
• (B) Post-restart envelope availability + mismatching hash invalidations at the boundary — missing/mismatched envelopes for blocks just above the EL head, which stall the node outright and mask (A).

The ceiling fix targets (A) but could not be validated because the restart used to test it put the node into state (B); a clean resync (fresh chaindata + Caplin) would be needed to exercise (A) in isolation. Blocker (B) appears to be its own forward-sync/boundary bug and is the more immediate stall.

[VBulikov]

yperbasis added Imp2 as Importance