ergo-nipopow: add prove_with_reader for db-backed proof construction#851
Open
mwaddip wants to merge 1 commit into
Open
ergo-nipopow: add prove_with_reader for db-backed proof construction#851mwaddip wants to merge 1 commit into
mwaddip wants to merge 1 commit into
Conversation
Direct port of JVM
`org.ergoplatform.modifiers.history.popow.NipopowProverWithDbAlgs.prove`,
the production-side prover used by Ergo nodes to serve NiPoPoW proofs
to peers. Unlike the existing `NipopowAlgos::prove(chain: &[PoPowHeader])`
(which is paper-style and requires the caller to materialize every
block as a `PoPowHeader` up front), `prove_with_reader` walks the
interlink hierarchy via a new `PopowHeaderReader` callback and only
requests `PoPowHeader`s for blocks the walk actually visits.
Asymptotic win: ~`m + k + m * log2(N)` `popow_header_by_id` calls per
proof, vs `O(N)` for the in-memory variant. At the JVM P2P defaults
`m = 6, k = 10` on a `N ~= 270k` testnet chain that's roughly 120
fetches versus 270k — three orders of magnitude fewer. The in-memory
`prove` makes serving NiPoPoW proofs over P2P unviable on long chains
(it stalls sync, mining, and mempool for the duration of a single
peer's `GetNipopowProof`); `prove_with_reader` is the foundation for
making the Rust ergo node a viable NiPoPoW serving peer.
Changes (all in `ergo-nipopow`, plus a test in `ergo-chain-generation`):
* New `PopowHeaderReader` trait (`ergo-nipopow/src/popow_header_reader.rs`)
with five methods: `headers_height`, `popow_header_by_id`,
`popow_header_at_height`, `last_headers`, `best_headers_after`.
Re-exported from `lib.rs`. The trait is the minimal Rust analogue of
the subset of `ErgoHistoryReader` that the JVM `prove` requires.
* New `NipopowAlgos::prove_with_reader<R: PopowHeaderReader + ?Sized>`,
with private helpers `links_with_indexes`,
`previous_header_id_at_level`, `collect_level`, and `prove_prefix`
ported line-by-line from the JVM source. The Scala `@tailrec`
recursion in `collectLevel` is translated to an explicit loop. The
`mutable.TreeMap[ModifierId, PoPowHeader]` accumulator becomes a
`BTreeMap<BlockId, PoPowHeader>`.
* New `NipopowProofError::MissingPopowHeader` variant. The JVM source
uses `Try` and `.get` on every `popowHeader` lookup; the Rust port
returns `Err(MissingPopowHeader)` instead, signalling a reader that
is inconsistent with the chain it claims to expose.
The existing `NipopowAlgos::prove(&[PoPowHeader])` is left untouched —
it remains in use by `ergo-chain-generation`'s tests and is still the
canonical paper-style implementation.
Known divergence from the JVM source: the JVM `prove` accepts a
`params.continuous` flag that, when set, embeds extra epoch-boundary
popow headers into the prefix so peers can validate difficulty for
blocks past the suffix without further sync. This Rust port does NOT
implement that mode: sigma-rust's `NipopowProof` currently has no
`continuous` field, so adding it requires coordinated changes to the
struct, the serializer, and the on-wire format — out of scope for
this PR. `prove_with_reader` always produces non-continuous proofs.
JVM peers applying non-continuous proofs still succeed
(`applyPopowProof` does not strictly require the flag), they just
cannot self-validate difficulty for blocks beyond the suffix until
they sync more headers — fine for the P2P serve use case. Continuous
mode is tracked as a follow-up.
Testing:
* `test_nipopow_prove_with_reader_matches_in_memory`
(`ergo-chain-generation/src/fake_pow_scheme.rs`) asserts byte-for-byte
equivalence between `prove(&chain)` and
`prove_with_reader(&reader, None, k, m)` on a 100-block synthetic
chain at the JVM P2P defaults `m = 6, k = 10`. This is the Rust
counterpart to the JVM
`PoPowAlgosWithDBSpec`-"proof(chain) is equivalent to proof(histReader)"
test. The test is in `fake_pow_scheme.rs` (rather than
`chain_generation.rs`) because byte-for-byte equivalence only holds
when every non-genesis block has `max_level_of >= 1`: the in-memory
`prove` adds blocks at its level-0 iteration that the db-backed
interlink walk never visits, so on a chain with level-0-only blocks
(which the real-Autolykos `chain_generation.rs` generator
produces) the prefixes legitimately diverge. The fake pow scheme
forces `d = order / (height + 1)`, guaranteeing positive level for
every block — exactly mirroring `DefaultFakePowScheme` in the JVM
spec, which uses `d = q / (height + 10)` for the same reason.
* `test_nipopow_prove_with_reader_valid_on_real_autolykos_chain`
(`ergo-chain-generation/src/chain_generation.rs`) is a sanity check
on the real-Autolykos generator: the resulting proof has valid
connections, the suffix has length `k`, and the db-backed prefix is
a subset of the in-memory prefix (since the db-backed walk only
visits level-1+ blocks via interlinks).
* `test_nipopow_prove_with_reader_explicit_header_id`
(`ergo-chain-generation/src/chain_generation.rs`) covers the
`header_id_opt = Some(..)` branch: the resulting proof's suffix
head matches the requested header, the suffix tail contains the
next `k - 1` headers in ascending-height order, and the proof has
valid connections.
Verification:
cargo test --workspace -p ergo-nipopow -p ergo-chain-generation
883 passed; 0 failed
cargo clippy --workspace -p ergo-nipopow --all-targets
No issues
cargo clippy --workspace -p ergo-chain-generation --all-targets
No issues
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mwaddip
added a commit
to mwaddip/enr-chain
that referenced
this pull request
Apr 8, 2026
Replace the O(N) full-chain materialization in build_nipopow_proof with a thin adapter over ergo_nipopow::NipopowAlgos::prove_with_reader, the direct port of JVM NipopowProverWithDbAlgs.prove. The reader walks the interlink hierarchy on demand and only fetches the popow headers the proof actually visits — O(m + k + m * log2 N) per call instead of O(N). For the standard m=6, k=10 defaults at N=270k that's ~120 fetches vs ~270 000 with the old in-memory prove path: three orders of magnitude fewer. Per-request cost on testnet drops from ~5 minutes (single-threaded under the chain mutex) to subsecond, eliminating the GetNipopowProof DoS window that stalled sync, mining, and mempool for the duration. The genesis (h=1) special case moves out of build_nipopow_proof and into the new private ChainPopowReader::popow_header_at_height path: real testnet/mainnet genesis extensions are empty and would yield wrong interlinks via the loader, so the reader synthesizes the canonical [genesis_id] interlinks vector in-process. The loader is never called for h=1, matching the contract invariant in facts/chain.md. Bumps the four sigma-rust git deps (ergo-chain-types, ergo-lib, ergo-nipopow, sigma-ser) to rev 79558180 (ergoplatform/sigma-rust#851) for the new PopowHeaderReader trait, prove_with_reader entry point, and NipopowProofError::MissingPopowHeader variant. The public signature of build_nipopow_proof is unchanged. All 10 existing nipopow_proof.rs unit tests pass without modification, including build_proof_skips_loader_for_genesis (the regression guard that the genesis path never queries the loader). Pulls facts submodule to 4feccee, which updates the Phase 6 contract: prove_with_reader algorithm, O(m + k + m * log2 N) cost, the non-byte-equivalent in-memory prove divergence note, the genesis-in- reader requirement, and the m+k precondition fix. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds prove_with_reader, a chain-reader-based port of JVM NipopowProverWithDbAlgs.prove. Walks interlinks via a PopowHeaderReader callback instead of materializing the whole chain — O(m + k + m·log N) fetches vs O(N). Without it, Rust nodes can't serve GetNipopowProof over P2P on long chains.
Additive; existing prove untouched.
Does NOT implement params.continuous (no field on NipopowProof yet) — follow-up.