feat(migration): trie migration

[MaksymMalicki]

Summary

One-shot migration that converts every deprecated Starknet trie on disk into the equivalent trie2 layout. After this migration runs, the new state package can read directly from the new buckets and the deprecated buckets are wiped.

Bucket mapping:

Deprecated	New	Hash
ClassesTrie	ClassTrie	Poseidon
StateTrie	ContractTrieContract	Pedersen
ContractStorage	ContractTrieStorage	Pedersen (per-owner)

Format differences

Three things change between the two layouts: how nodes are keyed on disk, how each node's value is encoded, and how path compression is expressed.

On-disk keying

Both layouts share a common prefix; only the suffix differs:

common (both) suffix
───────────── ─────────────────────────────────────────
bucket [|| owner] → path-length-byte || path-bytes (deprecated)
→ nodeType-byte || path-length-byte || path-bytes (new)

owner is present only for storage tries. The new layout's extra nodeType byte splits leaves from internal nodes into two index slices within the same bucket — trie2 state lookups use this to short-circuit between leaf reads and internal-node traversals.

Node encoding

Both layouts are raw byte streams (no length prefixes, no varints — field-element widths are fixed).

Deprecated: nodes are self-contained. Internal binary nodes embed the compressed paths to their children inline:

leaf value
binary value || left-child-path || right-child-path
[|| left-hash || right-hash, optional cache, ignored here]

value is the node's own Starknet trie hash, or the stored value when the node is a leaf. The trailing hash pair was a denormalised cache; the migrator does not read it (hashes are recomputed from scratch — see below).

New: every node has an explicit type tag and path compression lives in dedicated edge nodes:

value value
binary 0x01 || left-edge-hash || right-edge-hash
edge 0x02 || child-hash || encoded-path-segment

Path compression — the key structural change

The deprecated format compresses paths inside the parent binary node (via its embedded child-path fields). The new format moves compression into dedicated edge nodes sitting between binary nodes and their children:

deprecated: binary ──────── child-path ────────► child
new: binary ──► edge ──► child

One consequence: the deprecated root marker (a single entry at the bare bucket prefix recording the root's path) disappears. Whatever the deprecated root embedded becomes either a direct binary/leaf at the empty path or, when the deprecated root path is itself non-empty, an edge node at the empty path that points "down" to the real root.

Traversal

DFS is a natural fit here. In the new layout a binary node's payload is left-edge-hash || right-edge-hash — so before we can write the parent, we need both children's hashes. A bottom-up walk reads each deprecated node exactly once and produces the child hash that the parent needs at the moment the parent is encoded; no separate hashing pass, no intermediate caches sized to the trie. Going top-down would force us to either re-read every child later or hold every visited node in memory until its subtree is hashed.

For each trie:

1. Enumerate — scan the deprecated bucket once. Class and state tries each occupy a whole bucket; storage tries share ContractStorage keyed by bucket || owner, so the enumerator splits them into per-owner descriptors. Each descriptor records the root path (from the bare-prefix marker entry) and the node count.
2. DFS — recurse from the root path. A deprecated leaf becomes a value node at the same path. An internal binary node, after both subtrees have been visited, becomes a binary node plus up to two edge nodes (one per non-empty child segment). If the trie's stored root path is itself non-empty, a single edge node at the empty path is written after the traversal completes, replacing the root marker.
3. Resumability check — before doing any work for a trie, the migrator looks up its new-format root key. If present, the trie is credited toward progress and skipped.

Pipeline: IngestorCount worker goroutines pull descriptors from the enumeration source; a single committer flushes filled batches to disk. A semaphore caps in-flight batches at IngestorCount * 2. Every flush and every channel send observes ctx.Done; on cancel Migrate returns the shouldRerun sentinel and the migration runner re-invokes on the next process start.

After the full pipeline finishes, the three deprecated buckets are wiped via DeleteRange. The wipe is gated on full success — a crashed mid-migration leaves the deprecated source intact, so partially migrated tries either have a new-format root (skipped on the next pass) or don't (re-migrated from scratch).

Alternative considered: reverse-iteration BFS

An earlier attempt used reverse-iteration BFS over the deprecated bucket — iterating keys from longest-path to shortest so leaves are processed first, then their parents, then their parents, and so on. Each level's hashes are buffered until the next level up consumes them.

In practice it performed comparably to DFS on wall-clock time but with substantially higher peak memory — the buffer of "hashes waiting for their parent" grows roughly with the widest level of the trie, which for full or near-full subtrees is most of the leaves. DFS keeps only the current root-to-leaf path in flight, which is bounded by the trie depth (≤ 251), so memory stays flat regardless of trie size. Same correctness, worse memory profile, no speed win — dropped.

Hashing

Starknet trie hashes for the new layout:

leaf value
binary hashFn(left-edge-hash, right-edge-hash)
edge hashFn(child-hash, path-segment-as-felt) + segment-length

A zero-length edge short-circuits to the bare child-hash — the convention for absent edges. Class tries hash with Poseidon; contract and storage tries with Pedersen.

Performance: for small tries (below SmallTrieThreshold nodes) every edge hash is computed inline. Above the threshold, edge-hash jobs are batched (parallelHashBatchSize) and dispatched to a fixed-size worker pool for parallel computation. The scheduler preserves the original job order so the persisted bytes are byte-identical to a natively-built trie2 — verified end-to-end in the tests by comparing the migrated DB against one built directly through trie2.Trie.Update (see TestMigrationEndToEnd).

[codecov]

Codecov Report

❌ Patch coverage is 65.69767% with 177 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.98%. Comparing base (05dd000) to head (df52a01).

Files with missing lines	Patch %	Lines
migration/trie/hashworker.go	36.14%	50 Missing and 3 partials ⚠️
migration/trie/ingestor.go	72.43%	28 Missing and 15 partials ⚠️
migration/trie/trie.go	75.17%	20 Missing and 16 partials ⚠️
migration/trie/counter.go	45.94%	19 Missing and 1 partial ⚠️
migration/trie/hashpool.go	32.00%	16 Missing and 1 partial ⚠️
migration/trie/codec.go	92.72%	2 Missing and 2 partials ⚠️
migration/trie/committer.go	84.61%	1 Missing and 1 partial ⚠️
node/migration.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@                        Coverage Diff                        @@
##           maksym/statehistory-migration    #3659      +/-   ##
=================================================================
- Coverage                          76.10%   75.98%   -0.12%     
=================================================================
  Files                                408      415       +7     
  Lines                              36913    37428     +515     
=================================================================
+ Hits                               28091    28440     +349     
- Misses                              6792     6912     +120     
- Partials                            2030     2076      +46     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.