Port JIT costing from sigmastate-interpreter#854
Draft
mwaddip wants to merge 46 commits into
Draft
Conversation
11 tasks covering cost types, Context changes, eval integration for ~70 operations, type-based costs, method costs, cost propagation, transaction validation, tests, and cleanup. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add CostLimitExceeded error and jit_cost/jit_cost_limit fields to Context with add_jit_cost() and add_per_item_jit_cost() methods. Replace stub Cost/Costs types with JitCost newtype. Wire CostLimitExceeded into EvalError. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add fixed, per-item, type-based, and method-level costs to ~70 eval implementations. Fixed costs added before operation, per-item costs added after computing item count, type-based costs check for BigInt. Method dispatch functions in sbox, sgroup_elem, savltree, scoll, scontext, soption, sheader, spreheader, sglobal all get their costs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
reduce_to_crypto now resets jit_cost, evaluates, and reads accumulated cost into ReductionResult.cost. Verifier forwards reduction_result.cost to VerificationResult.cost. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…l cost validate() now returns Result<u64, TxValidationError> with total script cost. Sets per-script jit_cost_limit from MaxBlockCost parameter. Resets cost between inputs. Add MaxBlockCost to default Parameters. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three tests: trivial prop cost (Constant=5, block cost 0), SELF.value script cost (58 JitCost, block cost 5), and cost limit exceeded error. Also fix reduce_to_crypto to short-circuit on CostError instead of retrying with spanned expressions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Author
Note: relationship to #858 (lazy constant resolution)#858 eliminates the deep clone + tree walk in Whichever merges first, the other needs a small mechanical rebase:
|
validate() previously called context.reset_jit_cost() at the top of every per-input verify loop, and reduce_to_crypto::inner did the same at each invocation. Combined, this meant jit_cost_limit was checked only against a single input's cost — a tx with N inputs each just under the limit would pass despite the total cost exceeding MaxBlockCost, diverging from Scala's sum-then-check semantics (JIT_COSTING_FIX_PLAN.md gap S4). Both resets are removed. reduce_to_crypto now tracks cost as a delta from the caller's accumulator state (cost_before snapshot), so ReductionResult::cost stays per-call for existing consumers while the accumulator grows cumulatively across calls on the same Context. The diagnostic retry path rolls the accumulator back to cost_before so the re-invocation cannot spuriously trip the limit. Regression: test_validate_enforces_cumulative_jit_cost_across_inputs builds a 3-input Const(true) tx (15 JitCost cumulative) with max_block_cost=1 (→10 JitCost limit) and asserts the cumulative check fires on the third input. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ports estimate_crypto_cost from PR 846 as a new pure module and wires it into validate() so the JIT cost accumulator (Phase 1) also includes sigma-protocol verification cost alongside ErgoTree evaluation cost. Addresses JIT_COSTING_FIX_PLAN.md gap S3 (missing crypto cost). ergotree-interpreter/src/sigma_protocol/crypto_cost.rs is a verbatim port from PR 846 (arkadianet/jit-costing). Constants match Scala sigmastate-interpreter's estimateCryptoVerifyCost: ProveDlog=3980, ProveDhTuple=7140, conjunction=15, threshold polynomial terms per the n-k secret-sharing scheme. validate() no longer delegates to verify_tx_input_proof; it inlines the per-input path (update_context → storage-rent bypass → reduce_to_crypto → charge crypto cost → verify_signature → ReducedToFalse detection). This gives us a reference to reduction.sigma_prop between reduction and verification so we can cost the sigma proof. The public helper verify_tx_input_proof is unchanged for external callers (e.g. WASM). Crypto cost is charged through context.add_jit_cost, so it flows through the same cumulative limit check as eval cost — preserving Phase 1's S4 fix. estimate_crypto_cost returns u64 while add_jit_cost takes u32; realistic sigma props are well under u32::MAX (deep thresholds bounded by SigmaConjectureItems' BoundedVec cap of 255), but any overflow is surfaced as a CostLimitExceeded error rather than a silent wrap. Verification: 6 new unit tests in crypto_cost.rs (TrivialProp, ProveDlog, ProveDhTuple, Cand, Cor, Cthreshold), plus the Phase 1 regression test continues to pass since it uses Const(true) scripts that reduce to TrivialProp(true) (crypto_cost = 0, so the cumulative limit check still fires on the third input's eval step). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ports INTERPRETER_INIT_COST + compute_tx_init_cost from PR 846 and wires them into validate() so the per-tx JIT budget also includes the per-input, per-data-input, per-output, and per-token structural costs that Scala charges once before any script runs. Addresses JIT_COSTING_FIX_PLAN.md gaps S1 (INTERPRETER_INIT_COST) and S2 (compute_tx_init_cost). compute_tx_init_cost is a port of PR 846's formula (validated against 19,549 mainnet txs, zero mismatches), which in turn matches Scala's ErgoTransaction.computeInitiationCost. Returns block cost units; the caller multiplies by 10 to arrive at JitCost scale. Worst-case init cost on pathological txs (32767 inputs × 255 tokens) exceeds u32::MAX JitCost, so a new Context::add_jit_cost_u64 method is added alongside the existing add_jit_cost(u32). Per-opcode arms keep the u32 variant; only tx-structural cost uses u64. A new TxValidationError::InitCostExceeded(u64) variant surfaces the case where the init cost alone exceeds the per-tx limit — these txs can't honestly blame any specific input for the overflow. The Phase 1 regression test is updated to accommodate the init-cost floor: structural Parameters are zeroed so init reduces to the fixed INTERPRETER_INIT_COST baseline (100,000 JitCost), then max_block_cost is sized to 10,001 — putting the cumulative limit at init + 2×5 JitCost. Two Const(true) inputs fit, the third overflows on the shared accumulator exactly as S4 requires. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ports DataValueComparer from PR 846 as a new module and wires it into
BinOp's Eq/NEq dispatch. Closes JIT_COSTING_FIX_PLAN.md bugs 4 and 5.
Before this change, Eq/NEq charged no cost (the arm was literally `{}`
with a "Dynamic — no cost" comment), so an attacker could bytecode a
compare of two arbitrarily-large collections for free and the resulting
eval would diverge from Scala on cost numbers. This is bug 4.
Additionally, there was no length-mismatch short-circuit for collection
equality, so the per-item cost would be charged for mismatched-length
colls that Scala rejects immediately after MatchType dispatch — bug 5.
eq_with_cost charges per-type fixed costs (EQ_PRIM=3, EQ_BIGINT=5,
EQ_GROUP_ELEMENT=172, EQ_BOX=6, EQ_AVL_TREE=6, EQ_TUPLE=4, EQ_OPTION=4,
EQ_HEADER=6, EQ_PREHEADER=4) from Scala's DataValueComparer. For
collections it charges MatchType(1) first — paid always, even on length
mismatch — then short-circuits if lengths differ, else charges per-item
cost selected by element type (EQ_COLL_BYTE=(15,2,128), EQ_COLL_INT=
(15,2,64), …). Tuples and Options recurse on their children.
The cost constants live inline in data_value_comparer.rs as raw u32
rather than porting PR 846's FixedCost/PerItemCost typed wrapper system
(which would be a 400-LOC global `costs.rs` rewrite touching every
opcode). This keeps Phase 4's surface to 3 files (+~200, -3).
PR 854's existing arithmetic/Gt/Lt/Ge/Le/logical/bit costs in bin_op.rs
already match PR 846's values numerically, so no changes there.
Regression tests:
- primitive_eq_charges_prim_cost: Int == Int pays exactly 3 JitCost.
- coll_eq_charges_match_type_plus_per_item: Coll[Int] of 3 items pays
MatchType(1) + SInt per-item (15+2*ceil(3/64)=17) = 18 JitCost.
- coll_eq_length_mismatch_short_circuits: Coll[Int] of mismatched
lengths pays only MatchType(1) = 1 JitCost (bug 5 signal).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Before this change, SubstConstants charged per-item JIT cost based on the caller-supplied `positions.len()`, but the actual work (deserializing the template tree and walking its constants list during substitution) is proportional to the template's total constants_len. Closes JIT_COSTING_FIX_PLAN.md bug 3. Scala's ErgoTreeSerializer.substituteConstants walks every constant in the template regardless of how many positions the caller asks to swap, so mirror that: move the cost charge to after the tree is parsed and num_constants is known, and drive it from num_constants instead of positions.len(). Coefficients (base=100, per_chunk=100, chunk_size=1) are unchanged — PR 846's SUBST_CONSTANTS_COST uses the same values. Regression test (subst_const_cost_uses_template_count_not_positions_len): build a 3-constant template, run SubstConstants twice on identical input but with positions=[0] vs positions=[0,1,2]; assert total JIT cost is identical. Before the fix the charge diverged by 200 (positions.len() = 1 vs 3, each chunk at cost 100). Since Expr::Const eval is fixed-cost regardless of payload (eval/expr.rs:22), the two runs have no other divergence source. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Map, Filter, Fold, ForAll, and Exists all invoke a lambda per collection item, which binds the item into the environment. Scala's reference JIT costing charges ADD_TO_ENV_COST (5 JIT) on each of those bindings; sigma-rust previously charged only the collection's per-chunk cost, underpricing scripts that iterate over large collections. Insert `ctx.add_jit_cost(5)?` inside each op's lambda-call closure, between the env-capture and env-insert sites. Charging per invocation (rather than pre-charging 5 * n_items) preserves short-circuit correctness for ForAll and Exists: when the predicate terminates iteration early, the remaining items are neither evaluated nor charged. Add `forall_charges_add_to_env_per_iteration` regression test proving the delta between a 4-invocation full eval and a 1-invocation short-circuit is exactly 3 * (5 env + 5 Const body) = 30 JIT. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Slice previously charged `add_per_item_jit_cost(10, 2, 100, input_vec.len())` before extracting `from`/`until`. That made cost scale with the input collection size — so slicing a 2-item window from a 1000-item input was as expensive as touching all 1000 items, which can push otherwise-cheap scripts past the JIT cost limit. Scala's reference charges the slice cost based on the requested range (`max(0, until - from)`) before clipping it against the input bounds — this is "the work you asked for", regardless of how it ends up clipped. PR 846 follows the same formula. Move the charge to AFTER `from`/`until` extraction and use `max(0, until - from)` for the per-item count. The clipping for the actual returned slice is unchanged. Add `slice_charges_output_not_input_size` regression test asserting (a) the cost is identical for `slice(0, 2)` of a 5-elem vs a 1000-elem input and (b) the cost grows with the requested range size. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A tree whose proposition resolves to a plain `Expr::Const(SSigmaProp)` (bare P2PK — the most common input script on mainnet) was previously evaluated through the generic `Expr::Const` arm and paid only 5 JitCost. Scala's reference interpreter prices this case as a unit via `EvalSigmaPropConstant = 50` — a 10× undercharge on every P2PK input. Add a `trivial_reduce` short-circuit in `reduce_to_crypto` that recognises this pattern after `proposition()` and `substitute_deserialize` run, charges the flat 50 JitCost, and returns the SigmaBoolean directly without calling the full `inner` evaluator. The short-circuit only catches non-segregated SigmaProp constants; segregated trees (where the top-level is still `Expr::ConstPlaceholder`) fall through until the IR carries resolved constants. Add `p2pk_trivial_reduce_charges_50` regression test asserting the JitCost delta for a bare P2PK tree is exactly 50 and the returned SigmaBoolean round-trips the P2PK's ProveDlog. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Scala's reference interpreter costs `ConstantPlaceholder` nodes at 1 JitCost each, distinct from inline `Constant` at 5 JitCost. Pre-fix, sigma-rust erased this distinction at proposition time by rewriting placeholders into `Expr::Const` (via `substitute_constants`), so every segregated tree ended up paying 5 JitCost per placeholder — overcharging relative to Scala by 4 JitCost per constant. On mainnet, many scripts use constant segregation, so this overcharge is common. Lay the IR plumbing to preserve the distinction: * Add `ConstantPlaceholder::resolved: Option<Constant>` so the node can carry its resolved value without being rewritten into a Const. * Populate `resolved = None` on parse and at serialization-write time; it is filled in later by `Expr::resolve_placeholders`. * Add `Expr::resolve_placeholders(constants)` that sets the `resolved` field on every placeholder node in a tree without substituting the node itself — the dual of the existing `substitute_constants`. * Add `ErgoTree::proposition_for_cost_eval()` that routes segregated trees through `resolve_placeholders` instead of `substitute_constants`, preserving the `ConstPlaceholder` nodes for downstream costing. No behavior change yet: all existing callers still use `ErgoTree::proposition()`, which keeps calling `substitute_constants`. The interpreter-side switch to `proposition_for_cost_eval()` lands in Phase 9b. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Scala's reference interpreter costs `ConstantPlaceholder` nodes at 1
JitCost each, distinct from inline `Constant` at 5 JitCost. The 9a
commit laid the IR plumbing to preserve that distinction; this commit
flips the interpreter over to use it.
* `reduce_to_crypto` now calls `tree.proposition_for_cost_eval()`,
which routes segregated trees through `resolve_placeholders` instead
of `substitute_constants`. The root expression carries
`Expr::ConstPlaceholder { resolved: Some(_) }` nodes through to eval.
* The `Expr::ConstPlaceholder` eval arm (previously an error) now
reads `cp.resolved`, charges 1 JitCost, and returns the underlying
value.
* `trivial_reduce` is extended to match segregated P2PK —
`Expr::ConstPlaceholder { resolved: Some(c) }` where
`c.tpe == SSigmaProp` — so both segregated and non-segregated P2PK
scripts short-circuit to `EvalSigmaPropConstant = 50`.
* The pretty-printer now renders resolved placeholders as their
underlying constant, so diagnostic output (e.g. for the
trivial-false retry path) remains readable after the switch.
Test retunes for the arithmetic shift:
* `jit_cost_limit_exceeded` previously set `jit_cost_limit = Some(1)`
and expected the 5-JIT Const eval to trip it. Post-fix the tree's
placeholder costs 1 JIT, which equals (but does not exceed) the
limit. Lowered the limit to 0 so the very first cost addition
trips it.
* `test_validate_enforces_cumulative_jit_cost_across_inputs` hardcoded
`PER_INPUT_JIT = 5` against a segregated Boolean tree. Switched the
tree to non-segregated `SigmaProp(TrivialProp(true))`, which
trivial-reduces at `EVAL_SIGMA_PROP_CONSTANT = 50` — a clean multiple
of 10, so the JIT→block-cost round-trip in `Parameters` lands on the
overflow boundary without integer-division loss.
* `jit_cost_trivial_prop`'s comment updated for the new cost number.
New regression test:
* `segregated_constants_charge_1_not_5_per_placeholder` asserts a
segregated Boolean tree's reduction charges exactly 1 JitCost and
contrasts it with the non-segregated SigmaProp path (50 JitCost via
`trivial_reduce`), locking in the per-node cost distinction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ports PR 846's cost-parity smoke harness and reference vectors for mainnet heights 700000-700050, cross-validating sigma-rust's JIT cost totals against Scala-computed block costs for 78 real transactions. smoke_cost_parity replays the Scala block-cost pipeline explicitly (init + per-input eval + crypto + snap) through reduce_to_crypto, while smoke_validate_parity exercises the shipped TransactionContext::validate() path end-to-end. Both assert zero mismatches against the reference costs in tx_costs_700000_700050.json. Cost deltas are read via ctx.jit_cost_value() (not reduction.cost) to avoid precision loss from sigma-rust storing block cost (JitCost/10 floor) rather than the raw JitCost PR 846 carried in ReductionResult. A #[ignore]d full_corpus_cost_parity harness auto-discovers tx_costs_START_END.json / transactions_START_END.json / headers_..json triples under ERGO_COST_VECTORS_DIR for running the larger corpora PR 846 shipped separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each ValDef insertion into the interpreter environment now pays a flat 5 JitCost on top of the BlockValue per-item base, matching Scala's AddToEnvironment step. The ADD_TO_ENV_COST constant is already charged per lambda invocation in Map/Filter/Fold/ForAll/Exists (per-iteration env bind); extending it to ValDef closes the remaining source of env-bind undercounting. Observed on mainnet tx 518acecdd776ca50508f122abeec9f888f7efbc5ffa5fe7cb0729c446b533bf4 at height 700032: 4 ValDefs in its top-level BlockValue produced a -20 JitCost (-2 block cost) delta against Scala prior to this fix. The cost_parity harness (heights 700000-700050) now reports 78/78 exact matches on both smoke_cost_parity and smoke_validate_parity. Adds block_value_charges_add_to_env_per_val_def regression: asserts the per-ValDef delta equals Const-rhs (5) + ADD_TO_ENV (5) = 10 JIT, comparing a 4-ValDef block against a 1-ValDef block to isolate the per-item contribution from the (constant) BlockValue base. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Author
Update to the #858 rebase guidanceThe integration branch ( If #858 merges first → this PR rebases as follows:
The integration commit that captures this delta: If this PR merges first → #858 rebase: unchanged from the pre-existing note on that PR, with one correction: the ConstPlaceholder handler should charge 1 JitCost (matching Phase 9), not 5. |
Consolidates `add_jit_cost` (u32) and `add_jit_cost_u64` into a single `add_jit_cost(amount: u64)` with saturating addition. Since this API was introduced in PR 854 and has no pre-existing external users, the u32/u64 split carried no real benefit — it was just unnecessary ceremony around "is this contribution bounded by input structure or by tx structure". The u64 version's saturating-add is safer than the u32 version's wrapping-add (well-formed inputs can't hit it, but pathological cases could), so unify on that semantics. Ripple changes: * `Context::add_per_item_jit_cost` internally computes cost as u32 (base + chunks * per_chunk — the product is bounded by small cost constants × collection sizes) and widens to u64 at the final `add_jit_cost(u64::from(cost))` call. Keeps the u32 argument types for call-site ergonomics. * Per-opcode cost constants that were `u32` are now `u64`: `EVAL_SIGMA_PROP_CONSTANT` in `eval.rs`, and the ten `EQ_*_COST` / `COLL_MATCH_TYPE_COST` constants in `data_value_comparer.rs`. No literal call sites change — integer literals infer as u64 from the param type. * `tx_context.rs::validate()`: `add_jit_cost_u64(init_cost_jit)` becomes plain `add_jit_cost(init_cost_jit)`; the crypto-cost path drops the `u32::try_from` dance entirely and passes `estimate_crypto_cost`'s u64 return value straight through. The overflow-to-CostLimitExceeded fallback is now covered by the unified `add_jit_cost`'s own limit check — if the sum exceeds the limit, it returns `CostLimitExceeded` directly. Removes two unused imports (`CostLimitExceeded`, `EvalError`). Behavior-preserving: cost-parity smoke harness still 78/78 vs Scala, Phase 1 cumulative-limit regression passes unchanged, full workspace green, C bindings build clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…temCost Context::add_per_item_jit_cost computed chunks with ceiling division `(n + cs - 1) / cs`, which yields 0 chunks for an empty collection and so charged only `base`. Scala consensus PerItemCost.chunks is `(n - 1) / chunkSize + 1` with signed Int division: at n=0 that is 1 for chunkSize>=2 and 0 for chunkSize==1, and CErgoTreeEvaluator.addSeqCost charges `cost(nItems)` with no empty guard. So the JVM charges `base + per_chunk` for an empty Coll[T] of a chunkSize>=2 element type while sigma-rust charged only `base`. JIT cost feeds exactly one consensus check, the MaxBlockCost limit, so an undercharge lets a node accept a block the JVM rejects (miner-constructible fork risk). Observed on mainnet block 1,520,814 tx[12] input 0 (BigInt DEX script): 2 empty Coll[Long] equalities charged 15 each (base only); the JVM charges 17 each (base 15 + per_chunk 2). Replicate Scala's signed `(n-1)/cs + 1` using i64 to avoid the u32 underflow at n=0. For every n>=1 this equals `ceil(n/cs)`, so all non-empty per-item charges are byte-identical and only n=0 moves. Corrects all 29 add_per_item_jit_cost call sites at once (Fold / ForAll / Filter / Map / Slice / Append / AtLeast / XOR / hashing / sigma ops / per-type equality). Apply the same formula to the dead-code per_item_cost helper to keep it in lockstep, and drop a stale useless #[allow(dead_code)] on costs.rs's extern crate (clippy useless_attribute). Regression tests in data_value_comparer: empty Coll of a chunkSize>=2 type (Long/Int/Short/Boolean/BigInt/AvlTree/Byte) charges one chunk; empty Coll of a chunkSize==1 type (Box/GroupElement/Header/PreHeader) charges base only; non-empty Coll[Long] across a chunk boundary (len 1/48/49) unchanged. Verification: cost-parity smoke harness 78/78 vs Scala, full ergotree-ir + ergotree-interpreter suites green (358 + 8 signing), lib clippy clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lock the n=0 chunks fix (e11d418) on two more per-item op classes, cross-validated by the santa eval fixtures against sigma-state 6.0.3 under activated=3: - `and_empty` (tree 00960d00): And over an empty Coll[Boolean] constant. Const(empty) 5 + And per-item base 10 + 1 chunk * per_chunk 5 = 20 (was 15 before the n=0 fix; cs=32 >= 2 so chunks(0) moved 0 -> 1). - `calc_blake2b256_empty` (tree 00cb0e00): hash of an empty Coll[Byte] constant. Const(empty) 5 + per-item base 20 + 1 chunk * per_chunk 7 = 32 (was 25 before the n=0 fix; cs=128 >= 2). Same root cause as the empty-Coll equality regressions already covered in data_value_comparer; these pin the And and hashing call sites end-to-end. Zero-tolerance vs the JVM. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A ConcreteCollection of N boolean constants undercharged JIT cost vs the JVM/Scala consensus: sigma-rust charged only the ConcreteCollection Fixed(20), while the JVM also charges Constant.costKind = FixedCost(5) per BooleanConstant element. In sigma `values.scala`, ConcreteCollection.eval charges Fixed(20) then evaluates each item via `evalTo`, and Constant.costKind = FixedCost(JitCost(5)) (:380), so an N-bool collection costs 20 + 5n on the JVM but 20 in sigma-rust. This affected only the packed `BoolConstants` representation: the `Exprs` arm already pays 20 + 5n because each element `Expr::Const` eval charges 5 (expr.rs). The packed form converts the bits straight to a Value (`bools.clone().into()`) without per-element eval, skipping the N*5. NOT a ConcreteCollection-PerItemCost divergence: ConcreteCollection is FixedCost(JitCost(20)) in Scala (values.scala:878), confirmed; making it per-item would double-count the Exprs arm. Fix at the source: charge `5 * bools.len()` in the BoolConstants arm so both representations and the JVM agree (empty -> 20, n -> 20 + 5n). Undercharging feeds the MaxBlockCost consensus check (same fork class as the n=0 chunks bug). Surfaced by the santa eval fixture `coll_bool_constants_3` (tree 00850305): JVM 35 vs sigma-rust 20. Regression vectors: empty -> 20, 3 -> 35, 5 -> 45. Verification: cost-parity smoke harness 78/78 vs Scala, full interpreter suite green (361 + 8 signing), lib clippy clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Author
|
Pushed 3 commits aligning per-item JIT costing with the Scala reference at empty/edge inputs:
Cost-parity smoke harness still 78/78 vs Scala; full suite green. |
Author
|
Integration note: the mismatched-numeric arithmetic coercion (#869 vs develop — coerce Cost-bearing version staged: https://github.com/mwaddip/sigma-rust/tree/jit-cost-arith-coercion (= this PR + the coercion + its cost: per-op cost keyed on the wider type + the Upcast cost, bigint=30 else 10). Verified — 14 sweep vectors bit-exact (value + cost 35/25/60 + accept/reject), interpreter suite + parity smoke green. Fold in at integration, or fork it for a PR. |
Apply::eval binds each of a lambda's arguments into the interpreter environment but charged nothing for those insertions -- only the Apply=Fixed(30) base. Scala's reference JIT costing charges ADD_TO_ENV_COST (5 JIT) on every env insertion (in FuncValue.eval's returned closure: env1 = env + (arg -> vArg)), the same charge already applied to BlockValue ValDefs and per-item coll-op lambda bindings. Every user-lambda application therefore undercharged by 5 per bound argument: a single generic Apply was 5 JIT short, and a higher-order lambda chaining 5 applications was 25 short. Convert the binding loop from .for_each to a for loop so the per-insertion cost-limit error propagates via `?`, and charge ctx.add_jit_cost(5) before each env.insert, mirroring block.rs. The save/restore of shadowed variables is unchanged. The per-item coll-op path (Map/Filter/Fold/ForAll/Exists) binds its lambda arg through its own closure and never routes through Apply::eval, so it is not double-charged. Add apply_charges_add_to_env_per_arg_binding regression test proving the JIT delta between a 2-arg and a 1-arg application is exactly arg Const (5) + ADD_TO_ENV_COST (5) = 10 (pre-fix would be 5). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SigmaPropBytes charged a flat PerItemCost(35, 6, 1) with n hardcoded to 1, before evaluating its input. sigma-state's `SigmaPropBytes.eval` evaluates the input first, then charges that PerItemCost over `numNodes = wrappedValue.size` -- the SigmaBoolean node count. Every multi-node proposition was therefore undercharged: a ProveDHTuple alone counts as 4 nodes (one per EcPoint), and CAND/COR/CTHRESHOLD scale with their children. Surfaced by SANTA's v5 propBytes vectors (sigma-rust flat 111 vs JVM 111->321 across ProveDHTuple/CAND/CTHRESHOLD/COR). Add `SigmaBoolean::size` mirroring sigma-state's `SigmaBoolean.size` (ProveDlog/TrivialProp = 1, ProveDHTuple = 4, conjecture = 1 + children sizes) and charge `add_per_item_jit_cost(35, 6, 1, size)` after the input eval, matching the JVM's order. Tests: `SigmaBoolean::size` proptests (ProveDlog = 1, ProveDHTuple = 4) + a conjecture node-count case; `propbytes_cost_scales_with_node_count` asserts the ProveDHTuple-vs-ProveDlog cost delta is 6*(4-1) = 18. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Coll.flatMap charged PerItemCost(60, 10, 8) over the input collection length, before running the per-item lambda. sigma-state's `flatMap_eval` charges that PerItemCost over `res.length` -- the flattened OUTPUT length -- after building the result. flatMap output is typically much larger than its input, so sigma-rust under-charged (SANTA v5 flatMap vectors: 164 vs 329, delta up to -190). Build the flattened result first, then charge over its length, matching the JVM's count and ordering (after the per-item lambda calls). Value is unchanged (existing `eval_flatmap` test); cost parity is covered by the SANTA flatMap vectors + the cost-parity harness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Coll.indexOf charged PerItemCost(20, 10, 2) over the full collection length, up-front, ignoring `from` and where the element is found. sigma-state's `indexOf_eval` charges over `i - start` -- the iterations actually performed, scanning from `max(from, 0)` to the found index (inclusive) or the end. sigma-rust therefore over-charged when the element was found early and mischarged for non-zero `from` (SANTA v5 indexOf vectors: 12 entries, mixed-sign delta +/-3..16). Scan first, then charge over the iterations performed: `found - start + 1`, or `len - start` when not found. Values are unchanged (existing `eval_index_of` test); new `index_of_cost_scales_with_iterations` asserts the found-at-15 vs found-at-0 delta is (20+10*8)-(20+10*1) = 70 (was 0 pre-fix). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`eq_with_cost`'s Coll arm bulk-compared all elements via PartialEq after charging the collection's per-item cost. sigma-state's `DataValueComparer` only bulk-compares COA leaf-element collections (`equalCOA_*`); collections of composite elements (Coll/Tuple/Option/ SigmaProp) fall to generic `equalColls`, which recurses `equalDataValues` per element -- charging the nested MatchType + per-item each time, and short-circuiting on the first inequal element. sigma-rust therefore under-charged nested collection/tuple equality (SANTA v5 NEQ vectors: delta -1..-18). Recurse `eq_with_cost` per element for composite-element colls (the `coll_eq_cost` default arm, via new `is_coa_coll`); keep the bulk compare for COA leaves. Uses only the existing per-type cost constants, applied recursively. Mechanism cross-checked against the ergots fix (same COA set + recursion, JVM-validated by SANTA). New `nested_coll_eq_recurses_element_cost`: Coll[Coll[Int]] [[1,2,3],[4,5,6]] vs itself charges 51 (outer 15 + two inner 18); pre-fix (no recursion) was 15. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Author
PR 854 update notes — running logAccumulated notes for the PR description update after all fix-plan phases are done. Architectural findingsPR 846 does NOT enforce cumulative PR 854 + Phase 1 is strictly more correct on S4 than PR 846. After Phase 1, Known caveats to call out in the final PR description
Phase-by-phase status
|
flatmap_eval bound the lambda argument (env.insert) without charging the ADD_TO_ENV_COST(5) that every FuncValue application pays on the JVM (AddToEnvironmentDesc, values.scala:1047). coll_map and the other HOFs (filter/fold/exists/forall) all charge it per input element; flatMap was the lone omission, under-charging by 5 per input element. Separate from the output-length per-item charge (d404de5) already landed. Charge ctx.add_jit_cost(5)? before the binding, mirroring coll_map.rs. New flatmap_charges_add_to_env_per_input: the cost delta between a 1- and 2-element input equals one lambda-body eval plus 5 (the per-input ADD_TO_ENV); pre-fix it was just the body eval. Values unchanged (existing eval_flatmap). cost-parity smoke 2/2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The jit-costing cost-model commits were never run through `cargo fmt --all` or `cargo clippy --all-features --all-targets`, so the branch failed the CI rustfmt and clippy jobs. Fixed here with no production logic changes. rustfmt --all: bin_op, coll_slice, data_value_comparer, scoll, subst_const, bindings/ergo-lib-wasm/transaction. clippy --all-targets -D warnings: add #[allow(clippy::unwrap_used)] to the data_value_comparer / crypto_cost / block / ergotree-ir sigma_boolean test modules (matching the existing test-mod convention); drop two redundant `as u64` casts and two useless `.into()`; enumerate the VerifierError arms in the tx_context cost-limit proptest; fix needless-borrow, needless-range-loop and orphaned-doc lints in the new cost_parity test. No production logic changed. Suites green: ergotree-interpreter, ergotree-ir, ergo-lib cost_parity 2/2, tx_context proptests 8/8. fmt --all --check and clippy --all-features --all-targets -D warnings both clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
INDEX_OF_EVAL_FN scanned with a bare `==` (PartialEq), leaving the per-comparison cost uncharged. sigma-state indexOf_eval calls DataValueComparer.equalDataValues(xs(i), elem) on each scan step, charging the element-type equality cost (EQ_PRIM=3, EQ_BIGINT=5, EQ_GroupElement=172, ...). sigma-rust therefore under-charged indexOf by that per-element eq cost (SANTA v5 Coll_indexOf vectors). Route each scan comparison through the cost-charging eq_with_cost (data_value_comparer, also used by Eq/NEq), which post-84676c07 recurses correctly for composite elements. The PerItemCost(20,10,2) iteration charge (6b48aff) is unchanged -- JVM charges both. index_of_cost_scales_with_iterations now reflects the added per-compare eq cost (delta 70 -> 115 = PerItemCost 70 + EQ_PRIM 3*15). New index_of_charges_element_eq_cost isolates the eq charge by element type (Coll[Long] vs Coll[BigInt], 1 comparison): delta = EQ_BIGINT - EQ_PRIM = 2. Values unchanged (eval_index_of); cost-parity smoke 2/2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
updateMany's PerItemCost charged perChunkCost=1, but Scala consensus (sigma-state 6.0.3, SCollectionMethods.UpdateManyMethod, ast/methods.scala) specifies PerItemCost(baseCost=20, perChunkCost=2, chunkSize=10). The arm was copy-pasted from UPDATED_EVAL_FN (which is legitimately perChunk=1) -- the leftover "updated:" error strings in the arm confirm the origin. Undercharged by chunks(n)=(n-1)/10+1 per call: 1 for n<=10, 2 for 11<=n<=20. Matches the SANTA v5 blessed vectors (Coll_updateMany_method_equivalence) expected.cost: 158 for n<=3 and 160 for n=14 (sigma-rust previously emitted 157/158). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The numeric 6.0 method eval fns (toBytes/toBits/bitwiseInverse/Or/And/Xor/shiftLeft/Right/toUnsigned) computed their value but never charged the per-method cost, undercharging every call by 5 JitCost.
Scala assigns each FixedCost(JitCost(5)) (SNumericTypeMethods {ToBytes,ToBits,BitwiseOp}_CostKind; SBigIntMethods ToUnsignedCostKind), and every other method family already charges its costKind inside its own eval fn -- which is why v5 cost stayed exact. Closes the systematic v6 cost undercount across the Long/Int/Short/Byte/BigInt 6.0 methods (and the Global.fromBigEndianBytes vectors, which compose x.toBytes).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both charged a flat 10 JitCost. Scala assigns Global.encodeNbits EncodeNBitsCost = FixedCost(JitCost(25)) and Global.decodeNbits DecodeNBitsCost = FixedCost(JitCost(50)) -- the v6 -15 / -40 undercharges SANTA surfaced. Includes a regression test isolating each method's costKind (decode-minus-encode delta = 25). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Global.powHit charged a flat 900 JitCost; Scala's PowHitCostKind is input-dependent: baseCost(500) + (k+1) * (totalLen/chunkSize + 1) * perChunkCost, where chunkSize=128 and perChunkCost=7 come from CalcBlake2b256's costKind (the heaviest part is k+1 Blake2b256 invocations over msg||nonce||h). For Autolykos k=32 / short message this is 731, matching the JVM -- the flat 900 was the +169 overcharge SANTA surfaced. Cost is now charged after extracting the args it depends on; adds a k-delta regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ndsWith reverse charged a flat JitCost(20); Scala reverse_eval charges Append.costKind = PerItemCost(20, 2, 100) over the receiver xs.length (methods.scala 1124), a -2 undercharge on the v6 cost vectors. startsWith/endsWith likewise charged a flat 20; Scala uses Zip_CostKind = PerItemCost(10, 1, 10) over xs.length (methods.scala 1143/1163), a +9 overcharge each. Charge the per-item costKind over the receiver length to match addSeqCost; add_per_item_jit_cost already replicates Scala's chunks() incl. chunks(0)=1. Adds scoll_methods_charge_scala_costkinds, isolating each method's MethodCall Fixed(4) + costKind via the subtract-receiver/arg-eval pattern. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The UnsignedBigInt modular eval fns (toUnsignedMod, modInverse, plusMod, subtractMod, multiplyMod, mod, toSigned) took _ctx and charged nothing. Scala assigns each a FixedCost (methods.scala 551-609): toUnsignedMod 15, modInverse 150, plusMod 30, subtractMod 30, multiplyMod 40, mod 20, toSigned 10. Charge each per call, mirroring the numeric NUMERIC_METHOD_COST_KIND pattern. Latent until now: the v6 UnsignedBigInt surface was unrepresentable to the conformance runner until the SValue bridge landed, so these never surfaced in the 182 measured cost divergences. Adds unsigned_bigint_modular_methods_charge_scala_costkinds, isolating each method's MethodCall Fixed(4) + costKind via the subtract-receiver/arg-eval pattern. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SERIALIZE_EVAL_FN charged a flat JitCost(10) -- exactly Scala SigmaByteWriter.StartWriterCost -- and dropped every per-put cost accumulated during DataSerializer.serialize. Mirror Scala serialize_eval per-put callbacks (SigmaByteWriter.scala:235-262): StartWriterCost 10, PutByteCost 1, Put{Signed,Unsigned}NumericCost 3, PutChunkCost PerItemCost(3,1,1) => 3 + n.
Add an optional cost sink on SigmaByteWriter (gated -- None for all non-serialize callers, so normal serialization is unaffected) and record each put cost at its site in DataSerializer::sigma_serialize and BigInt256::sigma_serialize. SERIALIZE_EVAL_FN enables tracking, then charges the accumulated per-put total on top of StartWriterCost.
Covers directly-serialized types (Boolean/Byte/Short/Int/Long/String/Coll[Byte]/Coll[Bool]/Coll[_]/Tuple/Option) plus BigInt. Delegated nested serializers (GroupElement, SigmaProp, UnsignedBigInt, AvlTree, Box, Header) stay unmetered pending blessed JVM vectors -- latent only (v6 off mainnet).
Matches blessed JVM v6 vectors: Byte 90, Short/Int/Long 92, (Long,Long) 95, Coll[Byte] 95 empty / 98 for 3 bytes, BigInt roundtrip 247/310. Adds serialize_charges_writer_costkinds, isolating each value serialize cost via the subtract-arg-eval pattern.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…UnsignedBigInt Option-2 Phase A1 (after Option 1 direct types): meter the delegated nested serializers reachable at the ergotree-ir level. EcPoint (ergo-chain-types, ScorexSerializable) writes one GROUP_SIZE(33)-byte block -> PutChunkCost(33)=36, recorded at the data.rs GroupElement arm and in ProveDlog/ProveDhTuple (EcPoint cannot reach the cost sink itself). SigmaBoolean adds a 1-byte op code (PutByteCost) for every variant. UnsignedBigInt mirrors the BigInt256 fix (putU16 + putBytes). All gated by the cost sink, so non-serialize serialization is unaffected. Reproduces SANTA blessed JVM v6 totals: GroupElement 125, SigmaProp 126, UnsignedBigInt 95/96/127, Coll[GroupElement] 92/128/164 (+36/element). Extends serialize_charges_writer_costkinds with GroupElement + UnsignedBigInt deltas. Box/Header (Phase A2/B) remain unmetered pending their own pinning + (for AvlTree/Header) blitzen runner representability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Option-2 Phase A2a: meter ErgoBox box-level puts during Global.serialize. serialize_box_with_indexed_digests + ErgoBox::sigma_serialize: BoxValue put_u64 (numeric), ergoTree pre-serialized block (PutChunkCost over its length), creation height put_u32 (numeric), token count put_u8 (byte), register count put_u8 (byte), txId Digest32 block (chunk 32), index put_u16 (numeric). All gated by the cost sink, so normal box/tx serialization is unaffected. Reproduces SANTA blessed minimal Box (139); (Box,Int) follows via tuple recursion + Option-1 Int metering (142). Absolute cost verified by ./conform (no self-contained box fixture in the unit test -- box/ErgoTree construction is heavyweight; the per-put mechanism is already unit-tested in Option 1 / A1). Deferred (latent, no blessed vector): the tokens-loop body (tokenId chunk + amount) and the register type prefix (SType serialization). The type prefix is Phase A2b, which closes withR4 (143) / (Box,Int) withR4 (146). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Option-2 Phase A2b: meter the type-code byte in TypeCode::sigma_serialize (every serialized type code is one PutByteCost byte). During Global.serialize this fires for Constant type prefixes -- i.e. box register types -- completing the withR4 box cost (the register data was already Option-1 metered; the type prefix was the missing +1). Gated by the cost sink, so all non-serialize type serialization is unaffected. Closes SANTA blessed Box withR4 (143) + (Box,Int) withR4 (146); verified via ./conform. Residual latent gap (no blessed vector): embedded-type fast paths in SType::sigma_serialize (Option[prim]/Coll[prim]/tuple-pair) emit a combined type code via a direct put_u8 not routed through TypeCode, so such register types undercharge by 1 byte. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Meter AvlTreeData::sigma_serialize to match Scala's AvlTreeData.serializer under Global.serialize (CoreByteWriter): the 33-byte ADDigest putBytes is PutChunkCost(33)=36, the treeFlags putUByte is PutByteCost=1 (charged via the virtual put), and the valueLengthOpt putOption tag byte is PutByteCost=1. keyLength and the Some valueLength use Scala's no-info putUInt, which writes to the underlying writer directly and is unmetered. Total put cost is 38 for both the empty and with-value-length AvlTree, matching the blessed v6 cost 127. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SType::sigma_serialize emits a single combined type-code byte for embedded compound register types via a direct put_u8 that bypasses the metered TypeCode::sigma_serialize, leaving that byte unmetered — so a box register of such a type undercharges Global.serialize by 1. Meter the five reachable fast-path sites (COLL+prim, NESTED_COLL+prim, TUPLE_PAIR_SYMMETRIC, TUPLE_PAIR1, TUPLE_PAIR2) with add_put_byte_cost, matching the blessed v6 Global.serialize[Box] register entries: Coll[Byte] 178, Coll[Int] 152, Coll[Coll[Byte]] 151, (Int,Int) 146, (Int,Long) 147, (Coll[Byte],Int) 154. The OPTION/OPTION_COLL fast paths are left unmetered: Options are not data-serializable (CheckSerializableTypeCode), so unreachable via serialize. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ErgoBoxCandidate serialization charged a numeric put cost for creationHeight, but Scala writes it via the no-info putUInt (ErgoBoxCandidate.scala:143) which delegates to the underlying writer and is unmetered (unlike putULong/putUByte). This was a +3 base over-count on every serialized box, making all Global.serialize[Box] and [(Box,Int)] entries cost-coal (minimal 142 vs blessed 139). Removing the charge — together with the already-landed fast-path register type-code metering — lands the box family on the blessed costs. Adds serialize_box_body_does_not_charge_creation_height since box cost was previously only checked via ./conform. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports the JIT cost model from sigmastate-interpreter to ergotree-interpreter. validate() sums cost across all inputs against the tx cap via a shared Context, closing the gap where #846's per-input check lets a malicious tx distribute cost across sub-cap inputs.
Beware that there's some conflicts with #858, explained in comments below.