Upgrade VM, crypto, and rand dependencies#3146
Conversation
| @@ -1 +1,4 @@ | |||
| #![cfg_attr(not(feature = "concurrent"), no_std)] | |||
There was a problem hiding this comment.
Do we need to run benchmarks for no_std or why is this needed? I assume this has something to do with all dependencies being made optional as well?
There was a problem hiding this comment.
The issue is Cargo feature unification. The transaction benchmark uses the concurrent proving path. Since 0xMiden/crypto#1073, miden-lifted-stark/concurrent is part of the default crypto stack. In crypto 0.27, miden-crypto/concurrent enables std. On the protocol side, miden-tx/concurrent and miden-processor/concurrent also require std.
That works for the benchmark, but it must not leak into cargo build --no-default-features --target wasm32-unknown-unknown --workspace --lib. So the feature gate keeps normal benchmark builds as the default through default = ["concurrent"], while the no-default library build does not enable the std only benchmark stack.
There was a problem hiding this comment.
It seems like a large part of the changes are related to packages and the way we assemble MASM. Does this not result in many conflicts with #2896? That workstream currently has 4 open PRs. So it seems it would make sense to do this migration after that issue was resolved, wdyt?
There was a problem hiding this comment.
Yes, this PR is already doing part of issue #2896. The VM 0.24 upgrade removes the old shape of several Library and Program APIs, so this PR has to move some code to Package and MastForest. That is visible in account component code, note scripts, transaction scripts, and the build scripts.
The part this PR does not do is the project file migration. It adds no miden-project.toml files, and it does not move protocol or standards assembly to the declarative project layout from #3094 and #3107.
I think this leaves us with an ordering choice. If we want the dependency upgrade now, this PR can land first and the project file PRs can rebase over it. If we want less churn in the package migration work, this PR can wait and I can rebase it after #3094 and #3107.
There was a problem hiding this comment.
Looking through this PR, I think I agree with @bitwalker that it would be better to mere #3094 and #3107 first.
This could work well with timing as I think @huitseeker you are out tomorrow (Wednesday).
ce78962 to
f09c095
Compare
There was a problem hiding this comment.
Looks good! Really like the new import syntax. There are still many places where the imports can be consolidated. I opened an issue in the VM for adding unused import checking. I think the new import syntax makes it easier to have unused imports.
For the most part looks good so far, I didn't get through everything just yet, will re-review in 24-48h, or feel free to re-request my review if you address things sooner.
| @@ -1,4 +1,4 @@ | |||
| use agglayer::bridge::bridge_in -> bridge | |||
| use agglayer::bridge::bridge_in as bridge | |||
| pub use { | ||
| FUNGIBLE_ASSET_MAX_AMOUNT, | ||
| ASSET_SIZE, | ||
| ASSET_VALUE_MEMORY_OFFSET, |
| use { | ||
| ACCOUNT_GET_ID_OFFSET, | ||
| ACCOUNT_GET_NONCE_OFFSET, | ||
| ACCOUNT_GET_INITIAL_COMMITMENT_OFFSET, | ||
| ACCOUNT_COMPUTE_COMMITMENT_OFFSET, | ||
| ACCOUNT_GET_CODE_COMMITMENT_OFFSET, | ||
| ACCOUNT_GET_INITIAL_STORAGE_COMMITMENT_OFFSET, | ||
| ACCOUNT_COMPUTE_STORAGE_COMMITMENT_OFFSET, | ||
| ACCOUNT_GET_INITIAL_VAULT_ROOT_OFFSET, | ||
| ACCOUNT_GET_VAULT_ROOT_OFFSET, | ||
| ACCOUNT_GET_ITEM_OFFSET, | ||
| ACCOUNT_HAS_STORAGE_SLOT_OFFSET, | ||
| ACCOUNT_GET_INITIAL_ITEM_OFFSET, | ||
| ACCOUNT_GET_MAP_ITEM_OFFSET, | ||
| ACCOUNT_GET_INITIAL_MAP_ITEM_OFFSET, | ||
| ACCOUNT_GET_ASSET_OFFSET, | ||
| ACCOUNT_GET_INITIAL_ASSET_OFFSET, | ||
| ACCOUNT_GET_NUM_PROCEDURES_OFFSET, | ||
| ACCOUNT_GET_PROCEDURE_ROOT_OFFSET, | ||
| ACCOUNT_HAS_PROCEDURE_OFFSET, | ||
| } from miden::protocol::kernel_proc_offsets |
There was a problem hiding this comment.
When experimenting with this branch locally, I noticed its easy to add imports with this new import syntax, and not actually use the imported procedure or const. I opened an issue in the VM to discuss.
There was a problem hiding this comment.
Yeah that is a bug if those aren't being used anywhere - we do have unused import checking implemented, but it may be buggy still.
There was a problem hiding this comment.
For the record the imports being commented on in this comment are used.
a2502cf to
b2a6275
Compare
bitwalker
left a comment
There was a problem hiding this comment.
I only looked over the assembly-related bits, but looks good to me, aside from a couple notes.
There was a problem hiding this comment.
There was a problem hiding this comment.
After the rebase the project layout is in the base branch, but I would still focus this PR on the dependency upgrade.
The build scripts still contain compatibility work needed to get VM 0.24 packages generated and copied in the current tree. We can replace more of this with the project assembler and seeded package registry. as a follow-up.
b2a6275 to
1c2a973
Compare
| self.mast_store.insert(TransactionKernel::library().mast_forest().clone()); | ||
| self.mast_store.insert(program.mast_forest().clone()); |
There was a problem hiding this comment.
IIUC, this used to load the debug info in the previous VM versions, and will now fall back to LoadedMastForest::new(...) which no longer contains debug info
There was a problem hiding this comment.
and similarly in other users of TransactionMastStore::insert
There was a problem hiding this comment.
execute_code now inserts the kernel and assembled program through insert_package, so package debug info is preserved instead of falling back to a raw LoadedMastForest::new(...). Account code now uses load_account_code, and package values use insert_package. I kept insert for raw MastForest callers and documented that it is the MAST-only path.
| /// Registers all procedures of the provided [Package] with this store. | ||
| fn insert_package(&self, package: &Package) { | ||
| self.insert_loaded(loaded_mast_forest_from_package(package)); | ||
| } |
There was a problem hiding this comment.
The key function here seems to loaded_mast_forest_from_package which adds the debug info.
It might be worth marking this explicitly in this fn's docs
There was a problem hiding this comment.
loaded_mast_forest_from_package now has a doc comment saying it builds a loaded MAST forest from a package and includes package-owned debug info when trusted.
| /// Registers all procedures of the provided [MastForest] with this store. | ||
| pub fn insert(&self, mast_forest: Arc<MastForest>) { | ||
| self.insert_loaded(LoadedMastForest::new(mast_forest)); | ||
| } |
There was a problem hiding this comment.
similarly here (and IIUC) this fn will NOT add debug info, whereas previously it would have by default, because the debug info lived on the MastForest object
There was a problem hiding this comment.
I documented insert as the MAST-only path and made insert_package the explicit package path for preserving package debug info.
| #[must_use] | ||
| pub fn with_debug_mode(mut self) -> Self { | ||
| self.exec_options = self.exec_options.with_debugging(true); | ||
| pub fn with_debug_mode(self) -> Self { | ||
| self | ||
| } |
There was a problem hiding this comment.
Yes, it was dead after the VM API change. Removed the no-op debug mode path and the testing feature that only fed it.
20e523d to
b008e4e
Compare
PhilippGackstatter
left a comment
There was a problem hiding this comment.
Didn't get through everything yet, but left a few comments.
| pub struct TransactionScript { | ||
| mast: Arc<MastForest>, | ||
| entrypoint: MastNodeId, | ||
| package_debug_info: Option<Arc<PackageDebugInfo>>, | ||
| } |
There was a problem hiding this comment.
I think including package debug info in the script is the right approach, as providing this via a side-channel would make it almost impossible to do this (the source manager side channel already makes this very difficult).
There is unfortunately a regression in error message quality relative to next. No need to fix this in this PR, but would be great to get some insight into what we need to do in a follow-up to get back to the old state.
For example, add push.0 assert.err="test message" to the tx script in proven_tx_storage_maps_matches_executed_tx_for_new_account and execute that test. On next we get:
thread 'kernel_tests::tx::test_account_update::proven_tx_storage_maps_matches_executed_tx_for_new_account' (327070) panicked at crates/miden-tx/src/errors/mod.rs:143:22:
expected TransactionExecutorError::Unauthorized, got failed to execute transaction kernel program:
x assertion failed with error message: test message
,-[::$exec:7:18]
6 | begin
7 | push.0 assert.err="test message"
: ^^^^^^^^^^^^^^^^^^^^^^^^^
8 | # Update an existing key.
`----
help: assertions validate program invariants. Review the assertion condition and ensure all prerequisites are met
On this branch we get:
thread 'kernel_tests::tx::test_account_update::proven_tx_storage_maps_matches_executed_tx_for_new_account' (334019) panicked at crates/miden-tx/src/errors/mod.rs:143:22:
expected TransactionExecutorError::Unauthorized, got failed to execute transaction kernel program:
x assertion failed with error code: 7749212804582177978
I think to fix this we need to change CodeBuilder::compile_tx_script to use TransactionScript::from_package which should read the debug info, but also to bring back the debug mode of the TransactionExecutor in a different form, i.e. to use one of the debug APIs of the processor, afaiu.
| fn decode_package_debug_info(package: &Package) -> Option<Arc<PackageDebugInfo>> { | ||
| match package.debug_info() { | ||
| Ok(debug_info) => debug_info.map(Arc::new), | ||
| Err(PackageDebugInfoError::UntrustedSections) => None, | ||
| Err(_) => None, | ||
| } | ||
| } |
There was a problem hiding this comment.
Why does a getter like Package::debug_info return Result? Can Package not enforce that it contains only valid debug info? Propagating this to the caller and then handling it by ignoring the error seems suboptimal. It implies that the maintainers need to have deep knowledge of what error is returned when to be able to say with confidence that we can map it in this way. The API should be better for the user's sake, imo.
There was a problem hiding this comment.
Package cannot enforce this in general, because debug info is stored in custom sections - so you can deserialize a Package but never deserialize the custom sections it contains. When you request the debug info though, we have to deserialize that data from the custom sections, and at that point we may discover that the contents are missing (no custom section for debug info present), or invalid in some way (inconsistent/incorrect schema/etc.). While me might be able to special-case validating debug info specifically during deserialization, it's not essential to execution, so I think it makes sense to defer to that to the point where you actually are asking for it, and can decide how you want to handle invalid debug info then.
To be clear, callers don't have to really have deep knowledge of what the error variants mean - they only really need concern themselves with how they want to handle failure to extract debug info (i.e. proceed without it, or treat it as a hard error).
| .build_tx_context(account.id(), &[], &[])? | ||
| .extend_expected_output_notes(vec![RawOutputNote::Full(output_note)]) | ||
| .tx_script(tx_script) | ||
| .disable_debug_mode() | ||
| .build() | ||
| } |
There was a problem hiding this comment.
I don't know if it makes sense to remove the disable_debug_mode now, because I imagine in a follow-up PR we'll want to basically roll back the change that the transaction executor now always executes in release rather than debug mode. So the TransactionContext API would configure the TransactionExecutor to again run the processor with one of the debug mode APIs by default, and so disable_debug_mode would still be needed to opt out of that default.
| #! Inputs: [] | ||
| #! Outputs: [nonce] | ||
| #! | ||
| #! Where: | ||
| #! - nonce is the account nonce. | ||
| pub use memory::get_account_nonce->get_nonce | ||
| # Returns the id of the active account. | ||
| # | ||
| # Inputs: [] | ||
| # Outputs: [act_acct_id_prefix, act_acct_id_suffix] | ||
| # | ||
| # Where: | ||
| # - act_acct_id_{prefix,suffix} are the prefix and suffix felts of the ID of the active account. | ||
| pub use {get_account_id as get_id} from miden::tx_kernel_core::memory |
There was a problem hiding this comment.
Should we just remove the "doc" comments here entirely now that they are no longer actual doc comments and since they are a duplication of the original procedure's docs anyway?
There was a problem hiding this comment.
Yes, I'd suggest only placing the docs on the actual definitions, similar to how we do it in the VM. In the future we could provide tooling to resolve the docs of re-exported items (the LSP can already do this, but we may want some core tooling for generating HTML docs from doc comments like these).
| pub mod account | ||
| pub mod account_update | ||
| pub mod asset | ||
| pub mod asset_vault | ||
| pub mod callbacks | ||
| pub mod constants | ||
| pub mod epilogue | ||
| pub mod faucet | ||
| pub mod fungible_asset | ||
| pub mod input_note | ||
| pub mod link_map | ||
| pub mod memory | ||
| pub mod non_fungible_asset | ||
| pub mod note | ||
| pub mod output_note | ||
| pub mod prologue | ||
| pub mod tx |
There was a problem hiding this comment.
I was looking forward to that feature, great to see this is now a thing! I would make the modules private that don't need to be pulic:
| pub mod account | |
| pub mod account_update | |
| pub mod asset | |
| pub mod asset_vault | |
| pub mod callbacks | |
| pub mod constants | |
| pub mod epilogue | |
| pub mod faucet | |
| pub mod fungible_asset | |
| pub mod input_note | |
| pub mod link_map | |
| pub mod memory | |
| pub mod non_fungible_asset | |
| pub mod note | |
| pub mod output_note | |
| pub mod prologue | |
| pub mod tx | |
| pub mod account | |
| pub mod account_update | |
| pub mod asset | |
| mod asset_vault | |
| mod callbacks | |
| mod constants | |
| pub mod epilogue | |
| pub mod faucet | |
| mod fungible_asset | |
| pub mod input_note | |
| mod link_map | |
| pub mod memory | |
| mod non_fungible_asset | |
| pub mod note | |
| pub mod output_note | |
| pub mod prologue | |
| pub mod tx |
| /// Compacts this script's [`MastForest`], removing duplicate and unreachable nodes while | ||
| /// preserving the script root. | ||
| /// | ||
| /// See [`MastForest::clear_debug_info`] for more details. | ||
| /// Note script minification used to clear debug metadata from the forest. As of | ||
| /// `miden-core` v0.24, debug metadata is no longer stored in [`MastForest`], so minification | ||
| /// compacts the forest instead. | ||
| pub fn clear_debug_info(&mut self) { |
There was a problem hiding this comment.
Given the change in semantics, I'd rename the method to compact since it not only removes debug info. I think we need to remove the MastForest::clear_debug_info reference from the docs (I think that method no longer exists), and I'd avoid the PR level commentary "as of v0.24 ...", which is only meaningful for this PR.
But this also raises why we should not just compact the mast forest at instantiation time. IOW what is the use of retaining duplicate or unreachable nodes? Going further, why does the MastForest allow for this at all? Seems like users shouldn't have to do this explicitly.
There was a problem hiding this comment.
Duplicate nodes can exist in the MastForest for debug info to attach to - when you strip debug info, those duplicate nodes still exist until the forest gets compacted.
This has changed recently, so that the MastForest no longer needs compaction at all, but that won't be available until v0.25.
| let source_manager = Arc::new(DefaultSourceManager::default()); | ||
| let root = ModuleParser::new(Some(ModuleKind::Library)) | ||
| .parse_str(Some(Path::new("miden::testing::add")), ADD_CODE, source_manager.clone()) | ||
| .expect("add code should parse"); | ||
|
|
||
| *Assembler::new(source_manager) | ||
| .assemble_library("miden-testing-add", root, None::<&str>) | ||
| .expect("add code should be valid") |
There was a problem hiding this comment.
This could probably use assemble_test_library?
| let source_manager = Arc::new(DefaultSourceManager::default()); | ||
| let root = ModuleParser::new(Some(ModuleKind::Library)) | ||
| .parse_str( | ||
| Some(Path::new("miden::testing::noop_auth")), | ||
| NOOP_AUTH_CODE, | ||
| source_manager.clone(), | ||
| ) | ||
| .expect("noop auth code should parse"); | ||
|
|
||
| *Assembler::new(source_manager) | ||
| .assemble_library("miden-testing-noop-auth", root, None::<&str>) | ||
| .expect("noop auth code should be valid") |
There was a problem hiding this comment.
This could probably use assemble_test_library?
| let source_manager = Arc::new(DefaultSourceManager::default()); | ||
| let root = ModuleParser::new(Some(ModuleKind::Library)) | ||
| .parse_str( | ||
| Some(Path::new("miden::testing::note")), | ||
| DEFAULT_NOTE_SCRIPT, | ||
| source_manager.clone(), | ||
| ) | ||
| .expect("mock note script should parse"); | ||
| let library = Assembler::new(source_manager) | ||
| .assemble_library("miden-testing-note", root, None::<&str>) | ||
| .unwrap(); |
There was a problem hiding this comment.
This could probably use assemble_test_library?
| impl From<ProtocolLib> for Library { | ||
| fn from(value: ProtocolLib) -> Self { | ||
| Arc::unwrap_or_clone(Arc::unwrap_or_clone(value.0).mast) | ||
| } | ||
| } | ||
|
|
||
| impl From<ProtocolLib> for Package { |
There was a problem hiding this comment.
There was a problem hiding this comment.
I think it becomes irrelevant with the v0.24 changes - we no longer need that conversion, we can just have ProtocolLib wrap an Arc<Package>, similar to how it is done for CoreLibrary
b008e4e to
2ee08c8
Compare
| fn decode_package_debug_info(package: &Package) -> Option<Arc<PackageDebugInfo>> { | ||
| match package.debug_info() { | ||
| Ok(debug_info) => debug_info.map(Arc::new), | ||
| Err(PackageDebugInfoError::UntrustedSections) => None, | ||
| Err(_) => None, | ||
| } | ||
| } |
There was a problem hiding this comment.
this is a verbatim copy from mast_store.rs
| fn loaded_mast_forest( | ||
| mast: Arc<MastForest>, | ||
| package_debug_info: Option<Arc<PackageDebugInfo>>, | ||
| ) -> LoadedMastForest { | ||
| match package_debug_info { | ||
| Some(package_debug_info) => { | ||
| LoadedMastForest::with_package_debug_info(mast, Ok(Some((*package_debug_info).clone()))) | ||
| }, | ||
| None => LoadedMastForest::new(mast), | ||
| } | ||
| } |
There was a problem hiding this comment.
combined with the de-dup of decode_package_debug_info, I think there's also a way to de-dup loaded_mast_forest & loaded_mast_forest_from_package
| #[derive(Debug, Clone, PartialEq, Eq)] | ||
| #[derive(Debug, Clone)] | ||
| pub struct NoteScript { | ||
| mast: Arc<MastForest>, | ||
| entrypoint: MastNodeId, | ||
| package_debug_info: Option<Arc<PackageDebugInfo>>, | ||
| } |
There was a problem hiding this comment.
Maybe one variant could encapsulate the other? For example, we could have a stripped NoteScript containing only the pieces needed for proving, and then a separate execution variant wrapping it that carries debug info:
pub struct NoteScript {
mast: Arc<MastForest>,
entrypoint: MastNodeId,
}
pub struct NoteScriptWithDebugInfo {
script: NoteScript,
package_debug_info: Arc<PackageDebugInfo>,
}Also just thinking out loud - I'm not sure yet how this is actually useful in practice (i.e. I think the Optional debug info is enough but I might be missing the concrete use case)
| fn decode_package_debug_info(package: &Package) -> Option<Arc<PackageDebugInfo>> { | ||
| match package.debug_info() { | ||
| Ok(debug_info) => debug_info.map(Arc::new), | ||
| Err(PackageDebugInfoError::UntrustedSections) => None, | ||
| Err(_) => None, | ||
| } | ||
| } | ||
|
|
||
| fn loaded_mast_forest( | ||
| mast: Arc<MastForest>, | ||
| package_debug_info: Option<Arc<PackageDebugInfo>>, | ||
| ) -> LoadedMastForest { | ||
| match package_debug_info { | ||
| Some(package_debug_info) => { | ||
| LoadedMastForest::with_package_debug_info(mast, Ok(Some((*package_debug_info).clone()))) | ||
| }, | ||
| None => LoadedMastForest::new(mast), | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
same comment as earlier regarding duplication of these helpers
| fn decode_package_debug_info(package: &Package) -> Option<Arc<PackageDebugInfo>> { | ||
| match package.debug_info() { | ||
| Ok(debug_info) => debug_info.map(Arc::new), | ||
| Err(PackageDebugInfoError::UntrustedSections) => None, | ||
| Err(_) => None, | ||
| } | ||
| } | ||
|
|
||
| fn loaded_mast_forest( | ||
| mast: Arc<MastForest>, | ||
| package_debug_info: Option<Arc<PackageDebugInfo>>, | ||
| ) -> LoadedMastForest { | ||
| match package_debug_info { | ||
| Some(package_debug_info) => { | ||
| LoadedMastForest::with_package_debug_info(mast, Ok(Some((*package_debug_info).clone()))) | ||
| }, | ||
| None => LoadedMastForest::new(mast), | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
ditto regarding code re-use
| PackageExport::Procedure(proc_info) => Some(proc_info), | ||
| _ => None, | ||
| }) | ||
| .filter(|proc_info| proc_info.path.len() == 3) |
There was a problem hiding this comment.
path lenght filtering seems a bit fragile, is there no other way to obtain the kernel exports?
| let package = Package::create_with_modules( | ||
| PackageId::from("miden-core"), | ||
| Version::new(0, 23, 4), | ||
| Version::new(0, 24, 0), |
There was a problem hiding this comment.
this was not introduced in this PR, but I notice we're duplicating the version code in Rust and in .toml files. I wonder if there's a way to re-use the one defined in the toml?
| let source = ModuleParser::new(Some(ModuleKind::Library)) | ||
| .parse_str(Some(Path::new("test::module_1")), test_module_source, source_manager.clone()) | ||
| .unwrap(); |
There was a problem hiding this comment.
this one liner became very verbose
| #[must_use] | ||
| pub fn with_tracing(mut self) -> Self { | ||
| self.exec_options = self.exec_options.with_tracing(true); | ||
| pub fn with_tracing(self) -> Self { | ||
| self | ||
| } |
|
|
||
| - [BREAKING] Updated Miden VM dependencies to v0.24, Miden crypto dependencies to v0.27, and the MSRV to 1.96 ([#3064](https://github.com/0xMiden/protocol/issues/3064)). |
There was a problem hiding this comment.
| - [BREAKING] Updated Miden VM dependencies to v0.24, Miden crypto dependencies to v0.27, and the MSRV to 1.96 ([#3064](https://github.com/0xMiden/protocol/issues/3064)). |
This updates Miden VM to v0.24, Miden Crypto to v0.27, and rand to v0.10.
The VM upgrade moved package debug info out of MastForest. This PR keeps that debug info attached through scripts, account code, and MAST forest stores by carrying
PackageDebugInfoinLoadedMastForest.This also updates MASM imports and build helpers for the new assembly package API.
Closes #3064