Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
005a313
feat: add moonrun async wasm host runtime
peter-jerry-ye Jun 11, 2026
f113b8c
chore: add async upstream submodule
peter-jerry-ye Jun 11, 2026
4aa437a
test: run upstream async wasm package tests
peter-jerry-ye Jun 11, 2026
567af62
refactor: split async wasm host ports
peter-jerry-ye Jun 12, 2026
ddf60d7
refactor: namespace async wasm imports
peter-jerry-ye Jun 12, 2026
e24264b
fix: stabilize async wasm runtime boundary
peter-jerry-ye Jun 12, 2026
30f7013
fix: decode wasm job paths as utf8
peter-jerry-ye Jun 12, 2026
f449505
fix: decode wasm async paths as strings
peter-jerry-ye Jun 12, 2026
dd14bde
test: guard wasm async path encoding
peter-jerry-ye Jun 12, 2026
a5eaa5a
test: remove brittle async path source assertion
peter-jerry-ye Jun 12, 2026
7ed65f4
fix: port async process pipes for wasm fs
peter-jerry-ye Jun 12, 2026
73efd61
fix: pass owned env to async wasm spawn
peter-jerry-ye Jun 12, 2026
fc8d88f
fix: mirror async spawn signal attrs
peter-jerry-ye Jun 12, 2026
24ff659
fix: align wasm async timer draining
peter-jerry-ye Jun 12, 2026
0c81239
fix: rebase async wasm fs support
peter-jerry-ye Jun 12, 2026
7a627c8
fix: encode wasm async os strings as utf16
peter-jerry-ye Jun 12, 2026
3386cfc
fix: write wasm async os strings into guest storage
peter-jerry-ye Jun 12, 2026
7fe1601
chore: update squashed async wasm cleanup
peter-jerry-ye Jun 12, 2026
b2890f6
fix: stabilize async wasm CI checks
peter-jerry-ye Jun 12, 2026
6d1c147
test: serialize wasm async package tests
peter-jerry-ye Jun 12, 2026
eb2fec3
fix: stabilize wasm async host tests
peter-jerry-ye Jun 12, 2026
f4f896c
test: relax wasm async timing assertions
peter-jerry-ye Jun 12, 2026
ee24bec
fix: move wasm async copy-out to completion drain
peter-jerry-ye Jun 15, 2026
3a002b3
fix: align wasm async workers with native thread pool
peter-jerry-ye Jun 15, 2026
69dbdf6
chore: update async wasm submodule
peter-jerry-ye Jun 15, 2026
873eb38
chore: update async timer draining
peter-jerry-ye Jun 15, 2026
c06f7e1
fix: wake wasm async event loop from host
peter-jerry-ye Jun 15, 2026
2eba9cb
test: wait for worker completion before teardown
peter-jerry-ye Jun 15, 2026
d01b2e2
fix: keep wasm process argv live across spawn
peter-jerry-ye Jun 15, 2026
b17fad4
fix: own wasm process argv during spawn
peter-jerry-ye Jun 15, 2026
bdbd6b3
refactor: defer wasm process argv free
peter-jerry-ye Jun 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,6 @@
path = crates/moonutil/resources/error_codes
url = https://github.com/moonbitlang/moonbit-docs.git
branch = markdown-build
[submodule "third_party/moonbitlang_async"]
path = third_party/moonbitlang_async
url = https://github.com/moonbitlang/async.git
60 changes: 60 additions & 0 deletions CONTEXT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# MoonBuild Async Wasm Runtime

This context names the concepts used to describe `moonrun` support for running `moonbitlang/async` on the wasm backend.

## Language

**Semantic Stub Boundary**:
The operation-level contract exposed by `moonbitlang/async` native stubs, independent of native pointer layout or runtime object representation.
_Avoid_: Raw C ABI boundary

**Mapped Parity**:
A compatibility goal where wasm host behavior is tracked against native async stub operations without requiring a literal translation of the C files.
_Avoid_: Rewrite, line-by-line port

**Host Handle**:
An integer resource identity that the wasm guest can store and pass back while the host owns the underlying resource.
_Avoid_: Raw fd, raw HANDLE, externref

**Guest Owner Struct**:
A wasm-side value that keeps MoonBit-owned data reachable while the host has a pending operation referring to its guest-memory range.
_Avoid_: Pinned guest pointer

**Guest String Path**:
An async path argument passed from wasm to `moonrun` as a borrowed MoonBit `String` pointer plus a length measured in UTF-16 code units.
The guest must not pre-encode these paths as UTF-8 `Bytes`; `moonrun` converts the UTF-16 units into `OsString`, using the host's native path representation.
_Avoid_: UTF-8 path bytes, C string path

**Source Provenance Import**:
A `moonbit_v0` import declared together with the async C-stub source file and native symbol it tracks.
_Avoid_: Untraceable host helper

**Ported Symbol Origin**:
A V8-free Rust host implementation entry that records the async C-stub file and native symbol it is ported from. Active mapped imports must have both registry provenance and implementation provenance.
_Avoid_: Source comments that tests cannot verify

**Async Sys Module**:
A V8-free, source-shaped Rust module that owns the behavior of a native async C-stub operation. It may use `AsyncHost` for shared runtime state, resource tables, guest-memory helpers, or shared ABI representation types, but the operation logic belongs in the sys module.
_Avoid_: Thin wrappers around behavior hidden in `AsyncHost`

**Current Guest Memory**:
The `WebAssembly.Memory` object exposed as `moonbit_v0.memory` by the JS glue after instance creation or imported-memory discovery.
Host calls reacquire the current backing store for each import and never retain borrowed guest slices.
_Avoid_: Cached raw wasm pointer

**Async Monotonic Time**:
The wasm host value returned for async `ms_since_epoch`. It has millisecond precision and is monotonic from an arbitrary process-local origin; callers may compare values by subtraction, but the absolute value is not meaningful.
_Avoid_: Wall-clock epoch

## Boundary Decisions

- `moonrun` keeps V8 as the first adapter, but async host state remains outside V8 types.
- `AsyncHost` owns shared runtime state, resource tables, guest-memory helpers, and shared ABI representation types. `async_sys` owns ported operation behavior.
- `moonbit_v0` imports strip the native `moonbitlang_async_` prefix and do not add an `async_` prefix.
- Native C stubs are the semantic reference. Rust code should stay structurally close to the source files, but it does not link against `moonbit.h` object layouts.
- Wasm async time uses a monotonic host clock from an unspecified origin. Native C stubs currently use platform wall-clock APIs, but async timer semantics only require elapsed millisecond differences.
- The async wasm host currently supports only Unix-family and Windows hosts. Other host families are compile-time unsupported.
- Variable-length data crosses the boundary through guest offsets and explicit lengths. Async jobs store host-owned buffers plus guest offsets, then copy into freshly reacquired guest memory during a later host call.
- Async path arguments are the exception to byte-buffer transport: they cross as Guest String Paths so Windows reaches `OsString`/wide OS calls without a guest UTF-8 encode followed by host UTF-16 re-encode.
- V8 memory growth can replace the observable memory backing store. The runtime must not lend guest pointers to OS APIs that need pinned buffers across `memory.grow`; use host-owned pinned buffers and copy to/from wasm memory instead.
- Windows APIs that require stable buffers should receive host-owned memory, not raw wasm memory. This includes overlapped IO and other APIs where the OS may retain a pointer until asynchronous completion.
41 changes: 41 additions & 0 deletions TODO.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Async Wasm Runtime Audit TODO

## Narrative

And PR #1772 adds the `moonrun` host runtime needed for `moonbitlang/async` on the wasm backend, with a declared MVP surface around filesystem, fd, c_buffer, os_error, env, time, thread_pool, and limited process support.

But upstream async wasm tests have been failing, and serializing timing-sensitive tests can hide scheduler or ABI bugs rather than proving the Rust host runtime faithfully preserves the native C-stub semantics.

Therefore prioritize ABI, ownership, and completion correctness before relying on test serialization as evidence of stability.

## Priority List

1. [ ] Fix ABI and guest-memory lifetime safety.
And wasm jobs pass borrowed guest pointers that must remain valid until host completion.
But `Job::file_time_by_path` does not retain the output `FileTime`, and several copy-out paths are not transactional.
Therefore retain every guest owner needed by pending jobs and make `run_job`, `fetch_completion`, and `pipe` validate or roll back on copy-out failure.

2. [ ] Prevent hangs and lost completions.
And the wasm event loop depends on every running worker job eventually publishing a completion.
But stale or freed job handles can currently make a worker return without queueing a completion.
Therefore make worker failure paths publish a completion or surface a deterministic error so the event loop cannot wait forever.

3. [ ] Audit supported-surface parity only.
And the PR intentionally supports an MVP subset of async host imports.
But poll, direct IO, sockets, TLS, named pipes, and some spawn-job APIs are registered as unsupported.
Therefore verify parity for the supported surface and document unsupported imports as explicit scope boundaries.

4. [ ] Check process behavior separately.
And wasm process support is partly custom glue rather than a direct C-stub port.
But Unix signaled child status, Windows argv quoting, Windows handle inheritance, and silently ignored spawn options can diverge from native behavior.
Therefore either match native semantics or fail loudly for unsupported process options.

5. [ ] Stabilize tests after correctness fixes.
And upstream async tests include timing-sensitive cases.
But `--no-parallelize` or `--max-concurrent-tests` only reduces scheduling pressure.
Therefore apply serialization as a stability measure after ABI and completion bugs are addressed.

6. [ ] Add focused regression coverage.
And upstream package success gives useful end-to-end confidence.
But it does not stress malformed guest ranges, too-small logical buffers, failed copy-outs, lost completions, or ownership mistakes.
Therefore add targeted tests for invalid guest buffers, file-time copy-out, completion fetch failure, pipe output failure, `file_time_by_path` ownership, and process edge cases.
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
///|
async test "write and read file through async fs" {
let path = "async-fs-smoke.txt"
@fs.write_file(path, b"wasm async fs", create_mode=CreateOrTruncate)
let content = @fs.read_file(path).text()
inspect(content, content="wasm async fs")
{
let file = @fs.open(path, mode=ReadWrite)
defer file.close()
file.write_at(b"IO", position=5)
let buf = FixedArray::make(2, b'\x00')
let n = file.read_at(buf, position=5)
inspect(n, content="2")
json_inspect(buf.unsafe_reinterpret_as_bytes()[:n], content="IO")
}
let content = @fs.read_file(path).text()
inspect(content, content="wasm IOync fs")
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"import": [ "moonbitlang/async", "moonbitlang/async/fs" ]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
name = "moon/async_fs_workspace"

version = "0.1.0"

import {
"moonbitlang/async@0.19.1",
}

options(
source: ".",
)
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
members = [
"./app",
"@@ASYNC_MEMBER@@",
]
preferred_target = "wasm"
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"import": [ "moonbitlang/async" ]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
///|
async test "timer sleep resumes" {
@async.sleep(1)
println("timer resumed")
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
name = "moon/async_timer_workspace"

version = "0.1.0"

import {
"moonbitlang/async@0.19.1",
}

options(
source: ".",
)
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
members = [
"./app",
"@@ASYNC_MEMBER@@",
]
preferred_target = "wasm"
Loading
Loading