Skip to content

perf: io_uring msg_ring cross-shard hop (replace mpsc+oneshot+waker) #516

Description

@ELares

Parent: #507. Structural (est. 5-12%). Depends on the io_uring datapath.

Problem

The cross-shard hop uses a tokio mpsc + oneshot + a cross-thread waker. The waker path has real atomic + scheduling cost on the ~93% of ops that hop. Even after hop coalescing (#509) the per-batch handoff still goes through the tokio runtime's cross-thread wake.

Mechanism / fix

Replace the inter-shard handoff with io_uring IORING_OP_MSG_RING: shard A submits a message to shard B's ring directly (kernel-assisted), and B's reply rides back the same way. Removes the tokio waker atomics + the mpsc/oneshot allocations from the hop entirely; the hop becomes a ring-to-ring message.

Risk / notes

  • Requires both shards on io_uring rings (Linux + io_uring only; tokio path keeps mpsc).
  • Entangled with the runtime: may pair with evaluating a thread-per-core io_uring runtime (glommio/monoio) OR a hand-rolled ring, since tokio-uring does not expose msg_ring ergonomically.
  • Do AFTER hop coalescing (perf: cross-shard hop COALESCING at pipeline depth (squash pipelined ops per destination shard) #509) -- coalescing reduces the message COUNT; msg_ring reduces the per-message COST. Measure whether the cost still dominates after coalescing before investing.

Acceptance

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:concurrencyArea: concurrencyenhancementNew feature or requestsub-issueGranular child task split out from a parent design issue

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions