Consensus WAL should not block upon asynchronous writes

[cason]

Appending inputs to the WAL is handled by the following code snippet, which is blocking:

malachite/code/crates/engine/src/consensus.rs

Lines 1348 to 1351 in ba38f41

	Effect::WalAppend(height, entry, r) => {
	self.wal_append(height, entry, state.phase).await?;
	Ok(r.resume_with(()))
	}

which ends up in the following call to the actor/thread operating the WAL, which is also synchronous:

        let result = ractor::call!(self.wal, WalMsg::Append, height, entry);

The write action performed by the WalMsg::Append message, however, is asynchronous by definition, as synchronous writes, which are less frequent use the WalMsg::Flush message.

The main reason to make the consensus routine to wait for the reply of the WAL routine is error handling. But this possibly can be implemented in a more efficient way.

[Raneet10]

I think there are some consensus steps where writing to WAL needn't be blocking, propose for example. We can do the appending async (perhaps add a Effect::WalAppendAsync) and communicate to interrupt the process if there's an error in WAL. WDYT ? If I'm not missing any other concerns, I can take a stab at it.

[cason]

The Flush must be synchronous, appends must be asynchronous. The flush must make sure that all pending writes are flushed by the Flush call.

[Raneet10]

Yes got that. I was speaking from the perspective of WAL append that if the operation fails, we interrupt the calling consensus step. Sorry for not being elaborate on that.

[cason]

I see. I find this a bit complex, since we would have to interrupt an action that was possibly already performed.

[cason]

A design that I tend to prefer is to schedule the WAL writing and the action call (e.g., send this message). So the "working thread" persists everything that is needed, then trigger the action (in the case, emit the PublishMessage event). This moves the I/O logic from the main execution thread.

[Raneet10]

Sorry I missed your comment earlier. I actually made the WAL appends to be queued and then drain the queue & await when there's a flush operation. Perhaps it could be reviewed if I'm allowed to open a PR ?

[cason]

@romac, do you want to assign this issue to @Raneet10 ?