Publish Changed Resources Off the Indexer Thread

[alexanderkiel]

Problem

blaze.db.node/-changed-resources-publisher registers a watch on the node state atom:

(-changed-resources-publisher [node type]
  (let [publisher (SubmissionPublisher.)]
    (add-watch
     state publisher
     (fn [publisher _state _ {:keys [t error-t]}]
       (when (< error-t t)
         (let [changed-handles (changed-handles node type t)]
           (log/trace "Publish" (count changed-handles) "changed" type "resource handles")
           (flow/submit! publisher changed-handles)))))
    publisher))

Watches run synchronously in the thread performing the swap!. That swap! happens in
advance-t!, called from commit-success!, which runs on the single indexer thread. So all
of the following happens on the indexer thread:

• changed-handles opens a batch database and scans the type history index — real I/O whose
  cost grows with the number of resources changed in the transaction.
• flow/submit! calls SubmissionPublisher.submit, which blocks the calling thread when a
  subscriber's buffer is full (default capacity 256). A slow subscriber therefore throttles
  transaction indexing for the whole node.
• An exception thrown by either escapes through swap! into poll-and-index!, which stores it
  in :e and terminates the indexer.

The indexer is a single thread and is the sole limit on how fast transactions can be indexed, so
no avoidable work should run on it. The existing db-future watch already takes care to hand off
via ac/complete-async! for exactly this reason.

Additionally, the watch fires on every state change, including (swap! state assoc :e e). With
error-t still below t this re-publishes the handles of the last successful t a second time.

Solution

Drop the watch. Register the publishers on the node instead and have commit-success! submit a
Runnable to a dedicated single-thread executor that computes the changed handles and publishes
them. A single thread keeps the publication order per publisher intact, while blocking in
submit no longer stalls indexing.