Skip to content

fix: replace worker pool with semaphore to prevent deadlock in recurs…#1361

Open
jbuhagiar88 wants to merge 1 commit into
simulot:mainfrom
jbuhagiar88:fix/worker-pool-deadlock-semaphore
Open

fix: replace worker pool with semaphore to prevent deadlock in recurs…#1361
jbuhagiar88 wants to merge 1 commit into
simulot:mainfrom
jbuhagiar88:fix/worker-pool-deadlock-semaphore

Conversation

@jbuhagiar88

Copy link
Copy Markdown

…ive dir scan

The previous fixed-size worker pool pattern caused deadlocks when recursive directory scanning submitted tasks faster than workers could consume them. The pool's task channel would block, causing all goroutines to stall.

Changes:

  • worker.go: Replace Pool (fixed goroutines + task channel) with Semaphore (unlimited goroutines, N concurrent). Semaphore allows each submitted task to spawn its own goroutine immediately, avoiding channel backpressure. Pool type retained for backward compatibility; Stop() is now a no-op.
  • run.go: Buffer the gOut channel (10000) so folder scanning goroutines don't block waiting for the upload consumer. Removes go wrapper around pool.Submit() since Submit() now spawns its own goroutine internally.

Tested against large library (100k+ files) — no deadlock observed.

…ive dir scan

The previous fixed-size worker pool pattern caused deadlocks when recursive
directory scanning submitted tasks faster than workers could consume them.
The pool's task channel would block, causing all goroutines to stall.

Changes:
- worker.go: Replace Pool (fixed goroutines + task channel) with Semaphore
  (unlimited goroutines, N concurrent). Semaphore allows each submitted task
  to spawn its own goroutine immediately, avoiding channel backpressure.
  Pool type retained for backward compatibility; Stop() is now a no-op.
- run.go: Buffer the gOut channel (10000) so folder scanning goroutines
  don't block waiting for the upload consumer. Removes go wrapper around
  pool.Submit() since Submit() now spawns its own goroutine internally.

Tested against large library (100k+ files) — no deadlock observed.
@jbuhagiar88 jbuhagiar88 requested a review from simulot as a code owner May 29, 2026 11:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant