The background auto-update path already has the right lifecycle: close admission, return a retryable 503 for new work, let active requests finish, then restart. The operator path does not. darkbloom restart still uses launchctl kickstart -k, and darkbloom stop uses bootout, so either command can cut off a paid request.
This matters whenever an operator needs to apply a restart-required config change on a provider that receives steady traffic. Waiting until the local in-flight count happens to read zero is not enough because a new request can arrive between the check and the restart.
PR #392 attempted this earlier, but it now conflicts with master and has unresolved review findings. A current implementation should keep the drain inside the daemon, where admission and in-flight accounting share one actor.
Acceptance criteria:
darkbloom restart, darkbloom stop, and replacement darkbloom start ask the launchd-managed daemon to drain before changing the job.- The daemon rejects new coordinator work with the existing retryable capacity response while draining. Existing coordinator and local requests can finish.
- The command targets the exact launchd process identity and cannot signal a standalone local server or a reused PID.
- A drain timeout aborts the operator action and resumes serving. It does not force-cancel paid work by default.
- Concurrent drain requests coalesce. Update, watchdog, schedule, and operator shutdown paths cannot race into two restarts.
- Status and logs show that a drain is active and how much work remains.
- Tests cover clean drain, timeout and resume, duplicate requests, stale process identity, no-daemon behavior, and command ordering.
The auto-update path can keep its existing bounded force-cancel policy. This issue is about making manual administration safe by default.
The background auto-update path already has the right lifecycle: close admission, return a retryable 503 for new work, let active requests finish, then restart. The operator path does not.
darkbloom restartstill useslaunchctl kickstart -k, anddarkbloom stopusesbootout, so either command can cut off a paid request.This matters whenever an operator needs to apply a restart-required config change on a provider that receives steady traffic. Waiting until the local in-flight count happens to read zero is not enough because a new request can arrive between the check and the restart.
PR #392 attempted this earlier, but it now conflicts with
masterand has unresolved review findings. A current implementation should keep the drain inside the daemon, where admission and in-flight accounting share one actor.Acceptance criteria:
darkbloom restart,darkbloom stop, and replacementdarkbloom startask the launchd-managed daemon to drain before changing the job.The auto-update path can keep its existing bounded force-cancel policy. This issue is about making manual administration safe by default.