Skip to content

Activation reserve floor prevents single-model providers from fitting models that should fit #653

Description

@zulufoxtrot

Summary

The 5.5 GiB activation reserve floor in UnifiedMemoryCap.resolvedActivationReserveBytes is sized for the worst-case model (gemma-4-26b at B=8: 5.05 GiB peak). Operators serving only gpt-oss-20b (B=8 peak: 2.56 GiB) cannot lower it via DARKBLOOM_ACTIVATION_RESERVE_GB because the env override is clamped to max(env, default).

Reproduction

On a 32 GB Mac with gpt-oss-20b as the only model:

darkbloom doctor
# [FAIL] model fits in RAM — gpt-oss-20b needs ~20.0 GB but only 14.5 GB is usable

The 20.0 GB breaks down as:

  • Model weights (on-disk × 1.2 overhead): ~13.5 GB
  • Activation reserve: 5.5 GiB (sized for gemma-4 B=8)
  • Minimum serveable KV: 1.0 GiB

But gpt-oss-20b's measured B=8 activation peak is only 2.56 GiB (v0.8.0 benchmark data). A 2.75 GiB reserve (peak + slack) is sufficient, which would drop the model's need from 20.0 → 17.2 GB.

Root cause

resolvedActivationReserveBytes enforces a floor:

return max(bytes, defaultActivationReserveBytes)  // line 307

The env var can only raise the reserve, never lower it. The comment says this prevents a "legacy 3" from recreating the B=8 OOM — but for operators serving only gpt-oss-20b, the floor is unnecessarily conservative and blocks models that would fit with a model-appropriate reserve.

Impact

On 32 GB machines (the most common provider hardware), gpt-oss-20b fails the doctor check and refuses to load, even when the system has enough free memory. The operator sees "this box's RAM is too small" when the real issue is a one-size-fits-all reserve sized for a different model.

Proposed fix

Allow DARKBLOOM_ACTIVATION_RESERVE_GB to set the reserve directly (remove the max floor). Operators serving a single model can size the reserve against their model's measured peak. The default remains 5.5 GiB for operators who don't set the env var.

The coordinator must be updated in lockstep when changing this on a fleet provider (servabilityActivationFloorGB in servability.go).

Additional findings

darkbloom status reports "Inference memory: 28 GB available" (static physical - 4), while darkbloom doctor reports ~14.5 GB usable (live OS free+inactive minus load reserve). These are inconsistent and confusing to operators. The status number should also reflect actual system memory.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions