Skip to content

CHANGELOG.md

Latest commit

 

History

History
231 lines (143 loc) · 21.6 KB

CHANGELOG.md

File metadata and controls

231 lines (143 loc) · 21.6 KB
Raw

Changelog

v0.8.10 (2026-08-21)

Provider (Swift)

Fixes

  • Seed retained optimization latches before packaged-child exec — The v0.8.9 curl installer could download and verify the signed bundle but reject it with safe R1 was not latched as requested on hosts where MLX initialized Metal before runtime-smoke.run(). Installer, self-updater, and paged preflight children now receive the exact retained three-key environment (DARKBLOOM_GEMMA4_PREFILL_CHUNK_EVAL=18, MLX_GEMMA4_FUSED_WEIGHTED_UNSORT=1, MLX_GATHER_QMM_EXPERT_SLICES=1) at process launch, while the child still poisons/reapplies/verifies the values and AOT kernels. Existing installations remain untouched on any failed verification.

v0.8.9 (2026-08-21)

Provider (Swift)

Fixes

  • Emergency Qwen3.6 runtime rollback — Restores the v0.8.7 mlx-swift-lm pin (ab73a827) and removes v0.8.8's default-on GDN four-input projection fusion and direct weighted-expert reduction. In the fixed one-hour production comparison, Qwen success fell 85.52%→65.79%, p50 decode fell 38→26 tok/s, client timeouts approximately doubled, and the hard TTFT gate emitted 612 429s (602 marked counterfactually serveable). M1/M2 providers regressed even though they cannot use affine qmv_wide, isolating the Qwen runtime changes as the first rollback target. Gemma's merged qmv_wide MLX/MLX-Swift pins remain enabled for continued benefit and separate attribution.
  • Restore the retained runtime-smoke contract — Removes the retired Qwen process-global reduction key from serving projection, launchd passthrough, signed-child validation, and benchmark-report expectations. Provider artifact verification returns to the three retained Gemma controls.

v0.8.8 (2026-08-21)

Provider (Swift)

Performance

  • Default-on small-batch quantized matvec (qmv_wide) — Ports upstream MLX #3764 (548dd80e) into the Darkbloom MLX fork and regenerates MLX-Swift's embedded JIT Metal sources. On generation-15+ Apple GPUs, affine BF16 W4/W8 dense projections with 2 <= M < vector_limit reuse each decoded weight group across the small activation-row tile; M=1 remains on QMV, matrix-sized inputs remain on QMM, and gathered expert projections are unchanged. Source-built metallib and release-artifact checks require representative W4/W8 ordinary and batched symbols. Local M4 Max directional medians preserved B=1 and improved Gemma B=4 aggregate decode 195.93→216.94 tok/s at 512 context (+10.72%) and 143.18→155.72 tok/s at 8K (+8.76%); B=2 was +4.49%/−1.00%. The attempted 32K comparison is intentionally unclaimed because both benchmark arms entered a persistent degraded host/device state.

Default posture

  • qmv_wide is an automatic Metal dispatcher route, not a beta flag: eligible generation-15+ affine 2 <= M < vector_limit projections take it by default. Gemma layer-18 submission, coupled weighted-unsort/safe-R1, expert-tile trust, solo-prefill stripe, prompt narrowing, and packed-prefill defaults remain enabled for existing and new provider configurations.

v0.8.7 (2026-08-20)

Provider (Swift)

Fixes

  • Restore Qwen3.5/3.6 system-history normalization — The compatibility fix released on the v0.8.5 branch was absent from master and therefore from v0.8.6, causing Qwen's published template to reject OpenAI-compatible histories with a late system turn (System message must be at the beginning). Production Qwen 422s rose from 3.46–4.95% on v0.8.5 to 27.73–33.95% on v0.8.6. Text-only system turns are again folded into one leading system message before generic tool-history validation; structured/media system content remains fail-closed.

v0.8.6 (2026-08-20)

Provider (Swift)

Performance

  • CBv2 prefill stack, default-on — Cold prefill 6,406.8 → 4,636.9 ms at 8K on the M4 Max prod artifact (~1,766 tok/s, +38% vs v0.8.5 defaults); 4×8K burst aggregate 1,312 → ~1,500 tok/s (+13–17%) with token-checksum parity across every arrival pattern. Four independently escapable levers (#646, mlx-swift-lm#111):
    • Expert-tile trust serving default — skips the per-chunk descriptor retract drain (80 stream drains/chunk); exact MLX_GATHER_QMM_EXPERT_SLICES=1 restores the drain posture. (#638)
    • Solo-prefill stripe (2048) — when exactly one live text request holds the scheduler, its chunk widens 512→2048 (weights streamed 4× less often, full 32-row expert tiles). Armed per-plan; any company disarms to plain 512s; KV-capacity failure shrinks once, never preempts; the stripe budget belongs exclusively to the armed row. DARKBLOOM_CBV2_SOLO_PREFILL_STRIPE=0 disarms. Known trade: ~12% TTFT regression under Low Power Mode — throttled/battery providers should export the escape.
    • Recurrent prompt narrowing (Qwen LM head) — intermediate chunks return a one-element handle instead of the [1,512,248320] logits tensor (242.5 MiB/chunk); the frontier chunk norms + projects exactly one row. DARKBLOOM_CBV2_PREFILL_NARROWING=0 restores byte-old behavior.
    • Packed prefill (Qwen3.6) — equal-length prompt chunks from concurrent requests run as one [B,L] forward with per-row recurrent state (one weight stream per cohort; text-only v1).
  • Mean-TTFT prefill serialization (opt-in) — DARKBLOOM_CBV2_MAX_PARTIAL_PREFILLS=1 caps rows receiving prompt work per step (FCFS): burst TTFTs become a staircase instead of everyone waiting for the makespan. Paused rows hold no slot (a stalled consumer cannot head-of-line block admission). (#646)
  • Adaptive persistent-history MTP promoted onto master (still behind the mtp beta flag) — the v0.8.5-described capture-verify stack's adaptive width selection and persistent head KV now ship in the release pin. (#641, mlx-swift-lm#110)

Benchmarks / Tooling

  • Scheduler-prefill report schema 3 (records the effective stripe posture); Gemma contbatch wrapper schema 6 — baseline pins refuse pre-default-flip reports so the posture change can never masquerade as a code delta. 14 review-hardening scheduler fixes with regression tests; measurement methodology + posture discipline in docs/reports/2026-08-19-solo-prefill-stripe-experiment.md. (#646)

v0.8.5 (2026-08-14)

Provider (Swift)

Performance

  • Qwen3.6 E=256 expert-tile prefill route + fused gate_up - Instantiates the Gemma4 descriptor/tile kernel family for Qwen's 256-expert shapes (mlx d3c82db), fuses the routed gate/up projection into one gather (SwitchGLU(fuseGateUp: true), per-layer and per-load with heterogeneous-quantization split fallback across every checkpoint key space), and adds the opt-in trust refinement that skips the per-chunk retract drain. Measured on M4 Max, prod artifact: routed MoE block -26.3% at T=512; end-to-end prefill 1243→1364 tok/s default, 1433 with trust (+15.2%) at 8k; 2k +7.4%, 32k +6.6%. (#617, mlx-swift-lm#107)
  • Qwen3.6 MTP: adaptive persistent-history capture-verify stack (behind the mtp beta flag, default off) - Selects one rectangular k=0...4 per scheduler plan/decode-row bucket from request-local acceptance probabilities and shared marginal-cost evidence, obtains policy confidence with a lazy hierarchical Metal top-2 reduction, and keeps complete committed context in request-owned MTP-head KV; leading trusted history now appends K/V only, so each round computes full head output only for its final row. Target verification remains one [B,1+k] forward with target-prefix-authoritative acceptance at any temperature. Widths 1/2 use captured recurrent state; S>=3 runs one full-window recurrence, commits the final state directly on full acceptance, and retains compact transformed inputs for lazy strict-prefix replay instead of full per-position recurrent stacks. Request-owned head/history and recurrent replay residency are charged exactly by admission. A combined production-bundle DEBUG canary validation on this M4 Max measured median target 7.657386s vs MTP 3.813989s (2.0077x); this is validation evidence, not release throughput. (#616, mlx-swift-lm#106/#108/#110)
  • mlx gpu::eval use-after-free fix - A stale MTL::CommandBuffer captured across eval_gpu could crash any primitive that syncs mid-eval (deterministic SIGSEGV on the E=256 route; previously survived on allocator luck). (mlx#5/#7)

Fixes

  • Inline-MTP inspection resolves HF-cache symlinks and rejects loudly - Symlinked snapshots (the standard HF blobs/ layout) silently disabled inline MTP: inspectInlineArtifact required regular files and reported nothing. Inspection now resolves links and validates targets; every genuine rejection logs a concrete reason and path. Untrusted operator-path inspection stays symlink-rejecting. (#618)

Observability

  • MTP posture and acceptance on the local /metrics endpoint - mtp_enabled, mtp_active, mtp_rounds_total, mtp_tokens_proposed_total, mtp_tokens_accepted_total, and mtp_inactive_reason{model,reason} (including inline_artifact_invalid) in both --local and unified serving modes - acceptance was previously observable only in Datadog Logs. (#619)

Coordinator

Fixes

  • Expose exact Hugging Face repositories in model feeds - Registry metadata can now override hugging_face_id independently of the internal routing ID. Both /v1/models and /v1/models/openrouter honor the override for concrete and aliased models, with an authenticated hugging-face-id admin action for existing registry rows. (#620)

v0.8.4 (2026-08-13)

Provider (Swift)

Fixes

  • Stream Qwen3.6 reasoning deltas immediately (TTFT fix) - Qwen3.6-style chat templates pre-open the <think> block at the prompt tail, so model output carries only the closing tag and the streaming think parser buffered the entire block before emitting anything: measured prod TTFT was 755ms + 12.51ms x reasoning_tokens (r = 0.9878) while the first byte arrived in ~76ms. The engine now probes the rendered prompt tail (ReasoningPromptProbe) and injects one synthetic <think> open ahead of model output — gated on an active think-format parser and streaming — so reasoning_content streams per chunk and TTFT reflects real first-token latency. Text and VLM paths; the marker never reaches the prompt, the consumer, or the TB-007 hash domain. (#614)

v0.8.3 (2026-08-12)

Provider (Swift)

Features

  • Qwen3.6-35B-A3B VLM with inline MTP - Adds production-path text, image, and tool inference for the combined Qwen artifact. The runtime preserves request-owned recurrent and three-axis mRoPE state, causal vision attention, exact rollback, and source-matched target/assistant memory accounting. MTP remains depth-one, serial, and exact-target-verified; video, prefix reuse, paged KV, compiled decode, packed prefill, and rectangular MTP remain fail-closed.

Release Safety

  • The model is registered as beta/ready without an alias or active-version promotion. Provider rollout and model promotion remain separate reviewed operations after the signed v0.8.3 bundle passes a controlled fleet canary.

v0.8.2 (2026-08-10)

Provider (Swift)

Performance

  • Gemma 4 26B-A4B v0.8.2 optimization stack — Layer-18 lazy prefill submission; coupled weighted-expert-unsort + safe-R1 expert-QMM gate (both default-on via [gemma_optimizations]); the VLM wrapper's directly shared text tower; packed multimodal prefill inside q=128 query blocks; source-matched metallib enforced across CI/release/packaged smoke. Final performance and retention deltas are pending a same-tree A/B measurement on the reviewed release tree. Earlier gitignored measurements predated the final kernel edits and are not release evidence. Dropped before the final cut: expert gate/up packing, dense gate/up packing, standalone weighted-unsort, standalone R1. 0cc5fc9c9

Security

  • Keep inline video plaintext off disk — The provider decodes coordinator-inlined MP4/QuickTime bytes through a bounded, memory-backed AVFoundation asset, retains the byte owner through metadata probing and frame sampling, and rejects external asset references. Exact-name legacy vlm-<UUID>.mp4 files are purged once after single-instance lock acquisition on both coordinator-connected and standalone launch paths.
  • Close unintended provider-derived plaintext egress paths — Provider inference failures cross the WebSocket and client boundary only as closed-vocabulary codes/reasons, while browser/provider free-form telemetry and automatic provider log reporting are retired. The explicit darkbloom report support command remains operator-initiated, preserves macOS unified-log privacy redaction, supports local --dry-run review, and uses authenticated upload plus admin-only retrieval.

Unreleased (Apr 26 - May 25, 2026)

26 commits since aa74499.

Coordinator

Features

  • DB-backed model registry (#203) -- Model catalog is now stored in Postgres with R2-hosted manifests. Includes readable prefixes, runtime limits, runtime parameters, hardened validation, and provider inventory preservation across catalog updates. 50e8887b
  • Token-budget routing with engine-level admission (#171) -- Replaces heuristic-based routing with engine-reported capacity signals. Providers report real activeTokens, maxTokensPotential, and token budget usage. Coordinator uses EWMA observed TPS, fleet median fallback, and token-budget admission. 5 new fields on BackendSlotCapacity (backward-compatible). 25+ new tests. 78314b4e
  • Speculative TTFT dispatch (#171) -- Parallel dispatch to a backup provider at 50% of the TTFT deadline. First provider to deliver a token wins; loser is cancelled. No double-billing. OpenRouter TTFT SLA enforcement (5s base + 1ms/input token). 78314b4e
  • Early 429 with Retry-After for capacity signaling (#171) -- Returns 429 instead of 503 when fleet is at capacity (no uptime penalty on OpenRouter). GET /v1/models/capacity endpoint for observability. ModelCapacitySnapshot with per-model routable/warm/cold providers, aggregate TPS, estimated TTFT, and token budget headroom. 78314b4e
  • Coordinator-driven model preload protocol (#110) -- New load_model / load_model_status WebSocket messages allow the coordinator to push model warm-up requests to providers ahead of demand. 56b050b4
  • Datadog observability stack (#143) -- DogStatsD, APM, journald log collection on dev GCE VM. Structured metrics: attestation counters, model_type tags, provider-count gauges, completion-tokens counter, fleet version/binary hash observability, billing histograms (reservation, settlement, provider credits, platform fees), store latency, input token metrics. 56b050b4
  • X-Timing latency decomposition header (#136) -- Single JSON header with per-phase microsecond breakdown: parse_us, reserve_us, route_us, queue_us, encrypt_us, dispatch_us, provider_us. 56b050b4

Bug Fixes

  • Structured JSON 404 for unimplemented /v1/ endpoints* (#168) -- Catch-all handler returns application/json errors instead of Go's default text/plain 404. Prevents OpenAI SDK parse failures on /v1/embeddings, /v1/moderations, etc. Added openai-go SDK compatibility tests. e108da5f
  • OpenAI error response code and param fields (#144) -- errorResponse now populates code and param per the OpenAI API spec. insufficient_quota canonical code, param="model" on model errors. All 202 existing call sites backward-compatible. e108da5f
  • Require country for Stripe payout onboarding (#179) -- 2e262b73
  • Stripe dashboard metadata -- 35582c82
  • Prevent double-decrement on untrusted provider disconnect (#143) -- MarkUntrusted race fix: hold write lock through counter decrement. Heartbeat no longer revives untrusted providers. 56b050b4
  • Skip Python/dangerous-modules check for Swift runtime (#143) -- Private text routing gate correctly bypasses Python-specific checks for Swift providers. 56b050b4
  • Fix planner pending leak (#171) -- Changed planner.complete() to planner.cancel() in request completion path. Without this, pending entries accumulated until maxQueuedRequests (128), permanently bricking the provider. 78314b4e
  • Refund provider-specific extra on generic dispatch (#171) -- All 14 failure paths after reserveAdditionalForProvider now refund the delta in handleGenericInference. 78314b4e
  • activeRequests counted per-model, not per-provider (#171) -- ModelCapacitySnapshot now counts only pending requests matching the specific model. 78314b4e
  • Link test providers to user account (#174) -- Ensures payout destination check passes for test providers. f4219c4f

Breaking / Protocol Changes

  • Go module path changed -- github.com/eigeninference/coordinator/internal/X -> github.com/eigeninference/d-inference/coordinator/X. Module path is now github.com/eigeninference/d-inference. coordinator/internal/ flattened to coordinator/. 56b050b4
  • Bundle filename changed -- Coordinator now accepts darkbloom-bundle-<platform>.tar.gz (was eigeninference-bundle-). 56b050b4

Provider (Swift)

Features

  • Swift provider runtime shipped (#110) -- Full darkbloom CLI with serve, start, stop, status, doctor, models, benchmark, login, logout, enroll, update, verify subcommands. Production inference via MLX-Swift on Apple Silicon. GPU-only enforcement. Rename from eigeninference to darkbloom with backward compatibility. 56b050b4
  • Continuous batching (#110) -- All concurrent requests merged into one batched forward pass per step via BatchGenerator. Bit-identical against single-stream greedy. Near-linear throughput scaling (B=4/B=1 = 3.8x on Qwen, 2.9x on Gemma MoE). 56b050b4
  • Multi-model concurrent serving (#167) -- 953b8f02
  • MLXLMServer adoption for OpenAI protocol (#208) -- ca8983c4
  • BatchedEngine migration (#207) -- BatchScheduler migrated from BatchGenerator to BatchedEngine. 80fc0ee7
  • Idle-timeout model unload (#110) -- Provider unloads model after 60 minutes idle (configurable). Next request lazy-reloads. 56b050b4
  • Persistent Secure Enclave key (#146) -- Replaces ephemeral CryptoKit SE keys with persistent Security framework keys in the macOS data protection keychain. Bound to signing team's keychain access group. .app bundle with embedded provisioning profile. 56b050b4
  • Token budget engine-level admission (#171) -- BatchScheduler reports real token budget usage. EWMA decode TPS tracker. Engine-level admission gate rejects with token_budget_exhausted. Dynamic token budget sized from model weight bytes and available memory. 78314b4e
  • Architecture-aware kvBytesPerToken (#171) -- Computed from config.json metadata (layer count, KV heads, head dim) instead of weight-bytes heuristic. Handles hybrid attention (Gemma 4), GQA/MQA, recurrent layers (Qwen3.5), and VLM wrappers. 4x reduction on Qwen3.5 models. 78314b4e
  • Rust-to-Swift bridge auto-update (#110) -- Rust provider auto-updates to Swift bundles, rewrites launchd plist, handles .app bundle layout. 56b050b4

Performance

  • Greedy fast-path optimization: nil sampler for temperature=0 uses vectorized fallback (+6-13% decode TPS). 56b050b4
  • mlx-swift-lm double buffering, UInt32 token tensors. 56b050b4
  • Release-mode BatchGenerator B=4 matches mlx_lm Python reference (Qwen: ~1130 vs 1119 tok/s; Gemma: ~186 vs 181 tok/s). 56b050b4

Console UI

  • Refresh earn calculator and landing page (#185) -- ed6d655e
  • Fix Next.js version vulnerability (#172) -- 2f65bb41
  • Analytics tracking fix -- f7dab6fa

Testbed / E2E

  • Integration test suite (#136) -- 12 E2E tests with real Swift provider (Postgres + coordinator + provider per test). Tests: NonStreaming, Streaming, Concurrent, Encryption, Billing, Payout, Referral, InsufficientBalance, InvalidModel, AttestationHeaders. 56b050b4
  • Load generator and profiling (#136) -- Configurable concurrency, streaming, benchmark CI with PR comment posting. Heavy-load 100-concurrent 10KB benchmark. Latency regression assertions. 56b050b4
  • Performance test suite (#110) -- Warm/cold TTFT, encrypted E2E, batched throughput, decode-TPS bracket tests for Qwen 0.6B and Gemma 26B MoE. 56b050b4

Security

  • Harden release registration and binary hash policy (#99) -- Release download URL derived from allowlist. b5dd0488
  • Harden release workflow protections (#103) -- e515244f
  • Rust-to-Swift cutover hardening (#110) -- Post-codesign verification of entitlements, provisioning profile validation (team ID, access group, expiration), MLX wheel pinning, prod hard-fail on Swift tests. 56b050b4
  • STRIDE threat model (#110) -- 40 threats across 9 trust boundaries. Automated PR review workflow via Claude API. 56b050b4
  • Typed response structs for OpenAI endpoints (#166) -- 7fbfa9fc

Billing

  • Remove deprecated Solana/wallet-based provider payouts (#178) -- fe994fc9

CI / Infrastructure

  • Migrate CI workflows to Blacksmith (#182) -- ff8527a4
  • CI runs on any PR (#119) -- Not just master/main. 98a3a024
  • Remove racing deploy-dev-coordinator workflow (#137) -- Eliminates race condition with Cloud Build. cf4c0efa
  • DEV_/PROD_ prefixed repo secrets -- Environment-scoped R2 + coordinator secrets for release isolation. 56b050b4
  • Native Postgres fallback for CI -- Docker/colima replaced with initdb + postgres on macOS runners. 56b050b4
  • Correct version comments for SHA-pinned actions (#160) -- 85cedc7e

Housekeeping

  • Remove unused dependencies (#112) -- 7ccc592f
  • Remove stale Python integration test (#109) -- e6d63a86
  • Bump mlx-swift and mlx-swift-lm submodules (#206) -- Re-homed to Layr-Labs forks. 5919dac1
  • Darkbloom license agreement (#173) -- dde67b28
  • Update README (#176) -- 7451a473