v0.2.0 — Sprint 0 consolidation: relevance gate, multi-seed, pool scaling, honest native-regression correction, Phase E→F

Author: Alex-888888

CHANGELOG.md

@@ -0,0 +1,20 @@
+# Changelog
+
+## v0.2.0 — Sprint 0 consolidation (2026-06)
+
+### Added
+- `docs/SPRINT0.md`: the hidden native-knowledge regression, the learned relevance gate (six iterations), multi-seed reproducibility, and pool-scaling findings.
+
+### Changed (honest corrections & new results)
+- **Native-knowledge regression documented.** The v0.1 claim "native preserved 100 % → 100 %" was a greedy-recall artifact. Formal metrics show the always-on memory taxes general competence: PPL +19.9 % (neutral) / +49.6 % (factual), TriviaQA ~43 % → ~32 % (single run, small set — indicative).
+- **Relevance gate fix.** A small learned per-token gate (~0.5M params/layer, backbone *and* memory frozen, only the gate trains) recovers it: PPL −0.5 % vs backbone, synthetic recall 100 %, TriviaQA 40.0 % → 47.3 % (vs 50.0 % backbone, n=300) — ~73 % of the loss recovered.
+- **Multi-seed reproducibility.** Recipe validated on seeds {137, 7, 23}: 100 % synthetic recall on 3/3, standard deviation 0. The v0.1 "single run, no multi-seed" caveat is lifted at micro-scale.
+- **Pool ceiling 50k → ~200k.** The offloaded optimizer *does* train the pool (the original 0 % was loss + packing, not the optimizer). Dense is VRAM-capped ~50–100k; offload reaches 200k at 100 % recall. 500k pending a ROCm HSA allocator fix.
+- **Phase E → Phase F.** Signal/ghost fusion was found unrealizable in its initial formulation; reframed as Phase F (open to Vector Symbolic Architectures or similar), off the critical path.
+
+### Notes
+- Gate implementation code is planned for a later release (v0.3).
+- Donations now available via GitHub Sponsors (Sponsor button at the top of the repository).
+
+## v0.1.0 — Initial release
+- Phase A→D reconstruction of Memory Layers integrated into a frozen Qwen2.5-7B on a single 24 GB consumer GPU; the working warm-up recipe; diagnostic of the 0 %→100 % recall fix.

README.md

@@ -2,13 +2,16 @@
 
 ![License](https://img.shields.io/badge/license-Apache--2.0-blue)
 ![Python](https://img.shields.io/badge/python-3.10%2B-blue)
+![Version](https://img.shields.io/badge/version-v0.2.0-blue)
 
 An **independent, from-scratch reconstruction** of *Memory Layers at Scale* (Berges et al., 2024, [arXiv:2412.09764](https://arxiv.org/abs/2412.09764)) integrated into **Qwen2.5-7B-Instruct**, trained on a **single consumer GPU** (AMD RX 7900 XTX, 24 GB, ROCm/WSL2).
 
 This is **not** a reproduction of the paper at scale, and **not** a SOTA claim. Its value is **practical reproducibility on constrained hardware**, with the **lessons learned** documented honestly — including what did *not* work and why.
 
 > This work is a technical brick extracted from **J.A.R.V.I.S.**, a larger private project (a self-hosted sovereign personal AI assistant). The other components of that project remain private; only this Memory Layers reconstruction is released openly.
 
+> **v0.2.0 (Sprint 0 consolidation).** Since v0.1 we measured formal metrics, which revealed a hidden cost of the always-on memory and led to a fix (a learned relevance gate), confirmed recipe reproducibility across seeds, and clarified the pool-scaling path. See [`docs/SPRINT0.md`](docs/SPRINT0.md) and [`CHANGELOG.md`](CHANGELOG.md).
+
 ## Why this exists
 
 - **From scratch, not Meta's code.** The architecture is reconstructed from the paper. It does **not** reuse Meta's reference implementation (which is CC-BY-NC); this repository is an independent reimplementation and is released under Apache 2.0.
@@ -34,26 +37,39 @@ A naive warm-up (full-sequence loss, packed sequences, offloaded sparse optimize
 
 - **Answer-only loss** — compute the loss only on the answer tokens, not the whole sequence (the answer signal is otherwise drowned).
 - **One sequence per fact** — no packing of multiple facts per window (packing lets the model take an in-window copy shortcut instead of using the memory).
-- **Dense AdamW on the value pool** — a sparse offloaded optimizer left the pool essentially at its initialisation; a dense optimizer actually trains it.
+- **Dense AdamW on the value pool** — a sparse offloaded optimizer left the pool essentially at its initialisation; a dense optimizer actually trains it. *(Sprint 0 refined this: see Pool scaling below — offload does train the pool; the real culprit was the loss + packing.)*
 - **MLP-ADD injection** at layers 6/14/22, backbone frozen — the memory output is added to the frozen MLP, which keeps native knowledge intact.
 
 The full investigation (seven diagnostic steps, refuted hypotheses, root cause) is in [`docs/DIAGNOSTIC.md`](docs/DIAGNOSTIC.md) — this is the most useful part for anyone attempting their own reconstruction.
 
-## Results (single run, this hardware)
+### Relevance gate (Sprint 0, v0.2) — removing a hidden cost
+
+Sprint 0 metrics (below) showed the always-on MLP-ADD memory **taxes general competence** even while preserving stored-fact recall. The fix: a small **learned per-token relevance gate** (~0.5M params per memory layer, an MLP on the hidden state) at each memory layer — **backbone *and* memory frozen; only the gate trains**. It opens on stored-fact contexts and closes on general text and general factual questions. Result: **PPL within −0.5 % of the backbone**, synthetic recall **100 %**, TriviaQA **40.0 % → 47.3 %** (vs 50.0 % backbone, n=300) — about **73 % of the ungated loss recovered**. To our knowledge this frozen-backbone relevance gating is not addressed by Berges et al. (which trains jointly from scratch); it is specific to retrofitting memory onto a pre-trained frozen model. Details in [`docs/SPRINT0.md`](docs/SPRINT0.md). *(Gate code planned for a later release.)*
+
+## Results
+
+**Phase A → D (single run, this hardware):**
 
 - EmbeddingBag bandwidth on RX 7900 XTX: **151 GB/s** (above the 150 GB/s go threshold).
 - VRAM, Qwen2.5-7B + memory pool: **22.44 GB** static.
 - Toy task (Phase C): **100 % top-1** retrieval, gradcheck passes at machine epsilon.
-- Integrated model (Phase D): synthetic factual recall **0 % → 100 %** (sample of 40 over 5000 trained facts), **native known facts preserved 100 % → 100 %**, backbone frozen.
+- Integrated model (Phase D): synthetic factual recall **0 % → 100 %** (sample of 40 over 5000 trained facts), backbone frozen.
+
+**Sprint 0 (v0.2) consolidation:**
+
+- **Hidden regression found (honest correction).** The v0.1 claim “native knowledge preserved 100 % → 100 %” was a *greedy-recall* artifact. Finer metrics show the always-on memory taxes general competence: perplexity **+19.9 %** (neutral prose) / **+49.6 %** (factual prose), and TriviaQA recall **~43 % → ~32 %** (single run, small validation set — indicative). The memory helps the facts it stores but injects noise on general tokens.
+- **Relevance gate fixes it** (see recipe section): PPL −0.5 % vs backbone, synthetic recall 100 %, TriviaQA recovered (~73 % of the loss, n=300).
+- **Recipe reproducibility:** the corrective recipe validated across seeds **{137, 7, 23} → 100 % synthetic recall on 3/3, standard deviation 0**.
+- **Pool scaling:** an offloaded (CPU-state) optimizer **does** train the pool — the original 0 % came from full-sequence loss + sequence packing, not the optimizer. Dense AdamW is capped ~50–100k entries on 24 GB; **offload reaches 200k with 100 % recall**. Practical ceiling today: **~200k** (up from 50k), not yet the 1M target.
 
 ## Limits (honest, not minimised)
 
-- **Pool size 50k (dense) instead of the paper's 1M target.** The offloaded optimizer intended for a large (1M) pool did **not** actually train the pool; the working recipe uses a smaller dense pool. A repair path is identified but not done. Memory capacity is therefore well below the paper's.
-- **Formal metrics still to complete**: perplexity vs the backbone alone, and a public factual benchmark (e.g. TriviaQA). Only native-knowledge preservation (100 %) is measured so far.
-- **Single run, no multi-seed cross-validation.**
-- **Signal/ghost fusion (the larger project's next step) is not started.**
+- **Pool practical ceiling ~200k**, not the paper's 1M target. Dense AdamW is VRAM-capped (~50–100k); the offloaded optimizer reaches 200k but 500k currently hits a **ROCm HSA allocation error** during training (allocator tuning pending).
+- **Native-knowledge cost is real but mitigated.** The always-on memory degrades PPL/general recall (see Results); the relevance gate brings it back to within −0.5 % PPL and ~73 % TriviaQA-loss recovery, but a small residual remains. The gate is trained on a single fact domain (sensor-style questions); generalising it to heterogeneous fact types is future work.
+- **Indicative single-run metrics.** PPL and TriviaQA figures are single-run on small validation sets; the full 5000-fact multi-seed run remains a longer job.
+- **Signal/ghost fusion** (as initially formulated in Phase E) was found unrealizable in its initial formulation and is currently marked as **Phase F**, open to reformulation through Vector Symbolic Architectures or similar approaches. Not on the critical path of this repository.
 
-Future versions are expected to address these (larger pool, formal benchmarks, multi-seed).
+Future versions are expected to address these (larger pool via allocator tuning, multi-domain gate, formal benchmarks at larger n).
 
 ## Repository layout
 
@@ -65,19 +81,19 @@ src/
   stages/                # the staged build: product-key, Memory+, Qwen injection
   data/                  # corpus generators (synthetic + public facts + fluency)
 benchmarks/              # offload-optimizer micro-benchmark
-docs/                    # METHODOLOGY.md, DIAGNOSTIC.md, REPRODUCE.md
+docs/                    # METHODOLOGY.md, DIAGNOSTIC.md, REPRODUCE.md, SPRINT0.md
 data/synthetic_sample.jsonl   # tiny deterministic sample for a quick smoke test
 ```
 
-See [`docs/REPRODUCE.md`](docs/REPRODUCE.md) for environment, install and per-stage commands.
+See [`docs/REPRODUCE.md`](docs/REPRODUCE.md) for environment, install and per-stage commands, and [`docs/SPRINT0.md`](docs/SPRINT0.md) for the consolidation results.
 
 ## Supporting the project
 
-This work is done solo on consumer hardware (RX 7900 XTX, 24 GB). The hardware constraints forced architectural compromises — notably the 50k pool instead of 1M. Three ways to help, if you find it useful:
+This work is done solo on consumer hardware (RX 7900 XTX, 24 GB). The hardware constraints forced architectural compromises — notably the pool ceiling. Three ways to help, if you find it useful:
 
-- **Direct contributions** — to move to more powerful hardware and validate the recipe at larger scale. *(Link to be added soon.)*
+- **Direct contributions via GitHub Sponsors** (see the **Sponsor** button at the top of this repository) — to move to more powerful hardware and validate the recipe at larger scale.
 - **Company sponsorship** — for organisations interested in the outcomes of this research (native memory in LLMs, industrial application, technical sovereignty). *(Contact to be added soon.)*
-- **Technical or academic partnerships** — for labs, companies or researchers who want to collaborate on the next steps (notably the signal/ghost fusion). *(Contact to be added soon.)*
+- **Technical or academic partnerships** — for labs, companies or researchers who want to collaborate on the next steps. *(Contact to be added soon.)*
 
 If this is useful to you, that's already great; if not, no worries.
 

docs/DIAGNOSTIC.md

@@ -2,6 +2,8 @@
 
 This is the most useful part of the project for anyone attempting their own reconstruction. The first integrated warm-up **memorised nothing** (0 % recall on held-out synthetic facts) even though the training loss decreased cleanly. It took seven diagnostic steps — and two reversals — to find the real cause. The point: several "obvious" explanations were **empirically wrong**.
 
+> **See also [`SPRINT0.md`](SPRINT0.md)** for the Sprint 0 follow-up: a hidden native-knowledge regression revealed by formal metrics, and its fix (a learned relevance gate, six iterations). The pool-optimizer lesson below is refined there — the offloaded optimizer *does* train the pool; the real culprit was the full-sequence loss + packing.
+
 ## The symptom
 After a multi-epoch warm-up (loss ~0.9), recall of synthetic entity→value facts in greedy generation was **0 %**, while native known facts stayed at 100 %. The loss went down, so *something* learned — but not retrievable facts.
 
@@ -32,11 +34,13 @@ At full scale the integrated pipeline still recalled ~0–5 %. A direct probe of
 1. **Full-sequence loss** dilutes the answer signal (the value tokens are a tiny fraction of the sequence).
 2. **Sparse/offloaded optimizer + sequence packing**: the offloaded optimizer barely updated the pool, and packing several facts per window let the model exploit an in-window **copy shortcut** instead of the memory. The loss falls via the frozen backbone + dense projections, not via the value pool.
 
+*(Sprint 0 update: a later micro-benchmark showed the offloaded optimizer **does** train the pool when the recipe is correct — i.e. the dominant culprit was the full-sequence loss + packing, not the optimizer itself. See [`SPRINT0.md`](SPRINT0.md) §4.)*
+
 ## The fix (validated end-to-end)
 Porting the micro-overfit recipe into the real pipeline:
 - **answer-only loss**, **one sequence per fact** (no packing), **dense AdamW on the pool**, **MLP-ADD** with frozen backbone.
 
-Result at production scale (5000 facts, mixed corpus): synthetic recall **0 % → 100 %** (sample of 40), native known facts **100 % → 100 %**.
+Result at production scale (5000 facts, mixed corpus): synthetic recall **0 % → 100 %** (sample of 40), native known facts **100 % → 100 %** *(by greedy recall; see [`SPRINT0.md`](SPRINT0.md) for the finer PPL/TriviaQA picture and the relevance-gate fix)*.
 
 ## Incident worth noting
 An auxiliary **KL anchor** (a second reference forward with the memory disabled) caused a **deterministic HIP stall** at a fixed step on this ROCm setup. It was dropped (the known-fact anchor in the corpus sufficed to preserve native knowledge). The training watchdog was changed to a **heartbeat** (detect a frozen log) because a stalled process still looks "alive" to a simple process check.
@@ -45,5 +49,5 @@ An auxiliary **KL anchor** (a second reference forward with the memory disabled)
 - On a frozen-backbone setup, **raw cosine of hidden states is misleading** (outlier dims) — center before measuring.
 - A decreasing loss does **not** mean the memory learned — **probe the pool** (did its values move?).
 - **answer-only loss** and **one fact per sequence** are decisive for entity→value memorisation; packing invites copy shortcuts.
-- **The pool optimizer matters**: a sparse/offloaded optimizer can leave the pool at init; use a dense optimizer for pools that fit in VRAM.
+- **Greedy recall can hide a regression** — measure perplexity and a public benchmark too (Sprint 0 found the always-on memory taxes general knowledge; a relevance gate fixes it).
 - For long runs on flaky GPUs, watch **progress** (log mtime), not just process liveness.