feat(agent-memory): Phase 6 — consolidate()

[jamby77]

Stacked on #253 (Phase 5 — eviction & TTL).

What

Phase 6 of @betterdb/agent-memory: consolidation — summarize many old/low-value memories into fewer durable ones. Callable, not scheduled (per spec).

consolidate(options):

• Selects candidates in scope (threadId/agentId/namespace + tags) filtered by olderThanSeconds and maxImportance.
• Calls the caller-supplied summarize(items) => Promise<string> (the library never hard-codes an LLM — mirrors the judge-injection pattern).
• Writes the result as a normal memory with source: 'summary' at summaryImportance (default 0.7), scoped/tagged to the request.
• With deleteSources (default true) removes the originals; returns { consolidated, created, deleted }.

Design / review notes

• Write-before-delete: the summary is written before sources are deleted, so a failure can never destroy memories without leaving their consolidated replacement.
• Mass-delete guard: requires at least one selection criterion (scope, tags, olderThanSeconds, or maxImportance) — a criteria-less consolidate({summarize}) would otherwise summarize+delete the entire store. Mirrors forgetByScope's guard.
• Efficiency: the candidate scan uses an explicit RETURN of only the fields parseMemoryItem needs, so it never transfers vector blobs.
• Bounded scan (CONSOLIDATE_SCAN_LIMIT); a single summarize() over a whole huge scope is impractical anyway, and the remainder consolidates on subsequent calls.

Tests (MemoryStore.consolidate.test.ts, 8)

candidate summarize+write+delete+counts · olderThanSeconds filter · maxImportance filter · summary scope+summaryImportance · deleteSources:false keeps sources · nothing-matches → zeros & no summarize/write · no-criteria guard throws · defaults (0.7 / delete).

55/55 package tests green · tsc --noEmit clean · prettier clean.

---

Note

Medium Risk
Bulk delete and summary writes touch persistent agent memory; safeguards (criteria guard, write-before-delete, summary exclusion) reduce data-loss risk but misconfigured callers could still remove large candidate sets.

Overview
Adds MemoryStore.consolidate() so callers can merge many scoped, low-value or old memories into one durable summary via an injected summarize(items) callback (no built-in LLM).

Candidates are selected with FT.SEARCH using a new buildConsolidateFilter (scope/tags plus server-side created_at / importance bounds, and -@source:{summary} so prior summaries are not re-folded). The flow writes the summary first (source: 'summary', default importance 0.7), optionally deletes sources (default on), and returns { consolidated, created, deleted }. remember is refactored through a private writeMemory path so consolidation skips enforceCapacity and cannot evict the new summary while sources still count toward the cap. A criteria guard blocks unscoped whole-store runs.

Public ConsolidateOptions / ConsolidateResult are exported; tests cover filters, defaults, deleteSources: false, empty matches, and the capacity bypass.

^{Reviewed by Cursor Bugbot for commit f834211. Bugbot is set up for automated code reviews on this repo. Configure here.}

[KIvanow]

Phase 6 reads well: write-before-delete is genuinely safe, the mass-delete guard mirrors forgetByScope, and keeping summarize injected is the right call. Two things I would like addressed before merging:

1. Push olderThanSeconds and maxImportance into the query instead of filtering after the scan. Right now the FT.SEARCH uses only buildScopeFilter, fetches up to CONSOLIDATE_SCAN_LIMIT in index order, and then isConsolidationCandidate filters in JS. Both fields are indexed NUMERIC, so they can go straight into the filter as @created_at:[-inf (cutoff] and @importance:[-inf max]. As written, a scope larger than the scan limit returns an arbitrary first window and filters that, so matching old or low-importance candidates beyond the window are never selected even though a range query would find them. It also pulls content for rows that are then discarded. Pushing the predicates down makes the cap apply to actual matches and trims the transfer.

2. The scan does not exclude prior summaries, so default runs fold summaries into summaries. Because there is no @source:{summary} exclusion, a default consolidate({ summarize, threadId }) with no maxImportance re-selects the previous summary (importance 0.7) and deletes it into a new one. A maxImportance below 0.7 happens to protect them, but only implicitly. If rolling re-consolidation is intended, it is worth documenting; if not, the candidate scan should exclude @source:{summary}.

[jamby77]

@KIvanow Addressed in 6864f6b: (1) olderThanSeconds/maxImportance are now NUMERIC range clauses in the FT.SEARCH filter, so the scan limit applies to actual matches (not an arbitrary window) and content isn't transferred for discarded rows; (2) the scan excludes -@source:{summary}, so a default run no longer re-folds prior summaries (target them explicitly to do so). Extracted buildConsolidateFilter; tests assert the pushed-down predicates + exclusion.

[KIvanow]

Thanks, both points are addressed: the NUMERIC range clauses (@created_at:[-inf ...], @importance:[-inf ...]) are faithful to the old isConsolidationCandidate semantics, candidates now come straight from the response, and -@source:{summary} stops the default re-folding. I also confirmed it never degenerates into a pure-negation query (the criteria guard always contributes a positive clause) and joinClauses never returns * here, so the mass-delete guard stays intact.

One small wording fix before I approve: the commit message and the code comment both say prior summaries can be re-consolidated by "targeting them explicitly," but MemoryStore always passes excludeSource: SUMMARY_SOURCE, so -@source:{summary} is unconditional and there is no exposed way to include summaries. The behavior is fine (unconditional exclusion fully resolves the accidental-folding concern), but the comment promises an option that does not exist. Please either drop the "target them explicitly" phrasing or actually expose the toggle.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1b71d3e. Configure here.}

[jamby77]

@KIvanow Fixed in 1b71d3e: dropped the misleading 'target them explicitly' phrasing from the code comment, the commit message, and the PR description — the exclusion is unconditional (-@source:{summary}) and there's no toggle, as you noted. Behavior unchanged.

[KIvanow]

Verified at 384e10b. The outstanding wording fix is resolved: the "target them explicitly" phrasing is gone from the code comment, commit message, and PR description, and the consolidate comment now correctly states prior summaries are always excluded via -@source:{summary} with no toggle implied. The earlier predicate push-down (NUMERIC range clauses for olderThanSeconds/maxImportance) and the summary exclusion both hold.

384e10b is a good catch on top: writing the summary through remember() let enforceCapacity evict the just-written summary while the sources still inflated the scope, then the sources were deleted, losing the consolidated content. Routing consolidate through the new capacity-free writeMemory() (shared with remember(), which keeps its best-effort enforceCapacity) fixes it, and the reasoning holds since consolidation is a net reduction so the cap stays honored without a pass. Write-before-delete ordering is preserved.

Non-blocking minor: there does not appear to be a test covering the capacity-free consolidate path (consolidate with maxItemsPerScope set should keep the summary rather than drop it). Worth adding to lock in the 384e10b behavior, but not a blocker.

Approving.