feat(recall): TOON default + LRU cache + batched neighbour fetch

[pszymkowiak]

Three coordinated changes from the recall-perf audit. They share a PR because they all live in the recall hot path and it's easier to review them together than as three tiny PRs touching the same files.

Measured impact (real numbers, this machine)

Action	Metric	Result
1 — TOON default	bytes vs detail, summaries ~130 chars	−37%
1 — TOON default	bytes vs detail, summaries ~13 chars	−64%
1 — TOON default	bytes vs json	−67% to −86%
2 — LRU cache	warm get() (cache hit) vs cold (DB hit + fill)	155× faster (81 ns vs 12 560 ns)
3 — Batch fetch	get_many(50) vs 50 × get()	2.4× faster (215 µs vs 516 µs)

The TOON win scales inversely with summary length: when summaries are ~13 chars (short tag-style memories), TOON saves 64% because metadata dominates the payload. When summaries are ~130 chars (typical decisions-* content), TOON saves 37% because the summary itself dominates. Expect 35-50% in the wild on real ICM data. This is the corrected number — my initial estimate of "60-65%" was based on small synthetic memories and turned out optimistic for typical content.

The LRU cache 155× number applies to hot reads in long-running processes (icm serve, TUI). For one-shot CLI invocations the process exits before benefiting from cache hits, so the practical impact there is the cache acting as a tiny in-flight memo — neutral, not negative.

The batch 2.4× scales linearly with N — at the typical recall budget (max_neighbors = limit/3, default ~1) the absolute saving is small (~5-50 µs) but the ratio holds.

Reproduce with:

# Action 1 — token bytes (real CLI)
cargo build --release -p icm-cli
icm --no-embeddings recall "query" -l 5             # TOON (default)
icm --no-embeddings recall "query" -l 5 -f detail   # legacy view
icm --no-embeddings recall "query" -l 5 -f json     # tooling

# Actions 2 + 3 — microbenches (release)
cargo test --release -p icm-store --lib -- --ignored --nocapture \
  bench_cache_hit_vs_miss bench_get_many_vs_n_plus_one

The two #[ignore]'d benches live in crates/icm-store/src/store.rs and are explicit documentation of the gain rather than asserting ceilings (perf numbers fluctuate under load and the existing perf_* tests already enforce ceilings).

1. icm recall --format toon|detail|json (default toon)

Per-recall stdout drops from a multi-line labelled view to one row under a single header. Smoke-test on a fixture with 1 result:

=== TOON (default) ===
memories[1]{id,topic,importance,weight,summary}:
  01KQH...,preferences,high,0.975,User prefers Rust over Go

=== DETAIL ===
--- 01KQH... ---
  topic:      preferences
  importance: high
  weight:     0.975
  created:    2026-05-01 07:04
  accessed:   2026-05-01 07:04 (x1)
  summary:    User prefers Rust over Go

=== JSON ===  (17 lines for the same memory)

The legacy multi-line view stays available as --format detail. json is for tooling.

Implementation: new crates/icm-cli/src/recall_format.rs module with RecallFormat enum + 6 unit tests (TOON header shape with/without scores, comma escaping, detail labels, JSON shape, empty list).

2. LRU cache in SqliteStore

Mutex<LruCache<String, Memory>>, cap 256 (~400 KB worst-case incl. embedding blobs). Read-through on get/get_many. Invalidations:

Path	Strategy
update(memory)	invalidate by id
delete(id)	invalidate by id
update_access(id)	invalidate by id
batch_update_access(ids)	invalidate listed ids
apply_decay()	clear all (touches every non-critical row's weight)
prune(...)	clear all if any rows deleted
consolidate_topic(...)	clear all (bulk delete + re-insert)

5 new cache invalidation tests cover all the paths above plus a "raw SQL update is not seen by cache" canary that proves the cache is actually serving reads. Adds lru = "0.12" workspace dep.

3. Batch expand_with_neighbors

Previously: N round-trips via self.get() per neighbour. Now: candidate ids collected in priority order, then fetched in one SELECT … WHERE id IN (?,?,…) via a new public get_many. Preserves scoring, dedup, and the hop discount.

The 8 existing expansion tests still pass without modification. 4 new tests cover get_many directly (basic, empty input, missing ids dropped, dedup of repeated ids).

Test plan

• cargo fmt --all -- --check clean
• cargo clippy --workspace --all-targets -- -D warnings clean
• cargo test --workspace -- --test-threads=2 — 339 passed (+10 from new tests across the three actions). The existing perf_fts_search_100 and perf_store_with_embeddings_1000 perf tests are parallelism-sensitive on baseline (they fail under default cargo test even on main), so I ran with -j2 to verify.
• Manual smoke test: icm recall produces TOON by default, --format detail reproduces the legacy output verbatim, --format json parses cleanly.
• Empirical validation of all three claims (numbers above).
• Optional follow-up: switch icm list and cmd_recall_context to TOON in a separate PR (out of scope here — recall was the user-asked target).

🤖 Generated with Claude Code

[pszymkowiak]

wshm · Automated triage by AI

📊 Automated PR Analysis

✨ Type	feature
🟡 Risk	medium

Summary

Adds a compact TOON output format (default) for icm recall to reduce token usage when piped to LLMs, introduces an LRU cache (cap 256) in SqliteStore for hot-path read-through caching with invalidation on mutations, and batches neighbour fetches into a single SELECT … WHERE id IN query via a new get_many method. The cmd_recall function is also refactored to unify the hybrid-search and FTS-fallback code paths.

Review Checklist

• Tests present
• Breaking change
• Docs updated

---

Analyzed automatically by wshm · This is an automated analysis, not a human review.