chore: Secondary Index Cleanup Hardening (#563)

* harden and cleanup secondary index

Author: jiangzhe

docs/backlogs/000089-checkpoint-old-root-retention.md

@@ -0,0 +1,44 @@
+# Backlog: Checkpoint drops old active root before transaction GC can release it
+
+## Summary
+
+Table checkpoint currently drops the old active-root guard immediately after publishing a new table-file root, before the checkpoint transaction commits and before transaction GC can release it at a snapshot-safe point.
+
+## Reference
+
+docs/tasks/000121-secondary-index-cleanup-hardening.md; docs/rfcs/0014-dual-tree-secondary-index.md Phase 5; doradb-storage/src/table/persistence.rs; doradb-storage/src/file/cow_file.rs; doradb-storage/src/trx/mod.rs
+
+## Deferred From (Optional)
+
+docs/tasks/000121-secondary-index-cleanup-hardening.md; docs/rfcs/0014-dual-tree-secondary-index.md Phase 5
+
+## Deferral Context (Optional)
+
+- Defer Reason: Task 000121 is focused on secondary MemIndex cleanup. Fixing old active-root ownership affects checkpoint publication and transaction GC payloads, so it should be planned separately from the local cleanup API hardening.
+- Findings: Table::checkpoint receives OldRoot from MutableTableFile::commit and currently drops it immediately before committing the checkpoint transaction. ActiveTrx, PrecommitTrxPayload, and CommittedTrxPayload carry row/index GC data but do not carry old table roots.
+- Direction Hint: Prefer making root retention part of the transaction lifecycle: checkpoint owns the swapped root until commit, committed transaction GC releases it after the snapshot horizon proves no active transaction can still see it, and rollback/error paths release it safely.
+
+## Scope Hint
+
+Move old-root ownership into transaction commit and GC payloads, or an equivalent root-retention structure, so swapped active-root objects are released only after no active transaction can observe them.
+
+## Acceptance Hint
+
+Checkpoint no longer drops old_root directly; old active-root guards are released through transaction GC or equivalent visibility-gated cleanup; regression coverage proves a transaction that began before checkpoint can safely keep using the previous root object until it ends.
+
+## Notes (Optional)
+
+
+## Close Reason (Added When Closed)
+
+When a backlog item is moved to `docs/backlogs/closed/`, append:
+
+```md
+## Close Reason
+
+- Type: <implemented|stale|replaced|duplicate|wontfix|already-implemented|other>
+- Detail: <reason detail>
+- Closed By: <backlog close>
+- Reference: <task/issue/pr reference>
+- Closed At: <YYYY-MM-DD>
+```

docs/backlogs/000090-protect-disk-tree-root-lifetime.md

@@ -0,0 +1,44 @@
+# Backlog: Protect obsolete DiskTree roots from checkpoint reclamation while transactions can access them
+
+## Summary
+
+Define and implement retention rules so secondary DiskTree roots and pages reachable from active transaction snapshots are not reclaimed, reused, or overwritten until no active transaction can access them.
+
+## Reference
+
+docs/tasks/000121-secondary-index-cleanup-hardening.md; docs/rfcs/0014-dual-tree-secondary-index.md Phase 5; doradb-storage/src/table/gc.rs; doradb-storage/src/table/persistence.rs; doradb-storage/src/file/cow_file.rs
+
+## Deferred From (Optional)
+
+docs/tasks/000121-secondary-index-cleanup-hardening.md; docs/rfcs/0014-dual-tree-secondary-index.md Phase 5
+
+## Deferral Context (Optional)
+
+- Defer Reason: Task 000121 can make cleanup transaction-scoped, but full DiskTree root/page reclamation policy is broader and should be designed with checkpoint GC rather than folded into the local cleanup fix.
+- Findings: Cleanup can use a transaction STS to protect its snapshot, but DiskTree rewritten blocks are currently not reclaimed through a formal root-reachability policy. The weak duplicate candidate docs/backlogs/000086-secondary-index-dual-tree-access-path.md is about access-path optimization, not root lifetime.
+- Direction Hint: After old active-root ownership is fixed, design root-reachability GC around transaction visibility: old roots and pages remain readable for active snapshots, then become reclaimable only after the GC horizon passes their owning checkpoint.
+
+## Scope Hint
+
+Cover checkpoint A/B root swap, future DiskTree root-reachability GC, table-file gc_block_list integration, and calc_min_active_sts_for_gc visibility boundaries.
+
+## Acceptance Hint
+
+DiskTree root and page reclamation consults active transaction visibility; a cleanup transaction that captured a DiskTree root can finish reads after later checkpoints; regression tests cover cleanup/checkpoint overlap and root-GC safety.
+
+## Notes (Optional)
+
+
+## Close Reason (Added When Closed)
+
+When a backlog item is moved to `docs/backlogs/closed/`, append:
+
+```md
+## Close Reason
+
+- Type: <implemented|stale|replaced|duplicate|wontfix|already-implemented|other>
+- Detail: <reason detail>
+- Closed By: <backlog close>
+- Reference: <task/issue/pr reference>
+- Closed At: <YYYY-MM-DD>
+```

docs/backlogs/000091-page-level-secondary-mem-index-cleanup-scan.md

@@ -0,0 +1,45 @@
+# Backlog: Page-level secondary MemIndex cleanup scan
+
+## Summary
+
+Current secondary MemIndex cleanup materializes every MemIndex entry for an index before processing it. Future work should replace that with a page-level or bounded-batch scan interface that preserves cleanup proof rules while limiting peak memory and reducing unnecessary lookup work. The scan should also filter out live hot entries before encoded-key materialization when their row id is at or above the captured pivot, because those entries are not cleanup candidates.
+
+## Reference
+
+doradb-storage/src/table/gc.rs; doradb-storage/src/index/unique_index.rs; doradb-storage/src/index/non_unique_index.rs; doradb-storage/src/index/disk_tree.rs; docs/garbage-collect.md; docs/backlogs/000086-secondary-index-dual-tree-access-path.md
+
+## Deferred From (Optional)
+
+docs/tasks/000121-secondary-index-cleanup-hardening.md; docs/rfcs/0014-dual-tree-secondary-index.md Phase 5
+
+## Deferral Context (Optional)
+
+- Defer Reason: The direction is valid, but avoiding the memory spike and lookup work safely needs a separate design pass around BTree page cursors, DiskTree matching, and page-local revalidation/deletion. Folding it into the current hardening task would widen the scope.
+- Findings: Current cleanup calls scan_mem_entries() on each secondary index, which delegates to scan_encoded_entries() and returns a full Vec without filtering. The cleanup then performs per-entry DiskTree checks and compare-style MemIndex deletion. It already retains live entries with row_id >= captured pivot without a DiskTree probe, but only after those entries have been copied into the full scan vector and iterated. A page-level design can bound memory and skip these live hot entries before copying encoded keys. Cleanup decisions for remaining candidates still depend on async DiskTree/column-index reads, so the design must not simply hold leaf-page locks while deciding. Related backlog 000086 covers the broader dual-tree access contract; this follow-up is narrower and specifically targets cleanup scan shape, memory use, candidate filtering, and avoidable lookup cost.
+- Direction Hint: Prefer a page-batched or cursor-based MemIndex scan. First inspect row id and delete state from each leaf slot, skip live hot entries with row_id >= captured pivot, and only allocate/copy encoded keys for cleanup candidates. Deleted entries must not be skipped by the live-hot rule, and live entries below pivot remain candidates because they may be redundant with the captured DiskTree root. Evaluate whether DiskTree facts should be gathered by sorted range matching per batch, and whether final deletion should use page-local revalidation without falling back to root traversal. Retain entries when the scanned page/key no longer matches rather than weakening correctness.
+
+## Scope Hint
+
+Design and implement a bounded cleanup scan path for user-table secondary MemIndex cleanup. Keep DiskTree immutable during cleanup and preserve captured-root proof semantics. Avoid holding MemIndex leaf latches across async DiskTree or column-index reads. Include candidate filtering so live hot entries at or above the captured pivot are skipped before encoded-key materialization and before any DiskTree or compare-delete work.
+
+## Acceptance Hint
+
+Cleanup processes MemIndex entries in bounded page-sized batches or an equivalent streaming form, does not allocate one vector for all entries in an index, skips live hot entries before encoded-key allocation, exposes a separate skipped-live-hot statistic, preserves existing safety tests, and includes targeted coverage for cleanup results when entries change between scan and delete.
+
+## Notes (Optional)
+
+Future implementation should keep scanned/retained/removed focused on processed cleanup candidates and add a separate skipped_live_hot or equivalent per-index metric for live hot entries skipped by the scan filter.
+
+## Close Reason (Added When Closed)
+
+When a backlog item is moved to `docs/backlogs/closed/`, append:
+
+```md
+## Close Reason
+
+- Type: <implemented|stale|replaced|duplicate|wontfix|already-implemented|other>
+- Detail: <reason detail>
+- Closed By: <backlog close>
+- Reference: <task/issue/pr reference>
+- Closed At: <YYYY-MM-DD>
+```

docs/backlogs/next-id

@@ -1 +1 @@
-000089
+000092

docs/checkpoint-and-recovery.md

@@ -66,7 +66,7 @@ Instead:
   the table checkpoint
 - hot secondary-index state is rebuilt through normal redo replay of hot row
   operations
-- stale cold entries are shadowed at runtime by `MemTree` and deletion-buffer
+- stale cold entries are shadowed at runtime by `MemIndex` and deletion-buffer
   state until the next table checkpoint publishes updated `DiskTree` roots
 
 This removes the old `Index_Rec_CTS` problem entirely.
@@ -77,7 +77,7 @@ When a transaction commits on a table:
 
 1. its redo record is appended to the commit log
 2. heap undo state and deletion-buffer state become visible through commit CTS
-3. secondary-index changes remain in `MemTree`
+3. secondary-index changes remain in `MemIndex`
 
 Foreground commit never updates persistent `DiskTree` pages directly.
 
@@ -166,7 +166,7 @@ metadata-only root to advance `deletion_cutoff_ts` to the checkpoint cutoff.
 
 The design explicitly does **not** do the following:
 
-- scan dirty `MemTree` entries looking for committed work
+- scan dirty `MemIndex` entries looking for committed work
 - merge arbitrary committed batches into `DiskTree`
 - advance an index-only replay watermark from a batch max CTS
 
@@ -206,7 +206,7 @@ For each redo record after the coarse replay floor:
 - Heap / hot RowStore:
   - replay if the row belongs to hot RowStore and
     `CTS >= Heap_Redo_Start_TS`
-  - row replay also rebuilds hot secondary-index `MemTree` state through the
+  - row replay also rebuilds hot secondary-index `MemIndex` state through the
     normal row/index update logic
 - Cold-row deletions:
   - replay if `row_id < pivot_row_id` and
@@ -232,17 +232,17 @@ After redo reaches log end:
 
 - RowStore is reconstructed for hot rows
 - deletion buffer is reconstructed for post-checkpoint cold deletes
-- `MemTree` is reconstructed for post-checkpoint hot secondary-index state
+- `MemIndex` is reconstructed for post-checkpoint hot secondary-index state
 - `DiskTree` remains the checkpointed source of truth for cold secondary-index
   state
 
 The engine can then serve traffic.
 
 ## 6. Garbage Collection
 
-### 6.1 MemTree Cleanup
+### 6.1 MemIndex Cleanup
 
-`MemTree` entries may be cleaned or evicted only after the corresponding table
+`MemIndex` entries may be cleaned or evicted only after the corresponding table
 checkpoint makes their state durable:
 
 - hot rows checkpointed into persistent LWC plus companion `DiskTree` entries

docs/garbage-collect.md

@@ -0,0 +1,114 @@
+# Garbage Collection
+
+Garbage collection in the storage engine is split by ownership boundary. Each
+collector needs a proof that its target is unreachable for every active
+snapshot before it can physically remove state.
+
+## Transaction Purge
+
+Transaction purge is governed by `Global_Min_STS`, the oldest active snapshot
+timestamp used by the purge workers. A committed undo or index cleanup record is
+purgeable only when its commit timestamp is strictly older than
+`Global_Min_STS`.
+
+Purge removes runtime-only obligations:
+
+- row undo links whose next version is no longer visible to active snapshots
+- index undo branches after their owning row undo can no longer be rolled back
+- delete overlays created by transaction cleanup when the row/deletion proof is
+available
+
+## Row-Page Undo GC
+
+Row-page undo-chain GC is local to the row page and keeps historical versions
+needed by active readers. Runtime unique-key links are part of that same
+lifecycle: they preserve old unique-key ownership across row chains and are not
+owned by secondary-index full-scan cleanup.
+
+Runtime unique-key links are not collectible merely because a row crossed the
+column-store pivot, disappeared from the deletion buffer, or became cold. They
+are collectible only when rollback/index-undo obligations are gone and
+`Global_Min_STS` proves no active snapshot can require the old owner.
+
+## MemIndex Full-Scan Cleanup
+
+User-table secondary indexes use a hot `MemIndex` and a checkpointed cold
+`DiskTree`. Full-scan cleanup is a memory cleanup pass only. It never mutates
+`DiskTree` and never rebuilds checkpointed cold entries into `MemIndex`.
+
+The cleanup pass captures:
+
+- table checkpoint timestamp
+- `pivot_row_id`
+- `ColumnBlockIndex` root
+- secondary `DiskTree` roots
+- deletion checkpoint cutoff
+- caller-supplied `Global_Min_STS`
+
+Live entries can be removed when the captured row id is below the captured
+pivot and the captured `DiskTree` already has the same durable entry:
+
+- unique: same encoded logical key maps to the same row id
+- non-unique: same encoded exact `(logical_key, row_id)` key exists
+
+Delete overlays require overlay-obsolescence proof, not `DiskTree` absence. A
+unique delete-shadow or non-unique delete-marked exact entry can be removed
+when one of these facts is true:
+
+- a deletion-buffer marker is committed and older than `Global_Min_STS`
+- the captured table root is older than `Global_Min_STS`, the row id is below
+  the captured pivot, and the captured `ColumnBlockIndex` proves the row id is
+  absent
+- the captured table root is older than `Global_Min_STS`, the row id is below
+  the captured pivot, and the captured cold LWC row still exists but its current
+  indexed values encode to a different scanned MemIndex key
+
+The last case covers committed key changes for cold rows: the row still exists,
+but the scanned delete overlay no longer protects the row's current secondary
+key. If the captured cold row still owns the same encoded key, cleanup retains
+the overlay unless whole-row deletion is proven. Hot row-page key-obsolescence
+proof remains transaction index GC's job because it needs row-page undo-chain
+visibility.
+
+A matching stale cold entry may still exist in the captured `DiskTree` after a
+row-deletion overlay is removed. That is safe because normal row/deletion
+visibility checks filter the row; `DiskTree` mutation remains checkpoint-owned.
+
+Invalid cleanup proofs:
+
+- deletion-buffer absence
+- `row_id < pivot_row_id` by itself
+- a `RowLocation::NotFound` result from a moving current root
+- the existence of a newer `DiskTree` root not captured with the table snapshot
+- hot row-page key mismatch without transaction index GC's row-page proof
+
+Cleanup removes scanned entries with encoded compare-delete operations that
+also check the expected row id or delete-bit state. If an entry changed after
+the scan, cleanup retains it.
+
+## DiskTree And CoW Roots
+
+`DiskTree` roots are published as companion state of table checkpoint and
+deletion checkpoint. Old `DiskTree` pages become reclaimable only after the
+table-file CoW root that references them is no longer reachable by active
+readers. Root reachability GC is separate from `MemIndex` cleanup.
+
+## Deletion Buffer
+
+The deletion buffer tracks tombstones for persisted column-store rows. A marker
+is globally purgeable only after its delete timestamp is committed and older
+than `Global_Min_STS`.
+
+Deletion-buffer absence is deliberately not a general proof. Some hot-origin
+secondary-index overlays never had a cold delete marker, and a missing marker
+does not prove that every active snapshot can ignore the old row/key owner.
+
+## Summary
+
+Use the narrowest proof owned by the component being collected:
+
+- transaction purge uses `Global_Min_STS` and undo/index-undo ownership
+- row-page GC uses row undo-chain visibility
+- runtime unique-key links use the undo GC horizon
+- `MemIndex` cleanup uses captured checkpoint roots plus deletion proof
+- `DiskTree` page GC uses table-file CoW root reachability

docs/index-design.md

@@ -32,7 +32,7 @@ This pattern appears in both indexing layers:
   - hot routing in memory
   - cold routing in persisted CoW state
 - secondary index:
-  - hot `MemTree`
+  - hot `MemIndex`
   - cold `DiskTree`
 
 This design keeps foreground writes memory-first while letting checkpoint
@@ -51,11 +51,17 @@ Use the following documents as the living source of truth:
 
 2. [`secondary-index.md`](./secondary-index.md)
    - unique and non-unique secondary-index models
-   - `MemTree` and `DiskTree`
+   - `MemIndex` and `DiskTree`
    - read/write behavior
    - companion checkpoint maintenance
    - recovery behavior
 
+3. [`garbage-collect.md`](./garbage-collect.md)
+   - transaction and row undo purge
+   - runtime unique-key link lifecycle
+   - `MemIndex` cleanup proofs
+   - `DiskTree` and table-file CoW root reclamation
+
 ## 4. Summary
 
 `RowID` is the common identity across the storage engine. Block index resolves

docs/rfcs/0014-dual-tree-secondary-index.md

@@ -525,10 +525,10 @@ SAFETY:` comments, and run the repository lint gate. [D10], [C4], [C7]
     intentionally deferred optimization work as backlogs.
   - Non-goals: generic B+Tree backend unification and broad query-planner range
     scan features.
-  - Task Doc: `docs/tasks/TBD.md`
-  - Task Issue: `#0`
-  - Phase Status: `pending`
-  - Implementation Summary: `pending`
+  - Task Doc: `docs/tasks/000121-secondary-index-cleanup-hardening.md`
+  - Task Issue: `#562`
+  - Phase Status: done
+  - Implementation Summary: Implemented Phase 5 secondary-index cleanup hardening: removed the transitional MemIndex wrapper, added proof-based MemIndex cleanup, preserved no-cold-backfill recovery, refreshed docs, and validated with clippy and nextest. [Task Resolve Sync: docs/tasks/000121-secondary-index-cleanup-hardening.md @ 2026-04-17]
 
 ## Test Strategy