Skip to content

Commit 34bf56c

Browse files
authored
chore: DiskTree Checkpoint Sidecar Publication (#555)
* implement disk tree checkpoint
1 parent cd15b06 commit 34bf56c

File tree

10 files changed

+1730
-196
lines changed

10 files changed

+1730
-196
lines changed
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Backlog: Parallel secondary DiskTree checkpoint application
2+
3+
## Summary
4+
5+
Design and implement parallel application of secondary DiskTree checkpoint sidecar work.
6+
7+
## Reference
8+
9+
docs/tasks/000118-disk-tree-checkpoint-sidecar-publication.md; docs/rfcs/0014-dual-tree-secondary-index.md
10+
11+
## Deferred From (Optional)
12+
13+
docs/tasks/000118-disk-tree-checkpoint-sidecar-publication.md; docs/rfcs/0014-dual-tree-secondary-index.md Phase 2
14+
15+
## Deferral Context (Optional)
16+
17+
- Defer Reason: Parallel application needs mutable-file allocation, write staging, and deterministic root installation design beyond this local sidecar optimization pass.
18+
- Findings: Current sidecar application is sequential across secondary indexes because each writer borrows one MutableTableFile. DiskTree subtree rewrite is also sequential inside each root. Parallelism is plausible both across independent secondary indexes and across disjoint DiskTree subtrees.
19+
- Direction Hint: Design staged per-index and per-subtree rewrite outputs, then serialize root installation into MutableTableFile. Account for DiskTree's no-per-rewrite-GC policy and old-root readability.
20+
21+
## Scope Hint
22+
23+
Support parallelism across independent secondary indexes and, where feasible, across disjoint DiskTree subtree rewrites while preserving copy-on-write root publication semantics.
24+
25+
## Acceptance Hint
26+
27+
A future task proves parallel index-level and subtree-level DiskTree checkpoint application correctness, keeps old roots readable, preserves deterministic table-root publication, and includes performance evidence.
28+
29+
## Notes (Optional)
30+
31+
32+
## Close Reason (Added When Closed)
33+
34+
When a backlog item is moved to `docs/backlogs/closed/`, append:
35+
36+
```md
37+
## Close Reason
38+
39+
- Type: <implemented|stale|replaced|duplicate|wontfix|already-implemented|other>
40+
- Detail: <reason detail>
41+
- Closed By: <backlog close>
42+
- Reference: <task/issue/pr reference>
43+
- Closed At: <YYYY-MM-DD>
44+
```

docs/backlogs/next-id

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
000084
1+
000085

docs/rfcs/0014-dual-tree-secondary-index.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -437,10 +437,10 @@ SAFETY:` comments, and run the repository lint gate. [D10], [C4], [C7]
437437
metadata.
438438
- Non-goals: changing foreground lookup or uniqueness behavior, changing
439439
catalog checkpointing, and independent MemTree flush.
440-
- Task Doc: `docs/tasks/TBD.md`
441-
- Task Issue: `#0`
442-
- Phase Status: `pending`
443-
- Implementation Summary: `pending`
440+
- Task Doc: `docs/tasks/000118-disk-tree-checkpoint-sidecar-publication.md`
441+
- Task Issue: `#554`
442+
- Phase Status: done
443+
- Implementation Summary: Implemented checkpoint sidecar publication for user-table secondary DiskTree roots. [Task Resolve Sync: docs/tasks/000118-disk-tree-checkpoint-sidecar-publication.md @ 2026-04-14]
444444

445445
- **Phase 3: Recovery Source Migration**
446446
- Scope: use checkpointed DiskTree roots as a validated recovery source

docs/tasks/000118-disk-tree-checkpoint-sidecar-publication.md

Lines changed: 329 additions & 0 deletions
Large diffs are not rendered by default.

docs/tasks/next-id

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
000118
1+
000119

doradb-storage/src/index/column_block_index.rs

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1483,6 +1483,34 @@ impl<'a> ColumnBlockIndex<'a> {
14831483
)
14841484
}
14851485

1486+
/// Loads validated delete deltas and authoritative row ids for one
1487+
/// persisted entry from one leaf-node read.
1488+
pub(crate) async fn load_delete_deltas_and_row_ids(
1489+
&self,
1490+
entry: &ColumnLeafEntry,
1491+
) -> Result<(Vec<u32>, Vec<RowID>)> {
1492+
let node = self.read_node(entry.leaf_page_id).await?;
1493+
let view = self.read_entry_view(&node, entry)?;
1494+
let row_set = decode_logical_row_set(&view, self.file_kind(), entry.leaf_page_id)?;
1495+
let delete_set = self
1496+
.decode_logical_delete_set(&view, &row_set, entry.leaf_page_id)
1497+
.await?;
1498+
let delete_deltas = match delete_set {
1499+
LogicalDeleteSet::None { .. } => Vec::new(),
1500+
LogicalDeleteSet::Inline { row_id_deltas, .. } => row_id_deltas,
1501+
LogicalDeleteSet::External { row_id_deltas, .. } => {
1502+
row_id_deltas.ok_or(Error::InvalidState)?
1503+
}
1504+
};
1505+
let row_ids = decode_row_ids_from_row_set(
1506+
view.start_row_id,
1507+
&row_set,
1508+
self.file_kind(),
1509+
entry.leaf_page_id,
1510+
)?;
1511+
Ok((delete_deltas, row_ids))
1512+
}
1513+
14861514
async fn decode_logical_delete_set(
14871515
&self,
14881516
view: &LeafEntryView<'_>,

0 commit comments

Comments
 (0)