Skip to content

Add configurable scalar index on id column#50

Merged
beinan merged 2 commits into
lance-format:mainfrom
beinan:feat/id-index
Jun 7, 2026
Merged

Add configurable scalar index on id column#50
beinan merged 2 commits into
lance-format:mainfrom
beinan:feat/id-index

Conversation

@beinan

@beinan beinan commented Jun 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add IdIndexType enum (None/BTree/ZoneMap) for configurable scalar index on the id column
  • BTree indices are maintained by MemWAL during writes; ZoneMap indices are excluded from MemWAL maintenance (unsupported by MemWAL) and rebuilt after compaction
  • Index is created eagerly on store open and ensured after each compaction cycle
  • Exposed via Python API as id_index_type parameter ("btree", "zonemap", or None)

Test plan

  • Rust tests: btree index creation on open, persistence after compaction
  • Rust tests: zonemap index creation on open, persistence after compaction
  • Rust tests: no index created by default (IdIndexType::None)
  • Rust tests: idempotent ensure_id_index (no version bump on re-check)
  • Python tests: btree/zonemap creation, invalid type rejection, data integrity across compaction cycles
  • Full existing test suite passes (19/19)

🤖 Generated with Claude Code

Beinan Wang added 2 commits June 7, 2026 01:49
Support btree and zonemap index types on the id column via
IdIndexType enum, configurable through both Rust and Python APIs.
BTree indices are maintained by MemWAL during writes; ZoneMap
indices are excluded from MemWAL maintenance (unsupported) and
rebuilt after compaction.

Co-Authored-By: Beinan Wang <beinanwang@microsoft.com>
Co-Authored-By: Beinan Wang <beinanwang@microsoft.com>
@beinan beinan marked this pull request as ready for review June 7, 2026 20:45
@beinan beinan merged commit 1b3ce5d into lance-format:main Jun 7, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant