feat(index): support raw-query ivf rq search#7078
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
github-actions
Bot
added
A-python
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
BubbleCal
force-pushed
the
yang/ivfrq-pr3-split-code-query
branch
from
ce548a4 to
763ecc0
Compare
|
@claude review |
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 42b122a38b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
This is a substantial new feature touching the Lance format spec, the IVF_RQ storage layout, and the core distance-calculator path — I'm leaving it for a human reviewer.
Extended reasoning...
Overview
This PR introduces a new raw_query estimator for IVF_RQ search alongside the existing residual_query estimator, including multi-bit (num_bits > 1) split-code search. Changes span 13 files: a format-spec doc update, new metadata fields (query_estimator), substantial additions to bq/storage.rs (~1100 lines including new ex-code distance tables, FastScan paths, and lower-bound gating), a refactor of the flat index search loops to use new accumulate_topk_with_scratch trait methods, a fix to PERM0_INVERSE in the SIMD dist table (the prior value was incorrect — note the new test_perm0_inverse_matches_perm0 test), and IVF v2 plumbing for per-query rotated-query contexts and per-index rotated-centroid caches.
Security risks
No direct security risks — this is internal indexing code with no auth, crypto, or untrusted-input boundaries.
Level of scrutiny
High. This PR touches the format specification (per the format-change-vote reminder, this needs a PMC vote with 3 binding +1s and a 1-week voting period), modifies the storage layout for newly-built IVF_RQ indexes, and rewrites the search hot path with new lower-bound gating that affects recall/accuracy. The compatibility shim (default_query_estimator_compat → ResidualQuery) is the right approach for old indexes, but the correctness of the new raw-query estimator factors, error-factor gating, and FastScan ex-code path all warrant careful human review.
Other factors
- The PR has good test coverage: new unit tests for the estimator factors, FastScan ex-code paths, lower-bound gating, cache slice borrowing, and a parameterized integration test for multi-bit L2/Cosine search.
- A previously-incorrect
PERM0_INVERSEconstant is silently fixed here — worth a human verifying no callers depended on the buggy value. - The Cosine→L2 distance-type rewrite in
try_from_batchfor raw-query indexes is subtle and could surprise readers; worth confirming the rewrite is correct and intentional. - The format-spec change requires a PMC vote that has not yet been linked on this PR.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
|
@claude review |
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
2 participants
Feature
query_estimatormetadata so released indexes without the field continue to read asresidual_query, while newly written indexes useraw_query.num_bits == 1indexes and multi-bit split-code indexes, including ex-code factors and runtime-only rotated centroid caches derived from the originalIvfModelcentroids.num_bits > 1gate for supported metrics, including cosine via Lance's normalized-L2 handling.Compatibility
query_estimatormetadata still default to the legacy residual-query estimator.Performance Improvement
The benchmark below was run with
search-benchmarkon GCP VMyang-agent-00bd-ivfrq-rerun-20260605, datasetgist,k=10,max_threads=1,target_partition_size=4096, no refine. Latencies are converted from CSV seconds to milliseconds.Provenance:
61ef8f7b97589032a83eeae1e52664be9f035551437849118f380d92c1ea849f99996e9072be58dfce548a49766670b80275daae6f1bf97c70e885e4Additional DBpedia comparison on the same VM, current branch only, dataset
dbpedia,k=10,max_threads=1,target_partition_size=4096. For IVF_PQ,sub_vector_dim=8; one extra row includesrefine_factor=2atnprobes=24.Tests
cargo fmt --allcargo test -p lance-index raw_querycargo test -p lance-index try_from_batch_cargo test -p lance-index rabit_quantizercargo test -p lance test_rabitq_distance_typescargo test -p lance test_build_ivf_rqcargo clippy -p lance-index -p lance --tests --benches -- -D warningsuv run make build