When a vector column is nullable, IVF training reads up to 2× the requested sample size per prefetch round, while the non-nullable path reads exactly the requested size. The amplification is explicit in rust/lance/src/index/vector/utils.rs::sample_training_data_scan_from_fragments:
let target = sample_size_hint.saturating_mul(2)
and is the dominant peak-memory term for IVF training on nullable + fragment-limited inputs.
The over-fetch is also propagated into the consumer's output buffer.
sample_nullable_fsl only checks num_non_null < sample_size_hint before reading the next batch and then appends the whole filtered batch, so on low-null data the output MutableBuffer itself grows to roughly 2 × sample_size_hint × byte_width before the post-loop truncate.
A second, smaller amplification lives in sample_nullable_fallback where every prefetched batch is retained in a Vec<RecordBatch> and then materialised into one combined batch via concat_batches, doubling peak memory at the moment of concat.
When a vector column is
nullable, IVF training reads up to2×the requested sample size per prefetch round, while the non-nullable path reads exactly the requested size. The amplification is explicit inrust/lance/src/index/vector/utils.rs::sample_training_data_scan_from_fragments:and is the dominant peak-memory term for IVF training on nullable + fragment-limited inputs.
The over-fetch is also propagated into the consumer's output buffer.
sample_nullable_fslonly checksnum_non_null < sample_size_hintbefore reading the next batch and then appends the whole filtered batch, so on low-null data the outputMutableBufferitself grows to roughly2 × sample_size_hint × byte_widthbefore the post-loop truncate.A second, smaller amplification lives in
sample_nullable_fallbackwhere every prefetched batch is retained in aVec<RecordBatch>and then materialised into one combined batch viaconcat_batches, doubling peak memory at the moment of concat.