Skip to content

Memory consumption during IVF train prefetch #7126

Description

@geserdugarov

When a vector column is nullable, IVF training reads up to the requested sample size per prefetch round, while the non-nullable path reads exactly the requested size. The amplification is explicit in rust/lance/src/index/vector/utils.rs::sample_training_data_scan_from_fragments:

let target = sample_size_hint.saturating_mul(2)

and is the dominant peak-memory term for IVF training on nullable + fragment-limited inputs.

The over-fetch is also propagated into the consumer's output buffer.
sample_nullable_fsl only checks num_non_null < sample_size_hint before reading the next batch and then appends the whole filtered batch, so on low-null data the output MutableBuffer itself grows to roughly 2 × sample_size_hint × byte_width before the post-loop truncate.

A second, smaller amplification lives in sample_nullable_fallback where every prefetched batch is retained in a Vec<RecordBatch> and then materialised into one combined batch via concat_batches, doubling peak memory at the moment of concat.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions