Commit f03e1bc
authored
fix(ipc): handle duplicate projection indices in IPC reader (#9952)
# Which issue does this PR close?
- Closes #9950 .
# Rationale for this change
The current IPC reader does not correctly handle duplicate projection
indices.
`Schema::project`(in `arrow-schema/src/schema.rs`) and
`RecordBatch::project`(in `arrow-array/src/record_batch.rs`) both map
each requested index directly, preserve the projection order and allow
duplicate indices such as:
```rust id="n4pq0f"
vec![1, 1]
```
However, the IPC reader currently uses:
```rust id="gjklyo"
projection.iter().position(|p| p == &idx)
```
which only returns the first matching entry. As a result, only one
column is decoded even though the projected schema contains multiple
fields, leading to schema/column count mismatches when constructing the
`RecordBatch`.
This also affects reordered duplicate projections such as:
```rust id="jlwmku"
vec![2, 0, 2]
```
# What changes are included in this PR?
* Updated IPC projection handling in `arrow-ipc/src/reader.rs` to
preserve all matching projection entries
* Reused the decoded array for duplicate projection indices instead of
decoding the same field multiple times
* Preserved projection order for reordered duplicate projections
# Are these changes tested?
Yes.
Added `test_projection_duplicate_indices`, which verifies:
* duplicate projections (`vec![1, 1]`)
* reordered duplicate projections (`vec![2, 0, 2]`)
The test compares IPC projection results against `RecordBatch::project`.
The test fails before the fix and passes after it.
All existing `arrow-ipc` tests also pass `cargo test -p arrow-ipc --lib`
# Are there any user-facing changes?
No.1 parent 259cff2 commit f03e1bc
1 file changed
Lines changed: 38 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
560 | 560 | | |
561 | 561 | | |
562 | 562 | | |
563 | | - | |
564 | | - | |
565 | | - | |
566 | | - | |
567 | | - | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
568 | 577 | | |
569 | 578 | | |
570 | 579 | | |
| |||
2297 | 2306 | | |
2298 | 2307 | | |
2299 | 2308 | | |
| 2309 | + | |
| 2310 | + | |
| 2311 | + | |
| 2312 | + | |
| 2313 | + | |
| 2314 | + | |
| 2315 | + | |
| 2316 | + | |
| 2317 | + | |
| 2318 | + | |
| 2319 | + | |
| 2320 | + | |
| 2321 | + | |
| 2322 | + | |
| 2323 | + | |
| 2324 | + | |
| 2325 | + | |
| 2326 | + | |
| 2327 | + | |
| 2328 | + | |
| 2329 | + | |
| 2330 | + | |
| 2331 | + | |
| 2332 | + | |
2300 | 2333 | | |
2301 | 2334 | | |
2302 | 2335 | | |
| |||
0 commit comments