[WIP] feat(raster): view machinery for non-identity band views

[james-willis]

Summary

Adds the view-machinery layer on top of the N-D raster type. Bands can be constructed and read with non-identity view entries — slices, broadcasts, axis permutations, and reverse-iteration. The byte-access surface is a single zero-copy accessor, nd_buffer(), which exposes the source buffer plus the visible region's shape/strides/offset; consumers that need flat row-major bytes borrow them via NdBuffer::as_contiguous(), which errors on strided layouts rather than allocating. The slice/manipulation functions in #750 depend on this layer.

What's in this PR

Three layers, plus the GDAL bridge:

Schema / trait surface (rust/sedona-raster/src/traits.rs)

• validate_view() + unit tests (length checks, axis range, negative step, broadcast / step=0, i64 overflow guards).
• visible_shape_from_view() helper.
• NdBuffer::is_contiguous() — pure function of (shape, strides, data_type). C-order packed-stride check, innermost-first, offset-agnostic so an outer-axis slice from a nonzero offset still counts as contiguous. Broadcast (stride 0), reverse (negative stride), permuted, gapped, and inner-strided layouts are not contiguous; a zero-extent axis is trivially contiguous.
• NdBuffer::as_contiguous() -> Result<&[u8]> — borrows the visible bytes zero-copy when contiguous, else errors directing the caller to RS_EnsureContiguous (RS_EnsureContiguous UDF: materialize strided band views to contiguous bytes via an explicit plan node #899). Never allocates.

Builder (rust/sedona-raster/src/builder.rs)

• start_band_with_view() API (args bundled in StartBandWithViewArgs).
• with_view() API (args bundled in WithViewArgs) — compose a new view over an existing band.
• View-construction tests: slice, broadcast, transpose, negative-step, outer-axis-slice contiguity, multi-band mixed views, Arrow IPC round-trip.

Reader (rust/sedona-raster/src/array.rs)

• RasterRefImpl::band composes view → byte strides + byte offset using checked arithmetic; malformed views and arithmetic overflows route through sedona_internal_datafusion_err! so they surface with the standard "SedonaDB internal error" framing.
• nd_buffer() returns the source buffer + shape/strides/offset (zero-allocation strided access) and is the sole byte accessor. OutDb bands return an error from nd_buffer() (backend resolvers are tracked separately).

GDAL bridge (rust/sedona-raster-gdal/src/{gdal_common,gdal_dataset_provider}.rs)

• raster_ref_to_gdal_mem borrows each band via nd_buffer()?.as_contiguous()?. is_2d() already requires an identity view, so the borrow always succeeds — no owned-bytes copy is threaded through the dataset provider.

Behavior change vs main

• A band with a non-null view row now decodes via the composition path (previously rejected as a read error).
• Byte access never silently materializes a strided view. nd_buffer().as_contiguous() borrows zero-copy for contiguous layouts (identity views and offset-only outer-axis slices) and returns an error for strided ones. Row-major repacking is deferred to an explicit plan node, RS_EnsureContiguous (RS_EnsureContiguous UDF: materialize strided band views to contiguous bytes via an explicit plan node #899) — keeping the trait interface allocation-free.

Test plan

• cargo build -p sedona-schema -p sedona-raster -p sedona-raster-gdal -p sedona-raster-functions -p sedona-testing
• cargo test -p sedona-raster — 126 tests pass.
• cargo test -p sedona-raster-gdal — 53 tests pass.
• cargo clippy --all-targets -- -D warnings
• cargo fmt --all --check
• pre-commit run --from-ref origin/main --to-ref HEAD

[paleolimbot]

This is very cool!

I think I got there by the end in the inline comments, but I think a struct ViewEntries with associated functions in src/view_entries.rs where the non serialize-to-arrow and deserialize-from-arrow logic behind this lives will make this clearer.

The main issue I have with the implementation here is that it relies on hidden Vec<u8>s that aren't reusable. To manage memory properly we are going to have to register some scratch spaces with the session's memory pool and reuse them between iterations of a loop. Hiding them within various structures for convenience is going to make it tricky to implement a correct pattern going forward.

[paleolimbot]

A BandBuilder may help you quite a bit here since there are a lot of None that a default constructor could fill in

[james-willis]

I am generally unhappy with the band builder interfaces; I agree I am making them substantially worse. Not sure how you feel about deferring this but I started putting items for band builder improvements in this ticket:

#896

[paleolimbot]

These kinds of tests you probably don't want in this file. This file is about serializing stuff into Arrow and it's tricky to see what's going on...I'm not sure what the perfect abstraction is here but it should probably live in src/views.rs with tests and some pub functions that can be benchmarked.

Proabably struct ViewEntries { inner: Vec<ViewEntry> } with associated methods would make this a bit cleaner than the &[ViewEntry] + free functions.

[james-willis]

I reduce the tests here. they should now be just builder-integration specific. will confirm before resolving this comment

[james-willis]

Somewhat offtopic but are you going to have the same concern with lazy outdb loading as well? (Outdb lazy loading + zarr loader is going to be higher on my priority list next week than this PR)

The main issue I have with the implementation here is that it relies on hidden Vecs that aren't reusable. To manage memory properly we are going to have to register some scratch spaces with the session's memory pool and reuse them between iterations of a loop. Hiding them within various structures for convenience is going to make it tricky to implement a correct pattern going forward.

[paleolimbot]

Possibly? I know it seems like a lot of comments but what I'm getting at is:

let mut scratch = Vec::new();
for raster in rasters {
  let data_slice = raster.band(0)?.contiguous_data(&mut scratch)?;
  do_stuff(data_slice);
}

Or on the day we need to track memory:

let mut max_capacity = 0;
for raster in rasters {
  max_capacity = max_capacity.max(raster.band(0)?.contiguous_data_alloc_size());
}

if max_capacity > config_options.raster.max_scratch_alloc {
  // error
}

// If this ends up being a lot, frequently, we may also want to use a MemoryReservation to track it
// Probably this helps contiguous_data be faster since in theory there is only one heap allocation.
// You could probably use some of the unsafe modifiers to also ensure that there are no bounds
// checks in your materializer slowing things down
let mut scratch = Vec::with_capacity(max_capacity);
for raster in rasters {
  let data_slice = raster.band(0)?.contiguous_data(&mut scratch)?;
  do_stuff(data_slice);
}

I'm not sure exactly how loading works yet but you might want something similar (with the hiccup that you need some number of async workers doing IO so you also need the same number of scratch buffers).

(I have the same comment for pretty much everything that heap allocates in a loop, which is why, for example, our geometry writers write WKB directly into the Arrow output instead of returning Vec<u8>).