You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@sedona.apache.org by ja...@apache.org on 2026/08/28 17:51:38 UTC
(sedona-db) 01/01: feat(rust/sedona-raster): array-level column overrides for metadata setters
This is an automated email from the ASF dual-hosted git repository.
james-willis pushed a commit to branch james/db-99-array-level-override
in repository https://gitbox.apache.org/repos/asf/sedona-db.git
commit 0040157d22abae19fab29bbeefbf6dd09f2bd4b9
Author: jameswillis <ja...@wherobots.com>
AuthorDate: Fri Aug 28 10:51:24 2026 -0700
feat(rust/sedona-raster): array-level column overrides for metadata setters
Closes DB-99.
`RS_SetCRS`, `RS_SetSRID` and `RS_SetGeoReference` change only top-level
metadata — one of five columns on the raster `StructArray`. The bands column,
which holds every pixel byte, is untouched by definition.
Two ways to produce that output existed, and the setters were split across
both:
* `swap_crs_column` (private to `rs_setsrid.rs`, hardcoded to the CRS column)
replaced one column and carried the rest over as the same `Arc`. Correct,
but unreachable from anywhere else.
* `RasterBuilder::copy_raster_from` walked every band and re-emitted every
band field into a fresh builder. `RS_SetGeoReference` used this. The pixel
buffers were shared by refcount, but all the band metadata was rebuilt to
change one number — and the bands came out identical only if the rebuild
was faithful. DB-506 is what happens when it isn't: that rebuild silently
dropped band names.
Generalize the first into a shared `with_column_overrides` and move all three
setters onto it.
* `RasterColumnOverrides { crs, transform }` uses the `Override<T>` trinary
from #1158. `Keep` leaves a column in place sharing its `Arc`, `Clear` nulls
it, `Set(a)` installs a per-row array. `Override` rather than `Option`
because `RS_SetSRID(raster, 0)` must *clear* the CRS while keeping the
raster — distinct from leaving the column alone, which `Option` cannot
express. That is why `swap_crs_column` was bespoke.
* The helper also absorbs the input-null merge that was duplicated inline at
both `rs_setsrid.rs` call sites, including the length-1 broadcast for scalar
arguments. A null *argument* nulls the whole raster row; that is a separate
axis from `Clear`ing a column.
* It is a free function taking `&StructArray`, not a method on
`RasterStructArray`, so it does not require parsing and validating every
column for what is pure column surgery — and so introduces no failure mode
`swap_crs_column` did not have.
* `RS_SetGeoReference` builds a `List<Float64>` transform column directly and
drops `copy_raster_from`. Its bands are now provably untouched.
Note this keeps two override vocabularies on purpose. `RasterOverrides` stays
the *builder's* language — a scalar applied while a row is constructed. The
array path replaces whole columns and takes per-row values already in an
array. Forcing the setters onto the scalar struct would regress `RS_SetSRID`
from per-row CRS (`broadcast_string_view` passes an N-row array straight
through) to a single CRS per call. They share `Override<T>`, which is the
vocabulary that matters.
`with_column_overrides_shares_untouched_columns` asserts `Arc::ptr_eq` on
every carried-over column, which is the only assertion that distinguishes a
shared column from an equal-looking rebuild.
---
.../src/rs_set_georeference.rs | 87 ++++++---
rust/sedona-raster-functions/src/rs_setsrid.rs | 83 +++-----
rust/sedona-raster/src/array.rs | 214 ++++++++++++++++++++-
3 files changed, 295 insertions(+), 89 deletions(-)
diff --git a/rust/sedona-raster-functions/src/rs_set_georeference.rs b/rust/sedona-raster-functions/src/rs_set_georeference.rs
index 1af880477..69c3eb432 100644
--- a/rust/sedona-raster-functions/src/rs_set_georeference.rs
+++ b/rust/sedona-raster-functions/src/rs_set_georeference.rs
@@ -31,23 +31,27 @@
//! The shift uses the full affine (scale and skew halves), so it is exact for
//! skewed rasters and round-trips with the getter.
//!
-//! The raster is rebuilt with [`RasterBuilder::copy_raster_from`] overriding the
-//! transform; each band's pixel buffers are shared zero-copy, and the CRS plus
-//! every inherited band field — name, nodata, OutDb pointers — are carried over
-//! unchanged.
+//! This is a top-level metadata change, so the raster is edited at the array
+//! level: only the transform column is rebuilt and every other column — the
+//! CRS, the spatial dims/shape, and the whole bands column — is carried over as
+//! the same `Arc`. The bands are therefore byte-for-byte identical by
+//! construction; they are never read, copied, or rebuilt.
use std::sync::Arc;
-use arrow_array::Array;
+use arrow_array::{Array, ArrayRef, Float64Array, ListArray, StructArray};
+use arrow_buffer::{NullBuffer, OffsetBuffer};
use arrow_schema::DataType;
use datafusion_common::cast::as_string_array;
use datafusion_common::error::Result;
-use datafusion_common::{exec_datafusion_err, exec_err};
+use datafusion_common::{arrow_datafusion_err, exec_datafusion_err, exec_err};
use datafusion_expr::{ColumnarValue, Volatility};
use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
-use sedona_raster::builder::{RasterBuilder, RasterOverrides};
+use sedona_raster::array::{with_column_overrides, RasterColumnOverrides};
+use sedona_raster::traits::Override;
use sedona_schema::datatypes::SedonaType;
use sedona_schema::matchers::ArgMatcher;
+use sedona_schema::raster::RasterSchema;
use crate::executor::RasterExecutor;
// Shared with the getter so the two agree on accepted format strings and the
@@ -111,18 +115,33 @@ impl SedonaScalarKernel for RsSetGeoReference {
.map(|a| as_string_array(a))
.transpose()?;
- let mut builder = RasterBuilder::new(n);
- executor.execute_raster_void(|i, raster_opt| {
- let null_out = |b: &mut RasterBuilder| b.append_null().map_err(Into::into);
+ // Every row contributes six transform values; a null row still
+ // contributes six placeholders so the offsets stay uniform, because the
+ // struct row's null bit — not the list — is what makes it null.
+ let raster_array = args[0].clone().into_array(n)?;
+ let raster_struct = raster_array
+ .as_any()
+ .downcast_ref::<StructArray>()
+ .ok_or_else(|| exec_datafusion_err!("Expected StructArray for raster data"))?;
+ let mut values: Vec<f64> = Vec::with_capacity(n * 6);
+ let mut row_valid: Vec<bool> = Vec::with_capacity(n);
+
+ for i in 0..n {
// A null georef or format is a null *input* and yields a null raster
// (matching RS_SetCRS); check those first so a null-driven row never
// raises an error.
if georef_array.is_null(i) {
- return null_out(&mut builder);
+ values.extend_from_slice(&[0.0; 6]);
+ row_valid.push(false);
+ continue;
}
let format = match &format_array {
- Some(fmt) if fmt.is_null(i) => return null_out(&mut builder),
+ Some(fmt) if fmt.is_null(i) => {
+ values.extend_from_slice(&[0.0; 6]);
+ row_valid.push(false);
+ continue;
+ }
Some(fmt) => GeoReferenceFormat::from_str(fmt.value(i))?,
None => GeoReferenceFormat::Gdal,
};
@@ -133,20 +152,36 @@ impl SedonaScalarKernel for RsSetGeoReference {
// null raster.
let transform = parse_georeference(georef_array.value(i), format)?;
- let Some(raster) = raster_opt else {
- return null_out(&mut builder);
- };
- builder
- .copy_raster_from(
- raster,
- RasterOverrides {
- transform: Some(transform),
- },
- )
- .map_err(Into::into)
- })?;
-
- executor.finish(Arc::new(builder.finish()?))
+ if raster_struct.is_null(i) {
+ values.extend_from_slice(&[0.0; 6]);
+ row_valid.push(false);
+ continue;
+ }
+ values.extend_from_slice(&transform);
+ row_valid.push(true);
+ }
+
+ let DataType::List(transform_field) = RasterSchema::transform_type() else {
+ return exec_err!("Expected list type for transform");
+ };
+ let transform: ArrayRef = Arc::new(ListArray::new(
+ transform_field,
+ OffsetBuffer::from_lengths(std::iter::repeat_n(6usize, n)),
+ Arc::new(Float64Array::from(values)),
+ None,
+ ));
+
+ let out = with_column_overrides(
+ raster_struct,
+ RasterColumnOverrides {
+ transform: Override::Set(transform),
+ ..Default::default()
+ },
+ Some(&NullBuffer::from(row_valid)),
+ )
+ .map_err(|e| arrow_datafusion_err!(e.into()))?;
+
+ executor.finish(Arc::new(out))
}
}
diff --git a/rust/sedona-raster-functions/src/rs_setsrid.rs b/rust/sedona-raster-functions/src/rs_setsrid.rs
index 96735346c..a179a2e51 100644
--- a/rust/sedona-raster-functions/src/rs_setsrid.rs
+++ b/rust/sedona-raster-functions/src/rs_setsrid.rs
@@ -23,15 +23,16 @@ use arrow_buffer::NullBuffer;
use arrow_schema::DataType;
use datafusion_common::cast::{as_int64_array, as_string_view_array};
use datafusion_common::error::Result;
-use datafusion_common::{exec_err, DataFusionError, ScalarValue};
+use datafusion_common::{arrow_datafusion_err, DataFusionError, ScalarValue};
use datafusion_expr::{ColumnarValue, Volatility};
use sedona_common::{sedona_internal_datafusion_err, sedona_internal_err};
use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
use sedona_geometry::transform::CrsEngine;
+use sedona_raster::array::{with_column_overrides, RasterColumnOverrides};
+use sedona_raster::traits::Override;
use sedona_schema::crs::{normalize_crs, CachedSRIDToCrs};
use sedona_schema::datatypes::SedonaType;
use sedona_schema::matchers::ArgMatcher;
-use sedona_schema::raster::{raster_indices, RasterSchema};
/// RS_SetSRID() scalar UDF implementation
///
@@ -164,6 +165,22 @@ fn replace_raster_crs(
crs_array: &StringViewArray,
input_nulls: Option<NullBuffer>,
) -> Result<ColumnarValue> {
+ // Array-level edit: only the CRS column is rebuilt. Every other column —
+ // crucially the bands — is carried over as the same `Arc`, so the pixel
+ // bytes are untouched by construction rather than rebuilt.
+ let edit = |raster_struct: &StructArray| -> Result<StructArray> {
+ let crs: ArrayRef = broadcast_string_view(crs_array, raster_struct.len())?;
+ with_column_overrides(
+ raster_struct,
+ RasterColumnOverrides {
+ crs: Override::Set(crs),
+ ..Default::default()
+ },
+ input_nulls.as_ref(),
+ )
+ .map_err(|e| arrow_datafusion_err!(e.into()))
+ };
+
match raster_arg {
ColumnarValue::Array(raster_array) => {
let raster_struct = raster_array
@@ -172,56 +189,15 @@ fn replace_raster_crs(
.ok_or_else(|| {
sedona_internal_datafusion_err!("Expected StructArray for raster data")
})?;
-
- let num_rows = raster_struct.len();
- let new_crs: ArrayRef = broadcast_string_view(crs_array, num_rows)?;
- let new_struct = swap_crs_column(raster_struct, new_crs)?;
-
- let input_nulls = input_nulls.map(|nulls| {
- if nulls.len() == 1 && num_rows != 1 {
- if nulls.is_valid(0) {
- NullBuffer::new_valid(num_rows)
- } else {
- NullBuffer::new_null(num_rows)
- }
- } else {
- nulls
- }
- });
-
- // Merge input nulls: rows where the SRID/CRS input was null become null rasters
- let merged_nulls = NullBuffer::union(new_struct.nulls(), input_nulls.as_ref());
- let new_struct = StructArray::new(
- RasterSchema::fields(),
- new_struct.columns().to_vec(),
- merged_nulls,
- );
-
- Ok(ColumnarValue::Array(Arc::new(new_struct)))
- }
- ColumnarValue::Scalar(ScalarValue::Struct(arc_struct)) => {
- let new_crs: ArrayRef = Arc::new(crs_array.clone());
- let new_struct = swap_crs_column(arc_struct.as_ref(), new_crs)?;
-
- // Merge input nulls: null SRID/CRS input produces a null raster
- let merged_nulls = NullBuffer::union(new_struct.nulls(), input_nulls.as_ref());
- let new_struct = StructArray::new(
- RasterSchema::fields(),
- new_struct.columns().to_vec(),
- merged_nulls,
- );
-
- Ok(ColumnarValue::Scalar(ScalarValue::Struct(Arc::new(
- new_struct,
- ))))
+ Ok(ColumnarValue::Array(Arc::new(edit(raster_struct)?)))
}
- ColumnarValue::Scalar(ScalarValue::Null) => Ok(ColumnarValue::Scalar(ScalarValue::Null)),
- _ => exec_err!("Expected raster (Struct) input for RS_SetSRID/RS_SetCRS"),
+ ColumnarValue::Scalar(ScalarValue::Struct(arc_struct)) => Ok(ColumnarValue::Scalar(
+ ScalarValue::Struct(Arc::new(edit(arc_struct.as_ref())?)),
+ )),
+ _ => sedona_internal_err!("Expected raster array or struct scalar"),
}
}
-/// Broadcast a `StringViewArray` to a target length.
-///
/// If the array already has the target length, it is returned as-is (clone of Arc).
/// Otherwise the array must have length 1, and its single value is repeated.
fn broadcast_string_view(array: &StringViewArray, len: usize) -> Result<ArrayRef> {
@@ -246,17 +222,6 @@ fn broadcast_string_view(array: &StringViewArray, len: usize) -> Result<ArrayRef
}
}
-/// Swap only the CRS column of a raster StructArray, keeping all other columns intact.
-fn swap_crs_column(raster_struct: &StructArray, new_crs_array: ArrayRef) -> Result<StructArray> {
- let mut columns: Vec<ArrayRef> = raster_struct.columns().to_vec();
- columns[raster_indices::CRS] = new_crs_array;
- Ok(StructArray::new(
- RasterSchema::fields(),
- columns,
- raster_struct.nulls().cloned(),
- ))
-}
-
/// Extract a [NullBuffer] from the original SRID/CRS input argument.
///
/// For arrays, this returns the array's null buffer directly.
diff --git a/rust/sedona-raster/src/array.rs b/rust/sedona-raster/src/array.rs
index 719e0a1c8..d1771dbac 100644
--- a/rust/sedona-raster/src/array.rs
+++ b/rust/sedona-raster/src/array.rs
@@ -16,9 +16,11 @@
// under the License.
use arrow_array::{
- Array, BinaryArray, BinaryViewArray, Float64Array, Int64Array, ListArray, StringArray,
- StringViewArray, StructArray, UInt32Array,
+ new_null_array, Array, ArrayRef, BinaryArray, BinaryViewArray, Float64Array, Int64Array,
+ ListArray, StringArray, StringViewArray, StructArray, UInt32Array,
};
+use arrow_buffer::NullBuffer;
+use arrow_schema::DataType;
use datafusion_common::cast::{
as_binary_array, as_binary_view_array, as_float64_array, as_int64_array, as_list_array,
as_string_array, as_string_view_array, as_struct_array, as_uint32_array,
@@ -26,9 +28,11 @@ use datafusion_common::cast::{
use crate::band_builder::BandWriter;
use crate::error::RasterError;
-use crate::traits::{BandRef, NdBuffer, RasterRef};
+use crate::traits::{BandRef, NdBuffer, Override, RasterRef};
use crate::view_entries::{ViewEntries, ViewEntry};
-use sedona_schema::raster::{band_indices, band_view_indices, raster_indices, BandDataType};
+use sedona_schema::raster::{
+ band_indices, band_view_indices, raster_indices, BandDataType, RasterSchema,
+};
/// Arrow-backed implementation of BandRef for a single band within a raster.
///
@@ -409,6 +413,74 @@ fn check_view_buffer_bounds(
Ok(())
}
+/// A new raster `StructArray` with specific top-level columns replaced,
+/// sharing every untouched column's `Arc` with the input.
+///
+/// This is the array-level edit path, for setters that change only
+/// top-level metadata (`RS_SetCRS`, `RS_SetSRID`, `RS_SetGeoReference`).
+/// The bands column is never read, copied, or rebuilt, so the output's
+/// bands are byte-for-byte identical to the input's *by construction* —
+/// unlike the builder path
+/// ([`RasterBuilder::copy_raster_from`](crate::builder::RasterBuilder::copy_raster_from)),
+/// where identical output depends on the rebuild reproducing every band
+/// field faithfully.
+///
+/// `input_nulls` nulls whole rasters where the *setter's own argument* was
+/// null, and is unioned with the input raster's existing nulls. A length-1
+/// buffer is broadcast to every row, for the scalar-argument case. Note
+/// this is distinct from an override that [`Override::Clear`]s a column:
+/// `RS_SetSRID(raster, 0)` clears the CRS but keeps the raster, whereas
+/// `RS_SetSRID(raster, NULL)` nulls the raster itself.
+pub fn with_column_overrides(
+ raster: &StructArray,
+ overrides: RasterColumnOverrides,
+ input_nulls: Option<&NullBuffer>,
+) -> Result<StructArray, RasterError> {
+ let num_rows = raster.len();
+ // `to_vec` clones five `Arc`s, not five columns.
+ let mut columns: Vec<ArrayRef> = raster.columns().to_vec();
+
+ apply_column_override(
+ &mut columns,
+ raster_indices::CRS,
+ overrides.crs,
+ num_rows,
+ RasterSchema::crs_type(),
+ "crs",
+ )?;
+ apply_column_override(
+ &mut columns,
+ raster_indices::TRANSFORM,
+ overrides.transform,
+ num_rows,
+ RasterSchema::transform_type(),
+ "transform",
+ )?;
+
+ let broadcast;
+ let input_nulls = match input_nulls {
+ Some(nulls) if nulls.len() == num_rows => Some(nulls),
+ Some(nulls) if nulls.len() == 1 => {
+ broadcast = if nulls.is_valid(0) {
+ NullBuffer::new_valid(num_rows)
+ } else {
+ NullBuffer::new_null(num_rows)
+ };
+ Some(&broadcast)
+ }
+ Some(nulls) => {
+ return Err(RasterError::Invalid(format!(
+ "with_column_overrides: input nulls have {} rows, expected {num_rows} or 1",
+ nulls.len()
+ )))
+ }
+ None => None,
+ };
+ let nulls = NullBuffer::union(raster.nulls(), input_nulls);
+
+ Ok(StructArray::new(RasterSchema::fields(), columns, nulls))
+}
+
impl<'a> RasterRef for RasterRefImpl<'a> {
fn num_bands(&self) -> usize {
self.bands_list.value_length(self.raster_index) as usize
@@ -664,6 +736,53 @@ pub struct RasterStructArray<'a> {
band_data_array: &'a BinaryViewArray,
}
+/// Array-level overrides for [`with_column_overrides`].
+/// Each field replaces one whole top-level column:
+///
+/// - [`Override::Keep`] leaves the input's column in place, sharing its `Arc`.
+/// - [`Override::Clear`] replaces it with an all-null column.
+/// - [`Override::Set(a)`](Override::Set) installs `a`, which must have one
+/// entry per raster row.
+///
+/// This is the array-level counterpart to
+/// [`RasterOverrides`](crate::builder::RasterOverrides), which is the
+/// *builder's* vocabulary: a scalar value applied while a row is constructed.
+/// The two share [`Override`] but not a struct, because the operations differ
+/// — the builder emits rows one at a time, while this replaces a column
+/// wholesale and so takes per-row values already in an array.
+#[derive(Debug, Default, Clone)]
+pub struct RasterColumnOverrides {
+ /// Override the CRS column (`Utf8View`).
+ pub crs: Override<ArrayRef>,
+ /// Override the geotransform column (`List<Float64>`, 6 values per row).
+ pub transform: Override<ArrayRef>,
+}
+
+/// Resolve one column override in place against `columns`.
+fn apply_column_override(
+ columns: &mut [ArrayRef],
+ index: usize,
+ over: Override<ArrayRef>,
+ num_rows: usize,
+ data_type: DataType,
+ label: &str,
+) -> Result<(), RasterError> {
+ match over {
+ Override::Keep => {}
+ Override::Clear => columns[index] = new_null_array(&data_type, num_rows),
+ Override::Set(array) => {
+ if array.len() != num_rows {
+ return Err(RasterError::Invalid(format!(
+ "with_column_overrides: {label} override has {} rows, expected {num_rows}",
+ array.len()
+ )));
+ }
+ columns[index] = array;
+ }
+ }
+ Ok(())
+}
+
impl<'a> RasterStructArray<'a> {
/// Create a new RasterStructArray from an existing StructArray.
///
@@ -827,6 +946,93 @@ mod tests {
use sedona_testing::rasters::generate_test_rasters;
use std::sync::Arc;
+ #[test]
+ fn with_column_overrides_shares_untouched_columns() {
+ use sedona_schema::raster::raster_indices;
+
+ let transform = [0.0, 1.0, 0.0, 0.0, 0.0, -1.0];
+ let mut b = RasterBuilder::new(1);
+ b.start_raster_nd(&transform, &["x"], &[2], Some("OGC:CRS84"))
+ .unwrap();
+ b.start_band(StartBandArgs::new(&["x"], &[2], BandDataType::UInt8))
+ .unwrap();
+ b.band_data_writer().append_value([1u8, 2]);
+ b.finish_band().unwrap();
+ b.finish_raster().unwrap();
+ let input = b.finish().unwrap();
+
+ let new_crs: ArrayRef = Arc::new(StringViewArray::from(vec![Some("EPSG:4326")]));
+ let out = with_column_overrides(
+ &input,
+ RasterColumnOverrides {
+ crs: Override::Set(new_crs),
+ ..Default::default()
+ },
+ None,
+ )
+ .unwrap();
+
+ // The replaced column is new; every other column is the *same
+ // allocation*, which is the whole point of the array-level edit —
+ // `Arc::ptr_eq` is the only assertion that distinguishes a shared
+ // column from an equal-looking rebuild.
+ assert!(!Arc::ptr_eq(
+ input.column(raster_indices::CRS),
+ out.column(raster_indices::CRS)
+ ));
+ for idx in [
+ raster_indices::TRANSFORM,
+ raster_indices::SPATIAL_DIMS,
+ raster_indices::SPATIAL_SHAPE,
+ raster_indices::BANDS,
+ ] {
+ assert!(
+ Arc::ptr_eq(input.column(idx), out.column(idx)),
+ "column {idx} was rebuilt rather than shared"
+ );
+ }
+ }
+
+ #[test]
+ fn with_column_overrides_clear_and_null_merge() {
+ use sedona_schema::raster::raster_indices;
+
+ let transform = [0.0, 1.0, 0.0, 0.0, 0.0, -1.0];
+ let mut b = RasterBuilder::new(1);
+ b.start_raster_nd(&transform, &["x"], &[2], Some("OGC:CRS84"))
+ .unwrap();
+ b.start_band(StartBandArgs::new(&["x"], &[2], BandDataType::UInt8))
+ .unwrap();
+ b.band_data_writer().append_value([1u8, 2]);
+ b.finish_band().unwrap();
+ b.finish_raster().unwrap();
+ let input = b.finish().unwrap();
+
+ // `Clear` nulls the column but leaves the raster row valid — this is
+ // RS_SetSRID(raster, 0): the CRS goes away, the raster does not.
+ let cleared = with_column_overrides(
+ &input,
+ RasterColumnOverrides {
+ crs: Override::Clear,
+ ..Default::default()
+ },
+ None,
+ )
+ .unwrap();
+ assert!(cleared.column(raster_indices::CRS).is_null(0));
+ assert!(cleared.is_valid(0), "Clear must not null the raster itself");
+
+ // `input_nulls` is the other axis: a null *argument* nulls the row.
+ // This is RS_SetSRID(raster, NULL).
+ let nulled = with_column_overrides(
+ &input,
+ RasterColumnOverrides::default(),
+ Some(&NullBuffer::new_null(1)),
+ )
+ .unwrap();
+ assert!(nulled.is_null(0), "a null input argument must null the row");
+ }
+
#[test]
fn band_ref_exposes_its_name() {
// `name()` is the band-level accessor that lets `copy_into` inherit the