You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "james-willis (via GitHub)" <gi...@apache.org> on 2026/09/22 23:08:02 UTC
[PR] feat(rust/sedona-raster-functions): RS_SetBandNoDataValue replace overload [sedona-db]
james-willis opened a new pull request, #1343:
URL: https://github.com/apache/sedona-db/pull/1343
Adds the 4-argument `RS_SetBandNoDataValue(raster, band, nodata, replace)` form that Sedona Spark has. With `replace` true, every pixel in the addressed band holding that band's current nodata is rewritten to the new sentinel before it is declared — so the pixels that read as nodata before the call still read as nodata after it. Other bands are untouched.
Probed on both engines first, as usual for this suite.
## Design notes
**Byte-level matching.** The comparison is on raw little-endian bytes rather than decoded values, which makes it exact for every band data type without a dispatch over them: the new sentinel arrives already packed into the band's type by `nodata_f64_to_bytes`, so both sides are the same width by construction. The one behavior this inherits is that a NaN sentinel matches only pixels carrying the identical NaN bit pattern, where Sedona Spark compares numerically and NaN never equals itself. That is documented on the helper.
**Replacing against no existing nodata is an error.** There is nothing to match, and Sedona Spark raises there too.
**A null `nodata` still clears the band**, as in the 3-argument form. That leaves no sentinel to rewrite pixels to, so `replace` is moot and no pixel is touched.
## The `needs_pixels` tradeoff — worth a look
This form rewrites pixels, so unlike the metadata-only forms it cannot share the source buffer. Reading those bytes needs the band materialized InDb, which on an OutDb raster means the planner has to inject `RS_EnsureLoaded` — and that is keyed off `needs_pixels`, which is **per-UDF, not per-kernel**.
So tagging the UDF makes the 2- and 3-argument forms materialize too, where previously they passed an OutDb raster straight through via `copy_into`. Nothing in the suites regresses, but it is a real behavior change for a metadata-only call on a large OutDb raster.
The alternatives I considered were erroring on OutDb and telling the caller to wrap in `RS_EnsureLoaded` (honest, but a parity gap — Spark just works), or making the planner rule arity-aware (correct, but a cross-crate change to a rule with a duplicated string literal, well outside this change). I took the simple option; happy to switch if you'd rather not pay that.
Without the tag this does not work on the parity fixtures at all — they read rasters from GeoTIFF, so every band is OutDb. The Rust unit tests use in-memory rasters and never see it, which is exactly the hole the parity suite caught.
## Parity
The single-band replace case **agrees on the released Sedona jar** now that this kernel exists, and loses its xfail. Its previous reason claimed the Spark 4.1 binding could not evaluate the flag at all; that turns out to be wrong for the single-band case, which the released 1.9.1 jar handles correctly.
The multiband case still trips apache/sedona#3330, where Sedona Spark zeroes every band except the target. That is fixed on Sedona master by apache/sedona#3347, and I verified it there — Sedona publishes no usable Maven snapshot (the Apache Nexus snapshots stop at 1.8.1-SNAPSHOT with no Spark 4 artifact), so I ran the suite against a jar built by Sedona's own CI, where the case passes against the anchor. No release carries the fix yet and a release is what CI installs, so the case keeps an xfail naming the release it is waiting on, and will flip green on its own.
The first commit adds the `SEDONADB_SEDONA_SPARK_JARS` hook that made that check possible. It is deliberately not wired into CI: pinning the suite to a 90-day build artifact from another repo would be fragile. It is a local tool for confirming an upstream fix landed so our side can be implemented without waiting on a release.
## Verification
- `cargo test -p sedona-raster-functions --lib` — 280 passed
- Parity suite on the released jar — 205 passed, 100 xfailed, 0 xpassed
- Parity suite on a Sedona-master jar — the multiband case additionally xpasses
- Full Python suite — no new failures against the pre-existing baseline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-raster-functions): RS_SetBandNoDataValue replace overload [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #1343:
URL: https://github.com/apache/sedona-db/pull/1343#discussion_r4107334052
##########
rust/sedona-raster-functions/src/rs_set_band_nodata.rs:
##########
@@ -50,28 +64,46 @@ use datafusion_common::error::Result;
use datafusion_common::exec_err;
use datafusion_expr::{ColumnarValue, Volatility};
use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
-use sedona_raster::builder::{RasterBuilder, RasterOverrides};
-use sedona_raster::traits::{BandOverrides, Override, RasterRef, nodata_f64_to_bytes};
+use sedona_raster::builder::{RasterBuilder, RasterOverrides, StartBandArgs};
+use sedona_raster::traits::{BandOverrides, BandRef, Override, RasterRef, nodata_f64_to_bytes};
use sedona_schema::datatypes::SedonaType;
use sedona_schema::matchers::ArgMatcher;
use crate::executor::RasterExecutor;
+use crate::rs_ensure_loaded::NEEDS_PIXELS_METADATA_KEY;
/// RS_SetBandNoDataValue() scalar UDF implementation
pub fn rs_set_band_nodata_value_udf() -> SedonaScalarUDF {
SedonaScalarUDF::new(
"rs_setbandnodatavalue",
vec![
- Arc::new(RsSetBandNoDataValue { with_band: false }),
- Arc::new(RsSetBandNoDataValue { with_band: true }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: false,
+ with_replace: false,
+ }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: true,
+ with_replace: false,
+ }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: true,
+ with_replace: true,
+ }),
],
Volatility::Immutable,
)
+ // The replace kernel rewrites pixel bytes, so the raster argument must be
+ // materialised InDb first; the planner injects RS_EnsureLoaded on this flag.
+ .with_metadata(NEEDS_PIXELS_METADATA_KEY, "true")
Review Comment:
yeah if we have needs_pixels generally youll have to annotate returns_bytes as well because of the double-async-udf-in-one-expression bug. but I think we will abandon this PR in favor of dewey's proposal. on that new function we will make sure to have both metadata entries
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-raster-functions): add RS_ReplaceBandNoDataValue [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #1343:
URL: https://github.com/apache/sedona-db/pull/1343#discussion_r4158456758
##########
rust/sedona-raster-functions/src/rs_replace_band_nodata.rs:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `RS_ReplaceBandNoDataValue` — move a band's nodata to a new value, carrying
+//! the nodata pixels with it.
+//!
+//! ```text
+//! RS_ReplaceBandNoDataValue(raster, band, nodata) -> Raster
+//! ```
+//!
+//! Every pixel of the addressed band that reads as nodata today is rewritten to
+//! `nodata`, which then becomes the band's nodata value, so the same pixels
+//! read as nodata before and after the call. `RS_SetBandNoDataValue` only
+//! changes the declared value and leaves the pixels alone.
+//!
+//! A pixel reads as nodata by the same rules sampling uses
+//! ([`NodataMatcher`]): exact for integer bands, numeric for float bands, so
+//! `-0.0` matches a `0.0` nodata and any NaN matches a NaN nodata.
+//!
+//! The band must already have a nodata value; with none there is nothing to
+//! replace, so that is an error. A null raster, band or `nodata` yields a null
+//! raster. Other bands are carried over untouched and zero-copy. Only 2-D
+//! bands are supported.
+//!
+//! There is deliberately no form without a band: Sedona Spark's 2-argument
+//! forms default to band 1 where SedonaDB's reject a multiband raster, and a
+//! new function need not inherit that divergence.
+
+use std::ops::ControlFlow;
+use std::sync::Arc;
+
+use arrow_array::Array;
+use arrow_array::cast::AsArray;
+use arrow_array::types::{Float64Type, Int64Type};
+use arrow_buffer::Buffer;
+use arrow_schema::DataType;
+use datafusion_common::{Result, exec_datafusion_err, exec_err};
+use datafusion_expr::{ColumnarValue, Volatility};
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_raster::band_builder::check_band_data_len;
+use sedona_raster::builder::{RasterBuilder, RasterOverrides};
+use sedona_raster::traits::{BandOverrides, BandRef, Override, RasterRef, nodata_f64_to_bytes};
+use sedona_schema::datatypes::SedonaType;
+use sedona_schema::matchers::ArgMatcher;
+
+use crate::executor::RasterExecutor;
+use crate::pixel_scan::{NodataMatcher, scan_pixels, spatial_2d_buffer};
+use crate::rs_ensure_loaded::{NEEDS_PIXELS_METADATA_KEY, RETURNS_BYTES_METADATA_KEY};
+use crate::sampling::resolve_band;
+
+const FUNC: &str = "RS_ReplaceBandNoDataValue";
+
+/// `RS_ReplaceBandNoDataValue()` scalar UDF.
+pub fn rs_replace_band_nodata_value_udf() -> SedonaScalarUDF {
+ SedonaScalarUDF::new(
+ "rs_replacebandnodatavalue",
+ vec![Arc::new(RsReplaceBandNoDataValue)],
+ Volatility::Immutable,
+ )
+ // The kernel reads and rewrites pixel bytes, so the raster argument must be
+ // materialised InDb first; the planner injects RS_EnsureLoaded on this flag.
+ .with_metadata(NEEDS_PIXELS_METADATA_KEY, "true")
+ // The output is InDb too (the addressed band is fresh bytes, the others are
+ // copied from the loaded input), so a consumer must not wrap it in another
+ // RS_EnsureLoaded.
+ .with_metadata(RETURNS_BYTES_METADATA_KEY, "true")
+}
+
+#[derive(Debug)]
+struct RsReplaceBandNoDataValue;
+
+impl SedonaScalarKernel for RsReplaceBandNoDataValue {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ ArgMatcher::new(
+ vec![
+ ArgMatcher::is_raster(),
+ ArgMatcher::is_integer(),
+ ArgMatcher::is_numeric(),
+ ],
+ SedonaType::Raster,
+ )
+ .match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = RasterExecutor::new(arg_types, args);
+ let n = executor.num_iterations();
+
+ let band_array = args[1]
+ .clone()
+ .cast_to(&DataType::Int64, None)?
+ .into_array(n)?;
+ let band = band_array.as_primitive::<Int64Type>();
+ let nodata_array = args[2]
+ .clone()
+ .cast_to(&DataType::Float64, None)?
+ .into_array(n)?;
+ let nodata = nodata_array.as_primitive::<Float64Type>();
+
+ let mut builder = RasterBuilder::new(n);
+ executor.execute_raster_void(|i, raster_opt| {
+ let Some(raster) = raster_opt else {
+ return Ok(builder.append_null()?);
+ };
+ if band.is_null(i) || nodata.is_null(i) {
+ return Ok(builder.append_null()?);
+ }
+ // Clamp a negative band to 0 so resolve_band rejects it as not
+ // 1-based rather than wrapping it into a huge usize.
+ let band_num = band.value(i).max(0) as usize;
+ replace_band_nodata(&mut builder, raster, band_num, nodata.value(i))
+ })?;
+
+ executor.finish(Arc::new(builder.finish()?))
+ }
+}
+
+/// Copy `raster` into `builder` with band `band_num` (1-based) moved to the
+/// nodata value `nodata`: its nodata pixels are rewritten to the new value
+/// first, so they keep reading as nodata.
+fn replace_band_nodata(
+ builder: &mut RasterBuilder,
+ raster: &dyn RasterRef,
+ band_num: usize,
+ nodata: f64,
+) -> Result<()> {
+ let target = resolve_band(FUNC, raster, band_num)?;
+ let new_nodata = nodata_f64_to_bytes(nodata, &target.data_type())
+ .map_err(|e| exec_datafusion_err!("{FUNC}: {e}"))?;
+ let data = replace_nodata_pixels(target.as_ref(), &new_nodata)?;
+
+ builder.start_raster_from(raster, RasterOverrides::default())?;
+ for band_idx in 0..raster.num_bands() {
+ let band = raster.band(band_idx)?;
+ let overrides = if band_idx + 1 == band_num {
+ // The new bytes are the band's visible pixels, packed row-major, so
+ // they carry an identity view over the visible shape.
+ BandOverrides {
+ data: Override::Set(&data),
+ view: Override::Clear,
+ source_shape: Some(band.shape()),
+ nodata: Override::Set(&new_nodata),
+ ..Default::default()
+ }
+ } else {
+ BandOverrides::default()
+ };
+ band.copy_into(builder, overrides)?;
+ builder.finish_band()?;
+ }
+ builder.finish_raster()?;
+ Ok(())
+}
+
+/// The band's visible pixels, packed row-major, with every pixel that reads as
+/// nodata rewritten to `new_nodata`.
+fn replace_nodata_pixels(band: &dyn BandRef, new_nodata: &[u8]) -> Result<Buffer> {
+ let Some(matcher) = NodataMatcher::for_band(FUNC, band)? else {
+ return exec_err!(
+ "{FUNC}: the band has no nodata value to replace; use RS_SetBandNoDataValue to set one"
+ );
+ };
+ let buffer = spatial_2d_buffer(FUNC, band)?;
+ let (height, width) = (buffer.shape[0], buffer.shape[1]);
+
+ // A broadcast view can describe far more pixels than its source holds, so
+ // size the packed output with checked arithmetic and reject it before
+ // allocating rather than after.
+ let len = usize::try_from(width)
+ .ok()
+ .and_then(|w| w.checked_mul(usize::try_from(height).ok()?))
+ .and_then(|n| n.checked_mul(new_nodata.len()))
+ .ok_or_else(|| {
+ exec_datafusion_err!("{FUNC}: a {width} x {height} band is too large to materialise")
+ })?;
+ check_band_data_len(len).map_err(|e| exec_datafusion_err!("{FUNC}: {e}"))?;
+
+ let mut data = Vec::with_capacity(len);
+ scan_pixels(FUNC, &buffer, |_, _, pixel| {
Review Comment:
I think thats a reasonable optimization. probably the scan is cheaper than the copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-raster-functions): add RS_ReplaceBandNoDataValue [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #1343:
URL: https://github.com/apache/sedona-db/pull/1343#discussion_r4149696146
##########
integration/spark-parity/test_rs_setbandnodatavalue.py:
##########
@@ -222,9 +222,10 @@ def test_rs_setbandnodata_null_band(tmp_path):
@pytest.mark.xfail(
- reason="SedonaDB reads a NULL nodata as 'clear this band's nodata' (the "
- "trinary override semantics from #1198) and returns the raster; Sedona "
- "Spark propagates the NULL and returns a NULL raster"
+ reason="needs a Sedona release carrying apache/sedona#3312 — the released "
+ "1.9.1 jar propagates the NULL and returns a NULL raster where SedonaDB "
+ "clears the band's nodata (the trinary override semantics from #1198); "
+ "Sedona master now clears it too, verified against a jar built from it"
Review Comment:
For what it's worth, I missed these in review and I think that messing with null propagation isn't worth it. Using another function name (`RS_UnSetCrs(...)`) or a sentinel (`RS_SetCrs(..., zap())`) is cleaner.
##########
rust/sedona-raster-functions/src/rs_replace_band_nodata.rs:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `RS_ReplaceBandNoDataValue` — move a band's nodata to a new value, carrying
+//! the nodata pixels with it.
+//!
+//! ```text
+//! RS_ReplaceBandNoDataValue(raster, band, nodata) -> Raster
+//! ```
+//!
+//! Every pixel of the addressed band that reads as nodata today is rewritten to
+//! `nodata`, which then becomes the band's nodata value, so the same pixels
+//! read as nodata before and after the call. `RS_SetBandNoDataValue` only
+//! changes the declared value and leaves the pixels alone.
+//!
+//! A pixel reads as nodata by the same rules sampling uses
+//! ([`NodataMatcher`]): exact for integer bands, numeric for float bands, so
+//! `-0.0` matches a `0.0` nodata and any NaN matches a NaN nodata.
+//!
+//! The band must already have a nodata value; with none there is nothing to
+//! replace, so that is an error. A null raster, band or `nodata` yields a null
+//! raster. Other bands are carried over untouched and zero-copy. Only 2-D
+//! bands are supported.
+//!
+//! There is deliberately no form without a band: Sedona Spark's 2-argument
+//! forms default to band 1 where SedonaDB's reject a multiband raster, and a
+//! new function need not inherit that divergence.
+
+use std::ops::ControlFlow;
+use std::sync::Arc;
+
+use arrow_array::Array;
+use arrow_array::cast::AsArray;
+use arrow_array::types::{Float64Type, Int64Type};
+use arrow_buffer::Buffer;
+use arrow_schema::DataType;
+use datafusion_common::{Result, exec_datafusion_err, exec_err};
+use datafusion_expr::{ColumnarValue, Volatility};
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_raster::band_builder::check_band_data_len;
+use sedona_raster::builder::{RasterBuilder, RasterOverrides};
+use sedona_raster::traits::{BandOverrides, BandRef, Override, RasterRef, nodata_f64_to_bytes};
+use sedona_schema::datatypes::SedonaType;
+use sedona_schema::matchers::ArgMatcher;
+
+use crate::executor::RasterExecutor;
+use crate::pixel_scan::{NodataMatcher, scan_pixels, spatial_2d_buffer};
+use crate::rs_ensure_loaded::{NEEDS_PIXELS_METADATA_KEY, RETURNS_BYTES_METADATA_KEY};
+use crate::sampling::resolve_band;
+
+const FUNC: &str = "RS_ReplaceBandNoDataValue";
+
+/// `RS_ReplaceBandNoDataValue()` scalar UDF.
+pub fn rs_replace_band_nodata_value_udf() -> SedonaScalarUDF {
+ SedonaScalarUDF::new(
+ "rs_replacebandnodatavalue",
+ vec![Arc::new(RsReplaceBandNoDataValue)],
+ Volatility::Immutable,
+ )
+ // The kernel reads and rewrites pixel bytes, so the raster argument must be
+ // materialised InDb first; the planner injects RS_EnsureLoaded on this flag.
+ .with_metadata(NEEDS_PIXELS_METADATA_KEY, "true")
+ // The output is InDb too (the addressed band is fresh bytes, the others are
+ // copied from the loaded input), so a consumer must not wrap it in another
+ // RS_EnsureLoaded.
+ .with_metadata(RETURNS_BYTES_METADATA_KEY, "true")
+}
+
+#[derive(Debug)]
+struct RsReplaceBandNoDataValue;
+
+impl SedonaScalarKernel for RsReplaceBandNoDataValue {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ ArgMatcher::new(
+ vec![
+ ArgMatcher::is_raster(),
+ ArgMatcher::is_integer(),
+ ArgMatcher::is_numeric(),
+ ],
+ SedonaType::Raster,
+ )
+ .match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = RasterExecutor::new(arg_types, args);
+ let n = executor.num_iterations();
+
+ let band_array = args[1]
+ .clone()
+ .cast_to(&DataType::Int64, None)?
+ .into_array(n)?;
+ let band = band_array.as_primitive::<Int64Type>();
+ let nodata_array = args[2]
+ .clone()
+ .cast_to(&DataType::Float64, None)?
+ .into_array(n)?;
+ let nodata = nodata_array.as_primitive::<Float64Type>();
+
+ let mut builder = RasterBuilder::new(n);
+ executor.execute_raster_void(|i, raster_opt| {
+ let Some(raster) = raster_opt else {
+ return Ok(builder.append_null()?);
+ };
+ if band.is_null(i) || nodata.is_null(i) {
+ return Ok(builder.append_null()?);
+ }
+ // Clamp a negative band to 0 so resolve_band rejects it as not
+ // 1-based rather than wrapping it into a huge usize.
+ let band_num = band.value(i).max(0) as usize;
+ replace_band_nodata(&mut builder, raster, band_num, nodata.value(i))
+ })?;
+
+ executor.finish(Arc::new(builder.finish()?))
+ }
+}
+
+/// Copy `raster` into `builder` with band `band_num` (1-based) moved to the
+/// nodata value `nodata`: its nodata pixels are rewritten to the new value
+/// first, so they keep reading as nodata.
+fn replace_band_nodata(
+ builder: &mut RasterBuilder,
+ raster: &dyn RasterRef,
+ band_num: usize,
+ nodata: f64,
+) -> Result<()> {
+ let target = resolve_band(FUNC, raster, band_num)?;
+ let new_nodata = nodata_f64_to_bytes(nodata, &target.data_type())
+ .map_err(|e| exec_datafusion_err!("{FUNC}: {e}"))?;
+ let data = replace_nodata_pixels(target.as_ref(), &new_nodata)?;
+
+ builder.start_raster_from(raster, RasterOverrides::default())?;
+ for band_idx in 0..raster.num_bands() {
+ let band = raster.band(band_idx)?;
+ let overrides = if band_idx + 1 == band_num {
+ // The new bytes are the band's visible pixels, packed row-major, so
+ // they carry an identity view over the visible shape.
+ BandOverrides {
+ data: Override::Set(&data),
+ view: Override::Clear,
+ source_shape: Some(band.shape()),
+ nodata: Override::Set(&new_nodata),
+ ..Default::default()
+ }
+ } else {
+ BandOverrides::default()
+ };
+ band.copy_into(builder, overrides)?;
+ builder.finish_band()?;
+ }
+ builder.finish_raster()?;
+ Ok(())
+}
+
+/// The band's visible pixels, packed row-major, with every pixel that reads as
+/// nodata rewritten to `new_nodata`.
+fn replace_nodata_pixels(band: &dyn BandRef, new_nodata: &[u8]) -> Result<Buffer> {
+ let Some(matcher) = NodataMatcher::for_band(FUNC, band)? else {
+ return exec_err!(
+ "{FUNC}: the band has no nodata value to replace; use RS_SetBandNoDataValue to set one"
+ );
+ };
+ let buffer = spatial_2d_buffer(FUNC, band)?;
+ let (height, width) = (buffer.shape[0], buffer.shape[1]);
+
+ // A broadcast view can describe far more pixels than its source holds, so
+ // size the packed output with checked arithmetic and reject it before
+ // allocating rather than after.
+ let len = usize::try_from(width)
+ .ok()
+ .and_then(|w| w.checked_mul(usize::try_from(height).ok()?))
+ .and_then(|n| n.checked_mul(new_nodata.len()))
+ .ok_or_else(|| {
+ exec_datafusion_err!("{FUNC}: a {width} x {height} band is too large to materialise")
+ })?;
+ check_band_data_len(len).map_err(|e| exec_datafusion_err!("{FUNC}: {e}"))?;
+
+ let mut data = Vec::with_capacity(len);
+ scan_pixels(FUNC, &buffer, |_, _, pixel| {
+ data.extend_from_slice(if matcher.matches(pixel) {
+ new_nodata
+ } else {
+ pixel
+ });
+ ControlFlow::Continue(())
+ })?;
+ Ok(Buffer::from_vec(data))
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use arrow_array::{ArrayRef, Float64Array, Int64Array};
+ use datafusion_expr::ScalarUDF;
+ use sedona_schema::datatypes::RASTER;
+ use sedona_testing::raster_spec::{RasterSpec, assert_rasters_equal, raster_array};
+ use sedona_testing::testers::ScalarUdfTester;
+
+ fn i64_t() -> SedonaType {
+ SedonaType::Arrow(DataType::Int64)
+ }
+ fn f64_t() -> SedonaType {
+ SedonaType::Arrow(DataType::Float64)
+ }
Review Comment:
Can you inline these?
##########
rust/sedona-raster-functions/src/rs_replace_band_nodata.rs:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `RS_ReplaceBandNoDataValue` — move a band's nodata to a new value, carrying
+//! the nodata pixels with it.
+//!
+//! ```text
+//! RS_ReplaceBandNoDataValue(raster, band, nodata) -> Raster
+//! ```
+//!
+//! Every pixel of the addressed band that reads as nodata today is rewritten to
+//! `nodata`, which then becomes the band's nodata value, so the same pixels
+//! read as nodata before and after the call. `RS_SetBandNoDataValue` only
+//! changes the declared value and leaves the pixels alone.
+//!
+//! A pixel reads as nodata by the same rules sampling uses
+//! ([`NodataMatcher`]): exact for integer bands, numeric for float bands, so
+//! `-0.0` matches a `0.0` nodata and any NaN matches a NaN nodata.
+//!
+//! The band must already have a nodata value; with none there is nothing to
+//! replace, so that is an error. A null raster, band or `nodata` yields a null
+//! raster. Other bands are carried over untouched and zero-copy. Only 2-D
+//! bands are supported.
+//!
+//! There is deliberately no form without a band: Sedona Spark's 2-argument
+//! forms default to band 1 where SedonaDB's reject a multiband raster, and a
+//! new function need not inherit that divergence.
+
+use std::ops::ControlFlow;
+use std::sync::Arc;
+
+use arrow_array::Array;
+use arrow_array::cast::AsArray;
+use arrow_array::types::{Float64Type, Int64Type};
+use arrow_buffer::Buffer;
+use arrow_schema::DataType;
+use datafusion_common::{Result, exec_datafusion_err, exec_err};
+use datafusion_expr::{ColumnarValue, Volatility};
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_raster::band_builder::check_band_data_len;
+use sedona_raster::builder::{RasterBuilder, RasterOverrides};
+use sedona_raster::traits::{BandOverrides, BandRef, Override, RasterRef, nodata_f64_to_bytes};
+use sedona_schema::datatypes::SedonaType;
+use sedona_schema::matchers::ArgMatcher;
+
+use crate::executor::RasterExecutor;
+use crate::pixel_scan::{NodataMatcher, scan_pixels, spatial_2d_buffer};
+use crate::rs_ensure_loaded::{NEEDS_PIXELS_METADATA_KEY, RETURNS_BYTES_METADATA_KEY};
+use crate::sampling::resolve_band;
+
+const FUNC: &str = "RS_ReplaceBandNoDataValue";
+
+/// `RS_ReplaceBandNoDataValue()` scalar UDF.
+pub fn rs_replace_band_nodata_value_udf() -> SedonaScalarUDF {
+ SedonaScalarUDF::new(
+ "rs_replacebandnodatavalue",
+ vec![Arc::new(RsReplaceBandNoDataValue)],
+ Volatility::Immutable,
+ )
+ // The kernel reads and rewrites pixel bytes, so the raster argument must be
+ // materialised InDb first; the planner injects RS_EnsureLoaded on this flag.
+ .with_metadata(NEEDS_PIXELS_METADATA_KEY, "true")
+ // The output is InDb too (the addressed band is fresh bytes, the others are
+ // copied from the loaded input), so a consumer must not wrap it in another
+ // RS_EnsureLoaded.
+ .with_metadata(RETURNS_BYTES_METADATA_KEY, "true")
+}
+
+#[derive(Debug)]
+struct RsReplaceBandNoDataValue;
+
+impl SedonaScalarKernel for RsReplaceBandNoDataValue {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ ArgMatcher::new(
+ vec![
+ ArgMatcher::is_raster(),
+ ArgMatcher::is_integer(),
+ ArgMatcher::is_numeric(),
+ ],
+ SedonaType::Raster,
+ )
+ .match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = RasterExecutor::new(arg_types, args);
+ let n = executor.num_iterations();
+
+ let band_array = args[1]
+ .clone()
+ .cast_to(&DataType::Int64, None)?
+ .into_array(n)?;
+ let band = band_array.as_primitive::<Int64Type>();
+ let nodata_array = args[2]
+ .clone()
+ .cast_to(&DataType::Float64, None)?
+ .into_array(n)?;
+ let nodata = nodata_array.as_primitive::<Float64Type>();
+
+ let mut builder = RasterBuilder::new(n);
+ executor.execute_raster_void(|i, raster_opt| {
+ let Some(raster) = raster_opt else {
+ return Ok(builder.append_null()?);
+ };
+ if band.is_null(i) || nodata.is_null(i) {
+ return Ok(builder.append_null()?);
+ }
+ // Clamp a negative band to 0 so resolve_band rejects it as not
+ // 1-based rather than wrapping it into a huge usize.
+ let band_num = band.value(i).max(0) as usize;
+ replace_band_nodata(&mut builder, raster, band_num, nodata.value(i))
+ })?;
+
+ executor.finish(Arc::new(builder.finish()?))
+ }
+}
+
+/// Copy `raster` into `builder` with band `band_num` (1-based) moved to the
+/// nodata value `nodata`: its nodata pixels are rewritten to the new value
+/// first, so they keep reading as nodata.
+fn replace_band_nodata(
+ builder: &mut RasterBuilder,
+ raster: &dyn RasterRef,
+ band_num: usize,
+ nodata: f64,
+) -> Result<()> {
+ let target = resolve_band(FUNC, raster, band_num)?;
+ let new_nodata = nodata_f64_to_bytes(nodata, &target.data_type())
+ .map_err(|e| exec_datafusion_err!("{FUNC}: {e}"))?;
+ let data = replace_nodata_pixels(target.as_ref(), &new_nodata)?;
+
+ builder.start_raster_from(raster, RasterOverrides::default())?;
+ for band_idx in 0..raster.num_bands() {
+ let band = raster.band(band_idx)?;
+ let overrides = if band_idx + 1 == band_num {
+ // The new bytes are the band's visible pixels, packed row-major, so
+ // they carry an identity view over the visible shape.
+ BandOverrides {
+ data: Override::Set(&data),
+ view: Override::Clear,
+ source_shape: Some(band.shape()),
+ nodata: Override::Set(&new_nodata),
+ ..Default::default()
+ }
+ } else {
+ BandOverrides::default()
+ };
+ band.copy_into(builder, overrides)?;
+ builder.finish_band()?;
+ }
+ builder.finish_raster()?;
+ Ok(())
+}
+
+/// The band's visible pixels, packed row-major, with every pixel that reads as
+/// nodata rewritten to `new_nodata`.
+fn replace_nodata_pixels(band: &dyn BandRef, new_nodata: &[u8]) -> Result<Buffer> {
+ let Some(matcher) = NodataMatcher::for_band(FUNC, band)? else {
+ return exec_err!(
+ "{FUNC}: the band has no nodata value to replace; use RS_SetBandNoDataValue to set one"
+ );
+ };
+ let buffer = spatial_2d_buffer(FUNC, band)?;
+ let (height, width) = (buffer.shape[0], buffer.shape[1]);
+
+ // A broadcast view can describe far more pixels than its source holds, so
+ // size the packed output with checked arithmetic and reject it before
+ // allocating rather than after.
+ let len = usize::try_from(width)
+ .ok()
+ .and_then(|w| w.checked_mul(usize::try_from(height).ok()?))
+ .and_then(|n| n.checked_mul(new_nodata.len()))
+ .ok_or_else(|| {
+ exec_datafusion_err!("{FUNC}: a {width} x {height} band is too large to materialise")
+ })?;
+ check_band_data_len(len).map_err(|e| exec_datafusion_err!("{FUNC}: {e}"))?;
+
+ let mut data = Vec::with_capacity(len);
+ scan_pixels(FUNC, &buffer, |_, _, pixel| {
Review Comment:
Would it make sense or be a useful optimization to scan the pixels and look for a nodata value and return early for the case that the raster does not contain any nodata values? That would avoid a bunch of copying although also might not be faster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-raster-functions): add RS_ReplaceBandNoDataValue [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis merged PR #1343:
URL: https://github.com/apache/sedona-db/pull/1343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-raster-functions): RS_SetBandNoDataValue replace overload [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #1343:
URL: https://github.com/apache/sedona-db/pull/1343#discussion_r4085478701
##########
rust/sedona-raster-functions/src/rs_set_band_nodata.rs:
##########
@@ -50,28 +64,46 @@ use datafusion_common::error::Result;
use datafusion_common::exec_err;
use datafusion_expr::{ColumnarValue, Volatility};
use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
-use sedona_raster::builder::{RasterBuilder, RasterOverrides};
-use sedona_raster::traits::{BandOverrides, Override, RasterRef, nodata_f64_to_bytes};
+use sedona_raster::builder::{RasterBuilder, RasterOverrides, StartBandArgs};
+use sedona_raster::traits::{BandOverrides, BandRef, Override, RasterRef, nodata_f64_to_bytes};
use sedona_schema::datatypes::SedonaType;
use sedona_schema::matchers::ArgMatcher;
use crate::executor::RasterExecutor;
+use crate::rs_ensure_loaded::NEEDS_PIXELS_METADATA_KEY;
/// RS_SetBandNoDataValue() scalar UDF implementation
pub fn rs_set_band_nodata_value_udf() -> SedonaScalarUDF {
SedonaScalarUDF::new(
"rs_setbandnodatavalue",
vec![
- Arc::new(RsSetBandNoDataValue { with_band: false }),
- Arc::new(RsSetBandNoDataValue { with_band: true }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: false,
+ with_replace: false,
+ }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: true,
+ with_replace: false,
+ }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: true,
+ with_replace: true,
+ }),
],
Volatility::Immutable,
)
+ // The replace kernel rewrites pixel bytes, so the raster argument must be
+ // materialised InDb first; the planner injects RS_EnsureLoaded on this flag.
+ .with_metadata(NEEDS_PIXELS_METADATA_KEY, "true")
Review Comment:
yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-raster-functions): RS_SetBandNoDataValue replace overload [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #1343:
URL: https://github.com/apache/sedona-db/pull/1343#discussion_r4107311273
##########
rust/sedona-raster-functions/src/rs_set_band_nodata.rs:
##########
@@ -50,28 +64,46 @@ use datafusion_common::error::Result;
use datafusion_common::exec_err;
use datafusion_expr::{ColumnarValue, Volatility};
use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
-use sedona_raster::builder::{RasterBuilder, RasterOverrides};
-use sedona_raster::traits::{BandOverrides, Override, RasterRef, nodata_f64_to_bytes};
+use sedona_raster::builder::{RasterBuilder, RasterOverrides, StartBandArgs};
+use sedona_raster::traits::{BandOverrides, BandRef, Override, RasterRef, nodata_f64_to_bytes};
use sedona_schema::datatypes::SedonaType;
use sedona_schema::matchers::ArgMatcher;
use crate::executor::RasterExecutor;
+use crate::rs_ensure_loaded::NEEDS_PIXELS_METADATA_KEY;
/// RS_SetBandNoDataValue() scalar UDF implementation
pub fn rs_set_band_nodata_value_udf() -> SedonaScalarUDF {
SedonaScalarUDF::new(
"rs_setbandnodatavalue",
vec![
- Arc::new(RsSetBandNoDataValue { with_band: false }),
- Arc::new(RsSetBandNoDataValue { with_band: true }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: false,
+ with_replace: false,
+ }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: true,
+ with_replace: false,
+ }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: true,
+ with_replace: true,
+ }),
],
Volatility::Immutable,
)
+ // The replace kernel rewrites pixel bytes, so the raster argument must be
+ // materialised InDb first; the planner injects RS_EnsureLoaded on this flag.
+ .with_metadata(NEEDS_PIXELS_METADATA_KEY, "true")
Review Comment:
The issue here would be an async UDF bug. probably getting constant folded and that is some issues with asyc UDF support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-raster-functions): add RS_ReplaceBandNoDataValue [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #1343:
URL: https://github.com/apache/sedona-db/pull/1343#discussion_r4151656590
##########
rust/sedona-raster-functions/src/rs_replace_band_nodata.rs:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `RS_ReplaceBandNoDataValue` — move a band's nodata to a new value, carrying
+//! the nodata pixels with it.
+//!
+//! ```text
+//! RS_ReplaceBandNoDataValue(raster, band, nodata) -> Raster
+//! ```
+//!
+//! Every pixel of the addressed band that reads as nodata today is rewritten to
+//! `nodata`, which then becomes the band's nodata value, so the same pixels
+//! read as nodata before and after the call. `RS_SetBandNoDataValue` only
+//! changes the declared value and leaves the pixels alone.
+//!
+//! A pixel reads as nodata by the same rules sampling uses
+//! ([`NodataMatcher`]): exact for integer bands, numeric for float bands, so
+//! `-0.0` matches a `0.0` nodata and any NaN matches a NaN nodata.
+//!
+//! The band must already have a nodata value; with none there is nothing to
+//! replace, so that is an error. A null raster, band or `nodata` yields a null
+//! raster. Other bands are carried over untouched and zero-copy. Only 2-D
+//! bands are supported.
+//!
+//! There is deliberately no form without a band: Sedona Spark's 2-argument
+//! forms default to band 1 where SedonaDB's reject a multiband raster, and a
+//! new function need not inherit that divergence.
+
+use std::ops::ControlFlow;
+use std::sync::Arc;
+
+use arrow_array::Array;
+use arrow_array::cast::AsArray;
+use arrow_array::types::{Float64Type, Int64Type};
+use arrow_buffer::Buffer;
+use arrow_schema::DataType;
+use datafusion_common::{Result, exec_datafusion_err, exec_err};
+use datafusion_expr::{ColumnarValue, Volatility};
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_raster::band_builder::check_band_data_len;
+use sedona_raster::builder::{RasterBuilder, RasterOverrides};
+use sedona_raster::traits::{BandOverrides, BandRef, Override, RasterRef, nodata_f64_to_bytes};
+use sedona_schema::datatypes::SedonaType;
+use sedona_schema::matchers::ArgMatcher;
+
+use crate::executor::RasterExecutor;
+use crate::pixel_scan::{NodataMatcher, scan_pixels, spatial_2d_buffer};
+use crate::rs_ensure_loaded::{NEEDS_PIXELS_METADATA_KEY, RETURNS_BYTES_METADATA_KEY};
+use crate::sampling::resolve_band;
+
+const FUNC: &str = "RS_ReplaceBandNoDataValue";
+
+/// `RS_ReplaceBandNoDataValue()` scalar UDF.
+pub fn rs_replace_band_nodata_value_udf() -> SedonaScalarUDF {
+ SedonaScalarUDF::new(
+ "rs_replacebandnodatavalue",
+ vec![Arc::new(RsReplaceBandNoDataValue)],
+ Volatility::Immutable,
+ )
+ // The kernel reads and rewrites pixel bytes, so the raster argument must be
+ // materialised InDb first; the planner injects RS_EnsureLoaded on this flag.
+ .with_metadata(NEEDS_PIXELS_METADATA_KEY, "true")
+ // The output is InDb too (the addressed band is fresh bytes, the others are
+ // copied from the loaded input), so a consumer must not wrap it in another
+ // RS_EnsureLoaded.
+ .with_metadata(RETURNS_BYTES_METADATA_KEY, "true")
+}
+
+#[derive(Debug)]
+struct RsReplaceBandNoDataValue;
+
+impl SedonaScalarKernel for RsReplaceBandNoDataValue {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ ArgMatcher::new(
+ vec![
+ ArgMatcher::is_raster(),
+ ArgMatcher::is_integer(),
+ ArgMatcher::is_numeric(),
+ ],
+ SedonaType::Raster,
+ )
+ .match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = RasterExecutor::new(arg_types, args);
+ let n = executor.num_iterations();
+
+ let band_array = args[1]
+ .clone()
+ .cast_to(&DataType::Int64, None)?
+ .into_array(n)?;
+ let band = band_array.as_primitive::<Int64Type>();
+ let nodata_array = args[2]
+ .clone()
+ .cast_to(&DataType::Float64, None)?
+ .into_array(n)?;
+ let nodata = nodata_array.as_primitive::<Float64Type>();
+
+ let mut builder = RasterBuilder::new(n);
+ executor.execute_raster_void(|i, raster_opt| {
+ let Some(raster) = raster_opt else {
+ return Ok(builder.append_null()?);
+ };
+ if band.is_null(i) || nodata.is_null(i) {
+ return Ok(builder.append_null()?);
+ }
+ // Clamp a negative band to 0 so resolve_band rejects it as not
+ // 1-based rather than wrapping it into a huge usize.
+ let band_num = band.value(i).max(0) as usize;
+ replace_band_nodata(&mut builder, raster, band_num, nodata.value(i))
+ })?;
+
+ executor.finish(Arc::new(builder.finish()?))
+ }
+}
+
+/// Copy `raster` into `builder` with band `band_num` (1-based) moved to the
+/// nodata value `nodata`: its nodata pixels are rewritten to the new value
+/// first, so they keep reading as nodata.
+fn replace_band_nodata(
+ builder: &mut RasterBuilder,
+ raster: &dyn RasterRef,
+ band_num: usize,
+ nodata: f64,
+) -> Result<()> {
+ let target = resolve_band(FUNC, raster, band_num)?;
+ let new_nodata = nodata_f64_to_bytes(nodata, &target.data_type())
+ .map_err(|e| exec_datafusion_err!("{FUNC}: {e}"))?;
+ let data = replace_nodata_pixels(target.as_ref(), &new_nodata)?;
+
+ builder.start_raster_from(raster, RasterOverrides::default())?;
+ for band_idx in 0..raster.num_bands() {
+ let band = raster.band(band_idx)?;
+ let overrides = if band_idx + 1 == band_num {
+ // The new bytes are the band's visible pixels, packed row-major, so
+ // they carry an identity view over the visible shape.
+ BandOverrides {
+ data: Override::Set(&data),
+ view: Override::Clear,
+ source_shape: Some(band.shape()),
+ nodata: Override::Set(&new_nodata),
+ ..Default::default()
+ }
+ } else {
+ BandOverrides::default()
+ };
+ band.copy_into(builder, overrides)?;
+ builder.finish_band()?;
+ }
+ builder.finish_raster()?;
+ Ok(())
+}
+
+/// The band's visible pixels, packed row-major, with every pixel that reads as
+/// nodata rewritten to `new_nodata`.
+fn replace_nodata_pixels(band: &dyn BandRef, new_nodata: &[u8]) -> Result<Buffer> {
+ let Some(matcher) = NodataMatcher::for_band(FUNC, band)? else {
+ return exec_err!(
+ "{FUNC}: the band has no nodata value to replace; use RS_SetBandNoDataValue to set one"
+ );
+ };
+ let buffer = spatial_2d_buffer(FUNC, band)?;
+ let (height, width) = (buffer.shape[0], buffer.shape[1]);
+
+ // A broadcast view can describe far more pixels than its source holds, so
+ // size the packed output with checked arithmetic and reject it before
+ // allocating rather than after.
+ let len = usize::try_from(width)
+ .ok()
+ .and_then(|w| w.checked_mul(usize::try_from(height).ok()?))
+ .and_then(|n| n.checked_mul(new_nodata.len()))
+ .ok_or_else(|| {
+ exec_datafusion_err!("{FUNC}: a {width} x {height} band is too large to materialise")
+ })?;
+ check_band_data_len(len).map_err(|e| exec_datafusion_err!("{FUNC}: {e}"))?;
+
+ let mut data = Vec::with_capacity(len);
+ scan_pixels(FUNC, &buffer, |_, _, pixel| {
+ data.extend_from_slice(if matcher.matches(pixel) {
+ new_nodata
+ } else {
+ pixel
+ });
+ ControlFlow::Continue(())
+ })?;
+ Ok(Buffer::from_vec(data))
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use arrow_array::{ArrayRef, Float64Array, Int64Array};
+ use datafusion_expr::ScalarUDF;
+ use sedona_schema::datatypes::RASTER;
+ use sedona_testing::raster_spec::{RasterSpec, assert_rasters_equal, raster_array};
+ use sedona_testing::testers::ScalarUdfTester;
+
+ fn i64_t() -> SedonaType {
+ SedonaType::Arrow(DataType::Int64)
+ }
+ fn f64_t() -> SedonaType {
+ SedonaType::Arrow(DataType::Float64)
+ }
Review Comment:
need to get the agent to behave better here. probably will make a UDF skill based on my DEWEY.md file 😆
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-raster-functions): RS_SetBandNoDataValue replace overload [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #1343:
URL: https://github.com/apache/sedona-db/pull/1343#issuecomment-5799887023
@jiayuasu what do you think. I prefer Dewey's proposal to just porting the spark behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-raster-functions): add RS_ReplaceBandNoDataValue [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #1343:
URL: https://github.com/apache/sedona-db/pull/1343#discussion_r4158464116
##########
integration/spark-parity/test_rs_setbandnodatavalue.py:
##########
@@ -222,9 +222,10 @@ def test_rs_setbandnodata_null_band(tmp_path):
@pytest.mark.xfail(
- reason="SedonaDB reads a NULL nodata as 'clear this band's nodata' (the "
- "trinary override semantics from #1198) and returns the raster; Sedona "
- "Spark propagates the NULL and returns a NULL raster"
+ reason="needs a Sedona release carrying apache/sedona#3312 — the released "
+ "1.9.1 jar propagates the NULL and returns a NULL raster where SedonaDB "
+ "clears the band's nodata (the trinary override semantics from #1198); "
+ "Sedona master now clears it too, verified against a jar built from it"
Review Comment:
your comment is about CRSes but I assume you mean the same thing when talking about NoDataValue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-raster-functions): RS_SetBandNoDataValue replace overload [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #1343:
URL: https://github.com/apache/sedona-db/pull/1343#discussion_r4107311273
##########
rust/sedona-raster-functions/src/rs_set_band_nodata.rs:
##########
@@ -50,28 +64,46 @@ use datafusion_common::error::Result;
use datafusion_common::exec_err;
use datafusion_expr::{ColumnarValue, Volatility};
use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
-use sedona_raster::builder::{RasterBuilder, RasterOverrides};
-use sedona_raster::traits::{BandOverrides, Override, RasterRef, nodata_f64_to_bytes};
+use sedona_raster::builder::{RasterBuilder, RasterOverrides, StartBandArgs};
+use sedona_raster::traits::{BandOverrides, BandRef, Override, RasterRef, nodata_f64_to_bytes};
use sedona_schema::datatypes::SedonaType;
use sedona_schema::matchers::ArgMatcher;
use crate::executor::RasterExecutor;
+use crate::rs_ensure_loaded::NEEDS_PIXELS_METADATA_KEY;
/// RS_SetBandNoDataValue() scalar UDF implementation
pub fn rs_set_band_nodata_value_udf() -> SedonaScalarUDF {
SedonaScalarUDF::new(
"rs_setbandnodatavalue",
vec![
- Arc::new(RsSetBandNoDataValue { with_band: false }),
- Arc::new(RsSetBandNoDataValue { with_band: true }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: false,
+ with_replace: false,
+ }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: true,
+ with_replace: false,
+ }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: true,
+ with_replace: true,
+ }),
],
Volatility::Immutable,
)
+ // The replace kernel rewrites pixel bytes, so the raster argument must be
+ // materialised InDb first; the planner injects RS_EnsureLoaded on this flag.
+ .with_metadata(NEEDS_PIXELS_METADATA_KEY, "true")
Review Comment:
The issue here would be an async UDF bug. probably getting constant folded and that is some issues with asyc UDF support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-raster-functions): RS_SetBandNoDataValue replace overload [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #1343:
URL: https://github.com/apache/sedona-db/pull/1343#issuecomment-5828495571
Sounds good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-raster-functions): add RS_ReplaceBandNoDataValue [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #1343:
URL: https://github.com/apache/sedona-db/pull/1343#discussion_r4147682224
##########
rust/sedona-raster-functions/src/rs_set_band_nodata.rs:
##########
@@ -217,9 +290,48 @@ fn set_band_nodata(
Ok(())
}
+/// `band`'s visible bytes with every pixel equal to its current nodata
+/// rewritten to `new_nodata`.
+///
+/// The comparison is on the raw little-endian bytes rather than on decoded
+/// values, which makes it exact for every band data type without a dispatch
+/// over them — `new_nodata` arrives already packed into this band's type by
+/// `nodata_f64_to_bytes`, so both sides are the same width by construction.
+/// The one behavior this inherits from byte equality is that a NaN sentinel
+/// matches only pixels carrying the identical NaN bit pattern; Sedona Spark
+/// compares numerically, where NaN never equals itself and so never matches.
+fn replace_nodata_pixels(band: &dyn BandRef, new_nodata: &[u8]) -> Result<Vec<u8>> {
+ let Some(old_nodata) = band.nodata() else {
+ return exec_err!(
+ "RS_SetBandNoDataValue: replace requires the band to already have a nodata value \
+ to replace, but this band has none"
+ );
+ };
+
+ let width = band.data_type().byte_size();
+ if old_nodata.len() != width || new_nodata.len() != width {
+ return exec_err!(
+ "RS_SetBandNoDataValue: nodata width mismatch for a {width}-byte band \
+ (existing {} bytes, new {} bytes)",
+ old_nodata.len(),
+ new_nodata.len()
+ );
+ }
+
+ let buffer = band.nd_buffer()?;
+ let mut pixels = buffer.as_contiguous()?.to_vec();
+ for pixel in pixels.chunks_exact_mut(width) {
+ if pixel == old_nodata {
Review Comment:
fixed in the new function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-raster-functions): add RS_ReplaceBandNoDataValue [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #1343:
URL: https://github.com/apache/sedona-db/pull/1343#discussion_r4148048761
##########
rust/sedona-raster-functions/src/rs_set_band_nodata.rs:
##########
@@ -50,28 +64,46 @@ use datafusion_common::error::Result;
use datafusion_common::exec_err;
use datafusion_expr::{ColumnarValue, Volatility};
use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
-use sedona_raster::builder::{RasterBuilder, RasterOverrides};
-use sedona_raster::traits::{BandOverrides, Override, RasterRef, nodata_f64_to_bytes};
+use sedona_raster::builder::{RasterBuilder, RasterOverrides, StartBandArgs};
+use sedona_raster::traits::{BandOverrides, BandRef, Override, RasterRef, nodata_f64_to_bytes};
use sedona_schema::datatypes::SedonaType;
use sedona_schema::matchers::ArgMatcher;
use crate::executor::RasterExecutor;
+use crate::rs_ensure_loaded::NEEDS_PIXELS_METADATA_KEY;
/// RS_SetBandNoDataValue() scalar UDF implementation
pub fn rs_set_band_nodata_value_udf() -> SedonaScalarUDF {
SedonaScalarUDF::new(
"rs_setbandnodatavalue",
vec![
- Arc::new(RsSetBandNoDataValue { with_band: false }),
- Arc::new(RsSetBandNoDataValue { with_band: true }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: false,
+ with_replace: false,
+ }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: true,
+ with_replace: false,
+ }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: true,
+ with_replace: true,
+ }),
],
Volatility::Immutable,
)
+ // The replace kernel rewrites pixel bytes, so the raster argument must be
+ // materialised InDb first; the planner injects RS_EnsureLoaded on this flag.
+ .with_metadata(NEEDS_PIXELS_METADATA_KEY, "true")
Review Comment:
moot thread since we broke this out into its own function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-raster-functions): RS_SetBandNoDataValue replace overload [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #1343:
URL: https://github.com/apache/sedona-db/pull/1343#discussion_r4102143202
##########
rust/sedona-raster-functions/src/rs_set_band_nodata.rs:
##########
@@ -50,28 +64,46 @@ use datafusion_common::error::Result;
use datafusion_common::exec_err;
use datafusion_expr::{ColumnarValue, Volatility};
use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
-use sedona_raster::builder::{RasterBuilder, RasterOverrides};
-use sedona_raster::traits::{BandOverrides, Override, RasterRef, nodata_f64_to_bytes};
+use sedona_raster::builder::{RasterBuilder, RasterOverrides, StartBandArgs};
+use sedona_raster::traits::{BandOverrides, BandRef, Override, RasterRef, nodata_f64_to_bytes};
use sedona_schema::datatypes::SedonaType;
use sedona_schema::matchers::ArgMatcher;
use crate::executor::RasterExecutor;
+use crate::rs_ensure_loaded::NEEDS_PIXELS_METADATA_KEY;
/// RS_SetBandNoDataValue() scalar UDF implementation
pub fn rs_set_band_nodata_value_udf() -> SedonaScalarUDF {
SedonaScalarUDF::new(
"rs_setbandnodatavalue",
vec![
- Arc::new(RsSetBandNoDataValue { with_band: false }),
- Arc::new(RsSetBandNoDataValue { with_band: true }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: false,
+ with_replace: false,
+ }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: true,
+ with_replace: false,
+ }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: true,
+ with_replace: true,
+ }),
],
Volatility::Immutable,
)
+ // The replace kernel rewrites pixel bytes, so the raster argument must be
+ // materialised InDb first; the planner injects RS_EnsureLoaded on this flag.
+ .with_metadata(NEEDS_PIXELS_METADATA_KEY, "true")
Review Comment:
Could we also mark the pixel-rewriting function with `returns_bytes`? I hit an error with this on the current branch:
```sql
CREATE TABLE t AS SELECT RS_Example() AS r;
SELECT RS_Value(RS_SetBandNoDataValue(r, 1, 200, true), 1, 1, 1)
FROM t;
```
I get `async functions should not be called directly` instead of `1`. The two- and three-argument forms also fail when nested this way on a raster column.
It looks like the new `needs_pixels` flag causes the planner to wrap both the input and the setter's result. Adding `returns_bytes=true` made the nested calls work in my local probe. I think the separate replacement function will need that flag too.
##########
rust/sedona-raster-functions/src/rs_set_band_nodata.rs:
##########
@@ -217,9 +290,48 @@ fn set_band_nodata(
Ok(())
}
+/// `band`'s visible bytes with every pixel equal to its current nodata
+/// rewritten to `new_nodata`.
+///
+/// The comparison is on the raw little-endian bytes rather than on decoded
+/// values, which makes it exact for every band data type without a dispatch
+/// over them — `new_nodata` arrives already packed into this band's type by
+/// `nodata_f64_to_bytes`, so both sides are the same width by construction.
+/// The one behavior this inherits from byte equality is that a NaN sentinel
+/// matches only pixels carrying the identical NaN bit pattern; Sedona Spark
+/// compares numerically, where NaN never equals itself and so never matches.
+fn replace_nodata_pixels(band: &dyn BandRef, new_nodata: &[u8]) -> Result<Vec<u8>> {
+ let Some(old_nodata) = band.nodata() else {
+ return exec_err!(
+ "RS_SetBandNoDataValue: replace requires the band to already have a nodata value \
+ to replace, but this band has none"
+ );
+ };
+
+ let width = band.data_type().byte_size();
+ if old_nodata.len() != width || new_nodata.len() != width {
+ return exec_err!(
+ "RS_SetBandNoDataValue: nodata width mismatch for a {width}-byte band \
+ (existing {} bytes, new {} bytes)",
+ old_nodata.len(),
+ new_nodata.len()
+ );
+ }
+
+ let buffer = band.nd_buffer()?;
+ let mut pixels = buffer.as_contiguous()?.to_vec();
+ for pixel in pixels.chunks_exact_mut(width) {
+ if pixel == old_nodata {
Review Comment:
Could we use the same floating-point nodata comparison as sampling here? With pixels `[-0.0, +0.0]` and nodata `+0.0`, both pixels read as NULL before replacement, but `replace=true` only rewrites `+0.0`. The `-0.0` pixel then becomes valid data.
This can be covered in the existing test module with:
```rust
#[test]
fn replace_preserves_signed_zero_nodata() {
let input = RasterSpec::d2(2, 1)
.band_values(&[-0.0f64, 0.0])
.nodata(0.0f64);
let result = invoke_4arg(input, 1, -9999.0, true).unwrap();
let expected = RasterSpec::d2(2, 1)
.band_values(&[-9999.0f64, -9999.0])
.nodata(-9999.0f64);
assert_rasters_equal(&result, &[Some(expected)]);
}
```
I reproduced the masking change through `RS_Value` for both Float32 and Float64. A NaN pixel with a different payload from the nodata sentinel also changes from NULL to an unmasked NaN. Could we use the same float matching rules as the sampling code while keeping integer comparisons exact?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-raster-functions): RS_SetBandNoDataValue replace overload [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #1343:
URL: https://github.com/apache/sedona-db/pull/1343#discussion_r4078428187
##########
rust/sedona-raster-functions/src/rs_set_band_nodata.rs:
##########
@@ -50,28 +64,46 @@ use datafusion_common::error::Result;
use datafusion_common::exec_err;
use datafusion_expr::{ColumnarValue, Volatility};
use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
-use sedona_raster::builder::{RasterBuilder, RasterOverrides};
-use sedona_raster::traits::{BandOverrides, Override, RasterRef, nodata_f64_to_bytes};
+use sedona_raster::builder::{RasterBuilder, RasterOverrides, StartBandArgs};
+use sedona_raster::traits::{BandOverrides, BandRef, Override, RasterRef, nodata_f64_to_bytes};
use sedona_schema::datatypes::SedonaType;
use sedona_schema::matchers::ArgMatcher;
use crate::executor::RasterExecutor;
+use crate::rs_ensure_loaded::NEEDS_PIXELS_METADATA_KEY;
/// RS_SetBandNoDataValue() scalar UDF implementation
pub fn rs_set_band_nodata_value_udf() -> SedonaScalarUDF {
SedonaScalarUDF::new(
"rs_setbandnodatavalue",
vec![
- Arc::new(RsSetBandNoDataValue { with_band: false }),
- Arc::new(RsSetBandNoDataValue { with_band: true }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: false,
+ with_replace: false,
+ }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: true,
+ with_replace: false,
+ }),
+ Arc::new(RsSetBandNoDataValue {
+ with_band: true,
+ with_replace: true,
+ }),
],
Volatility::Immutable,
)
+ // The replace kernel rewrites pixel bytes, so the raster argument must be
+ // materialised InDb first; the planner injects RS_EnsureLoaded on this flag.
+ .with_metadata(NEEDS_PIXELS_METADATA_KEY, "true")
Review Comment:
The non-replace kernels don't need this, though, so this will unnecessarily load pixels even for the metadata-only change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org