You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "james-willis (via GitHub)" <gi...@apache.org> on 2026/09/02 16:41:28 UTC

[PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

james-willis opened a new pull request, #3312:
URL: https://github.com/apache/sedona/pull/3312

   ## Did you read the Contributor Guide?
   
   - Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/)
   
   ## Is this PR related to a ticket?
   
   - Yes, and the PR name follows the format `[GH-XXX] my subject`. Closes #3311
   
   ## What changes were proposed in this PR?
   
   The docs and the common Java implementation say a `null` `noDataValue` removes the band's no-data value, but from Spark SQL that path was unreachable: the catalyst expression wrapped `RasterBandEditors.setBandNoDataValue` with `inferrableFunction2/3/4`, which null-propagate as soon as any argument is null, so `RS_SetBandNoDataValue(raster, 1, NULL)` returned a NULL raster instead. Fixing this matches PostGIS `ST_SetBandNoDataValue` NULL-clears semantics and the behavior SedonaDB adopted in apache/sedona-db#1158.
   
   - Added `InferrableFunction.allowRightNull3`, an arity-3 counterpart of the existing `allowRightNull` pattern (allows the third argument to be null; still null-propagates when the raster or band index is null). It is a separate name rather than an overload because overloading `allowRightNull` breaks eta-expansion of the overloaded functions passed to it at existing call sites.
   - Wired the 2- and 3-argument overloads of `RS_SetBandNoDataValue` through `allowRightNull`/`allowRightNull3`. The 4-argument `replace` overload still null-propagates, since replacing pixels with a null no-data value is meaningless; the docs now note that clearing requires the 2-/3-argument form.
   - Documented the clearing behavior with an example in `RS_SetBandNoDataValue.md`.
   
   ## How was this patch tested?
   
   Strengthened the existing `RS_SetBandNoDataValue` test in `rasteralgebraTest.scala` so it distinguishes clearing from null-propagation: it sets a no-data value, clears it with NULL (both the 2- and 3-argument forms), and asserts the resulting raster is non-null while `RS_BandNoDataValue` on it is null; it also asserts a NULL raster input still yields NULL. Before this change the previous test passed under both semantics because `RS_BandNoDataValue` returns null for a null raster too. Verified the new assertions fail without the expression change and pass with it (`rasteralgebraTest`, Spark 3.5 / Scala 2.12 / JDK 17).
   
   ## Did this PR include necessary documentation updates?
   
   - Yes, I have updated the documentation.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #3312:
URL: https://github.com/apache/sedona/pull/3312#discussion_r3928025261


##########
common/src/main/java/org/apache/sedona/common/raster/RasterBandEditors.java:
##########
@@ -53,14 +55,22 @@ public static GridCoverage2D setBandNoDataValue(
 
     // Remove no-Data if it is null
     if (noDataValue == null) {
-      if (RasterBandAccessors.getBandNoDataValue(raster) == null) {
+      if (rasterNoData == null) {
         return raster;
       }
       GridSampleDimension[] sampleDimensions = raster.getSampleDimensions();
       sampleDimensions[bandIndex - 1] =
           RasterUtils.removeNoDataValue(sampleDimensions[bandIndex - 1]);
-      return RasterUtils.clone(
-          raster.getRenderedImage(), null, sampleDimensions, raster, null, true);
+      // The GC_NODATA sentinel also rides on the rendered image itself as an image
+      // property and survives serialization there, so cloning the image would let
+      // readers and writers resurrect the cleared value. Copy the pixels into a
+      // property-free image and drop the sentinel from the coverage properties too.
+      Map propertyMap = raster.getProperties();
+      if (propertyMap != null) {
+        propertyMap.remove(NoDataContainer.GC_NODATA);
+      }
+      WritableRaster pixels = raster.getRenderedImage().copyData(null);

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #3312:
URL: https://github.com/apache/sedona/pull/3312#issuecomment-5593470598

   I simplified this PR. The issues with bandwise no data values, GC_NODATA persisting after SetNoDataValueCalls, and writing geotiff No Data Values are orthogonal to this PR and thus out of scope.
   
   They are best handled separately.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #3312:
URL: https://github.com/apache/sedona/pull/3312#discussion_r3926865132


##########
spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/raster/RasterBandEditors.scala:
##########
@@ -22,13 +22,15 @@ import org.apache.sedona.common.raster.RasterBandEditors
 import org.apache.spark.sql.catalyst.expressions.Expression
 import org.apache.spark.sql.sedona_sql.expressions.InferrableFunctionConverter._
 import org.apache.spark.sql.sedona_sql.expressions.InferrableRasterTypes._
-import org.apache.spark.sql.sedona_sql.expressions.InferredExpression
+import org.apache.spark.sql.sedona_sql.expressions.{InferrableFunction, InferredExpression}
 
 private[apache] case class RS_SetBandNoDataValue(inputExpressions: Seq[Expression])
     extends InferredExpression(
       inferrableFunction4(RasterBandEditors.setBandNoDataValue),
-      inferrableFunction3(RasterBandEditors.setBandNoDataValue),
-      inferrableFunction2(RasterBandEditors.setBandNoDataValue)) {
+      // A null noDataValue removes the band's no-data value, so let it through instead of
+      // null-propagating the whole expression.
+      InferrableFunction.allowRightNull3(RasterBandEditors.setBandNoDataValue),

Review Comment:
   fixed. preexisting bug that was latent because this whole case was unreachable before.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #3312:
URL: https://github.com/apache/sedona/pull/3312#discussion_r3926863488


##########
common/src/main/java/org/apache/sedona/common/raster/RasterOutputs.java:
##########
@@ -62,6 +63,19 @@ public static byte[] asGeoTiff(
       throw new RuntimeException(e);
     }
     ParameterValueGroup defaultParams = writer.getFormat().getWriteParameters();
+    // Without this, GeoTiffWriter writes a default GDAL_NODATA of 0 for coverages that
+    // have no no-data value, so a cleared (or never set) no-data value comes back as 0
+    // when the written bytes are read again.
+    boolean hasNoDataValue = false;
+    for (int band = 1; band <= raster.getNumSampleDimensions(); band++) {
+      if (RasterBandAccessors.getBandNoDataValue(raster, band) != null) {
+        hasNoDataValue = true;
+        break;
+      }
+    }
+    defaultParams
+        .parameter(GeoTiffFormat.WRITE_NODATA.getName().toString())
+        .setValue(hasNoDataValue);

Review Comment:
   Do we need to preserve the GeoTools opt-out here? With `-Dgeotiff.writenodata=false`, the descriptor default is false, but this overwrites it with true whenever any band has NoData, so `RS_AsGeoTiff` emits `GDAL_NODATA` despite the explicit setting. Could we only force false for the no-NoData case and otherwise leave the configured default alone?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #3312:
URL: https://github.com/apache/sedona/pull/3312#discussion_r3963133344


##########
common/src/main/java/org/apache/sedona/common/utils/PropertyMaskedRenderedImage.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.sedona.common.utils;
+
+import java.awt.Image;
+import java.awt.Rectangle;
+import java.awt.image.ColorModel;
+import java.awt.image.Raster;
+import java.awt.image.RenderedImage;
+import java.awt.image.SampleModel;
+import java.awt.image.WritableRaster;
+import java.util.Arrays;
+import java.util.Vector;
+
+/**
+ * A {@link RenderedImage} that delegates everything to a source image but hides a single property.
+ *
+ * <p>Image properties cannot be removed in place: {@link javax.media.jai.RenderedImageAdapter}
+ * declares {@code getProperty} final and answers from the source, so {@code removeProperty} cannot
+ * mask an inherited value. Wrapping is also the only way to drop a property without touching pixels
+ * — copying the raster would materialize a lazily decoded image and lose a non-zero image origin.
+ */
+public class PropertyMaskedRenderedImage implements RenderedImage {
+  private final RenderedImage source;
+  private final String maskedProperty;
+
+  private PropertyMaskedRenderedImage(RenderedImage source, String maskedProperty) {
+    this.source = source;
+    this.maskedProperty = maskedProperty;
+  }
+
+  /**
+   * Wrap {@code source} so that {@code propertyName} reads back as {@link Image#UndefinedProperty}.
+   * Returns the source unchanged when it does not carry the property.
+   */
+  public static RenderedImage mask(RenderedImage source, String propertyName) {
+    if (source.getProperty(propertyName) == Image.UndefinedProperty) {
+      return source;
+    }
+    return new PropertyMaskedRenderedImage(source, propertyName);

Review Comment:
   Why does this matter?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #3312:
URL: https://github.com/apache/sedona/pull/3312#discussion_r3926880800


##########
common/src/main/java/org/apache/sedona/common/raster/RasterOutputs.java:
##########
@@ -62,6 +63,19 @@ public static byte[] asGeoTiff(
       throw new RuntimeException(e);
     }
     ParameterValueGroup defaultParams = writer.getFormat().getWriteParameters();
+    // Without this, GeoTiffWriter writes a default GDAL_NODATA of 0 for coverages that
+    // have no no-data value, so a cleared (or never set) no-data value comes back as 0
+    // when the written bytes are read again.
+    boolean hasNoDataValue = false;
+    for (int band = 1; band <= raster.getNumSampleDimensions(); band++) {

Review Comment:
   Isn't this a larger question than this PR about a mismatch between serialized and in memory representations of no data values? 
   
   I don't really see how we would fix this in this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #3312:
URL: https://github.com/apache/sedona/pull/3312#discussion_r3922403638


##########
spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/raster/RasterBandEditors.scala:
##########
@@ -22,13 +22,15 @@ import org.apache.sedona.common.raster.RasterBandEditors
 import org.apache.spark.sql.catalyst.expressions.Expression
 import org.apache.spark.sql.sedona_sql.expressions.InferrableFunctionConverter._
 import org.apache.spark.sql.sedona_sql.expressions.InferrableRasterTypes._
-import org.apache.spark.sql.sedona_sql.expressions.InferredExpression
+import org.apache.spark.sql.sedona_sql.expressions.{InferrableFunction, InferredExpression}
 
 private[apache] case class RS_SetBandNoDataValue(inputExpressions: Seq[Expression])
     extends InferredExpression(
       inferrableFunction4(RasterBandEditors.setBandNoDataValue),
-      inferrableFunction3(RasterBandEditors.setBandNoDataValue),
-      inferrableFunction2(RasterBandEditors.setBandNoDataValue)) {
+      // A null noDataValue removes the band's no-data value, so let it through instead of
+      // null-propagating the whole expression.
+      InferrableFunction.allowRightNull3(RasterBandEditors.setBandNoDataValue),

Review Comment:
   Could this use the `rasterNoData` value read for `bandIndex`? The null path in the common implementation currently calls `getBandNoDataValue(raster)`, which checks band 1. I think this leaves band 2 unchanged when band 1 has no NoData:
   
   ```scala
   val df = sparkSession.sql(
     "SELECT RS_MakeEmptyRaster(2, 20, 20, 0, 0, 8, 8, 0.1, 0.1, 0) AS raster")
   val result = df
     .selectExpr("RS_SetBandNoDataValue(raster, 2, 444) AS raster")
     .selectExpr("RS_BandNoDataValue(RS_SetBandNoDataValue(raster, 2, NULL), 2)")
     .first().getDouble(0)
   // 444.0
   ```
   
   I expected the last value to be null. An asymmetric multi-band test may be useful here.



##########
spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/raster/RasterBandEditors.scala:
##########
@@ -22,13 +22,15 @@ import org.apache.sedona.common.raster.RasterBandEditors
 import org.apache.spark.sql.catalyst.expressions.Expression
 import org.apache.spark.sql.sedona_sql.expressions.InferrableFunctionConverter._
 import org.apache.spark.sql.sedona_sql.expressions.InferrableRasterTypes._
-import org.apache.spark.sql.sedona_sql.expressions.InferredExpression
+import org.apache.spark.sql.sedona_sql.expressions.{InferrableFunction, InferredExpression}
 
 private[apache] case class RS_SetBandNoDataValue(inputExpressions: Seq[Expression])
     extends InferredExpression(
       inferrableFunction4(RasterBandEditors.setBandNoDataValue),
-      inferrableFunction3(RasterBandEditors.setBandNoDataValue),
-      inferrableFunction2(RasterBandEditors.setBandNoDataValue)) {
+      // A null noDataValue removes the band's no-data value, so let it through instead of
+      // null-propagating the whole expression.
+      InferrableFunction.allowRightNull3(RasterBandEditors.setBandNoDataValue),
+      InferrableFunction.allowRightNull(RasterBandEditors.setBandNoDataValue)) {

Review Comment:
   Could we also clear the image-level `GC_NODATA` metadata? On `raster_with_no_data/test5.tiff`, this round trip seems to bring `0` back:
   
   ```scala
   val result = sparkSession.read.format("binaryFile")
     .load(resourceFolder + "raster/raster_with_no_data/test5.tiff")
     .selectExpr("RS_SetBandNoDataValue(RS_FromGeoTiff(content), NULL) AS raster")
     .selectExpr("RS_AsGeoTiff(raster) AS bytes")
     .selectExpr("RS_BandNoDataValue(RS_FromGeoTiff(bytes)) AS nodata")
     .first().getDouble(0)
   // 0.0
   ```
   
   Before serialization the accessor reports null, but `clone(..., keepMetadata = true)` appears to preserve `GC_NODATA=0`. I think removing or recomputing that property along with the sample dimension would avoid downstream readers treating zero as NoData.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #3312:
URL: https://github.com/apache/sedona/pull/3312#discussion_r3926861747


##########
common/src/main/java/org/apache/sedona/common/raster/RasterOutputs.java:
##########
@@ -62,6 +63,19 @@ public static byte[] asGeoTiff(
       throw new RuntimeException(e);
     }
     ParameterValueGroup defaultParams = writer.getFormat().getWriteParameters();
+    // Without this, GeoTiffWriter writes a default GDAL_NODATA of 0 for coverages that
+    // have no no-data value, so a cleared (or never set) no-data value comes back as 0
+    // when the written bytes are read again.
+    boolean hasNoDataValue = false;
+    for (int band = 1; band <= raster.getNumSampleDimensions(); band++) {

Review Comment:
   I may be missing a GeoTIFF constraint here, but this still looks lossy for mixed-band metadata. I reproduced it with a two-band GeoTIFF where both bands start with `-999`:
   
   - clear band 2: `(-999, null)` in memory, `(-999, -999)` after `RS_FromGeoTiff(RS_AsGeoTiff(...))`
   - clear band 1: `(null, -999)` in memory, `(null, null)` after the same round trip
   
   GeoTools writes one dataset-wide NoData value derived from band 1, so `hasNoDataValue` cannot encode either heterogeneous state. Would it make sense to reject this state or use a lossless representation, and add both cases to the round-trip test?



##########
common/src/main/java/org/apache/sedona/common/raster/RasterBandEditors.java:
##########
@@ -53,14 +55,22 @@ public static GridCoverage2D setBandNoDataValue(
 
     // Remove no-Data if it is null
     if (noDataValue == null) {
-      if (RasterBandAccessors.getBandNoDataValue(raster) == null) {
+      if (rasterNoData == null) {
         return raster;
       }
       GridSampleDimension[] sampleDimensions = raster.getSampleDimensions();
       sampleDimensions[bandIndex - 1] =
           RasterUtils.removeNoDataValue(sampleDimensions[bandIndex - 1]);
-      return RasterUtils.clone(
-          raster.getRenderedImage(), null, sampleDimensions, raster, null, true);
+      // The GC_NODATA sentinel also rides on the rendered image itself as an image
+      // property and survives serialization there, so cloning the image would let
+      // readers and writers resurrect the cleared value. Copy the pixels into a
+      // property-free image and drop the sentinel from the coverage properties too.
+      Map propertyMap = raster.getProperties();
+      if (propertyMap != null) {
+        propertyMap.remove(NoDataContainer.GC_NODATA);
+      }
+      WritableRaster pixels = raster.getRenderedImage().copyData(null);

Review Comment:
   Could we keep this on a `RenderedImage` path? I reproduced a clear on a valid coverage with image origin `(5, 7)` and NoData `0`. `copyData(null)` keeps that origin, then the `BufferedImage` constructed in `RasterUtils.clone` throws `IllegalArgumentException: Raster ... has minX or minY not equal to zero: 5 7`. The previous clone path succeeds and preserves the origin. A translated-raster regression test would cover it.



##########
common/src/main/java/org/apache/sedona/common/raster/RasterBandEditors.java:
##########
@@ -53,14 +55,22 @@ public static GridCoverage2D setBandNoDataValue(
 
     // Remove no-Data if it is null
     if (noDataValue == null) {
-      if (RasterBandAccessors.getBandNoDataValue(raster) == null) {
+      if (rasterNoData == null) {
         return raster;
       }
       GridSampleDimension[] sampleDimensions = raster.getSampleDimensions();
       sampleDimensions[bandIndex - 1] =
           RasterUtils.removeNoDataValue(sampleDimensions[bandIndex - 1]);
-      return RasterUtils.clone(
-          raster.getRenderedImage(), null, sampleDimensions, raster, null, true);
+      // The GC_NODATA sentinel also rides on the rendered image itself as an image
+      // property and survives serialization there, so cloning the image would let
+      // readers and writers resurrect the cleared value. Copy the pixels into a

Review Comment:
   This may also be expensive for the streaming GeoTIFF path. `RasterConstructors.fromGeoTiff(ImageInputStream)` returns a lazily decoded image, but `copyData(null)` allocates a raster for the full bounds and walks every tile. A 10,000 × 10,000 four-band float raster is about 1.6 GB of copied samples. Could the property be filtered with a delegating `RenderedImage` instead, so clearing metadata does not force pixel materialization?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #3312:
URL: https://github.com/apache/sedona/pull/3312#discussion_r3927155990


##########
common/src/main/java/org/apache/sedona/common/raster/RasterOutputs.java:
##########
@@ -62,6 +63,19 @@ public static byte[] asGeoTiff(
       throw new RuntimeException(e);
     }
     ParameterValueGroup defaultParams = writer.getFormat().getWriteParameters();
+    // Without this, GeoTiffWriter writes a default GDAL_NODATA of 0 for coverages that
+    // have no no-data value, so a cleared (or never set) no-data value comes back as 0
+    // when the written bytes are read again.
+    boolean hasNoDataValue = false;
+    for (int band = 1; band <= raster.getNumSampleDimensions(); band++) {
+      if (RasterBandAccessors.getBandNoDataValue(raster, band) != null) {
+        hasNoDataValue = true;
+        break;
+      }
+    }
+    defaultParams
+        .parameter(GeoTiffFormat.WRITE_NODATA.getName().toString())
+        .setValue(hasNoDataValue);

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #3312:
URL: https://github.com/apache/sedona/pull/3312#discussion_r3927187081


##########
common/src/main/java/org/apache/sedona/common/raster/RasterOutputs.java:
##########
@@ -62,6 +63,19 @@ public static byte[] asGeoTiff(
       throw new RuntimeException(e);
     }
     ParameterValueGroup defaultParams = writer.getFormat().getWriteParameters();
+    // Without this, GeoTiffWriter writes a default GDAL_NODATA of 0 for coverages that
+    // have no no-data value, so a cleared (or never set) no-data value comes back as 0
+    // when the written bytes are read again.
+    boolean hasNoDataValue = false;
+    for (int band = 1; band <= raster.getNumSampleDimensions(); band++) {

Review Comment:
   AFAICT theres no standard for heterogenous no data values in Tiffs. We should just reject this state in `RS_AsGeoTiff`. But this is all out of scope for this PR; you can file a separate issue if you'd like



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #3312:
URL: https://github.com/apache/sedona/pull/3312#discussion_r3928023069


##########
common/src/main/java/org/apache/sedona/common/raster/RasterBandEditors.java:
##########
@@ -53,14 +55,22 @@ public static GridCoverage2D setBandNoDataValue(
 
     // Remove no-Data if it is null
     if (noDataValue == null) {
-      if (RasterBandAccessors.getBandNoDataValue(raster) == null) {
+      if (rasterNoData == null) {
         return raster;
       }
       GridSampleDimension[] sampleDimensions = raster.getSampleDimensions();
       sampleDimensions[bandIndex - 1] =
           RasterUtils.removeNoDataValue(sampleDimensions[bandIndex - 1]);
-      return RasterUtils.clone(
-          raster.getRenderedImage(), null, sampleDimensions, raster, null, true);
+      // The GC_NODATA sentinel also rides on the rendered image itself as an image
+      // property and survives serialization there, so cloning the image would let
+      // readers and writers resurrect the cleared value. Copy the pixels into a

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu merged PR #3312:
URL: https://github.com/apache/sedona/pull/3312


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #3312:
URL: https://github.com/apache/sedona/pull/3312#discussion_r3963244021


##########
common/src/main/java/org/apache/sedona/common/utils/PropertyMaskedRenderedImage.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.sedona.common.utils;
+
+import java.awt.Image;
+import java.awt.Rectangle;
+import java.awt.image.ColorModel;
+import java.awt.image.Raster;
+import java.awt.image.RenderedImage;
+import java.awt.image.SampleModel;
+import java.awt.image.WritableRaster;
+import java.util.Arrays;
+import java.util.Vector;
+
+/**
+ * A {@link RenderedImage} that delegates everything to a source image but hides a single property.
+ *
+ * <p>Image properties cannot be removed in place: {@link javax.media.jai.RenderedImageAdapter}
+ * declares {@code getProperty} final and answers from the source, so {@code removeProperty} cannot
+ * mask an inherited value. Wrapping is also the only way to drop a property without touching pixels
+ * — copying the raster would materialize a lazily decoded image and lose a non-zero image origin.
+ */
+public class PropertyMaskedRenderedImage implements RenderedImage {
+  private final RenderedImage source;
+  private final String maskedProperty;
+
+  private PropertyMaskedRenderedImage(RenderedImage source, String maskedProperty) {
+    this.source = source;
+    this.maskedProperty = maskedProperty;
+  }
+
+  /**
+   * Wrap {@code source} so that {@code propertyName} reads back as {@link Image#UndefinedProperty}.
+   * Returns the source unchanged when it does not carry the property.
+   */
+  public static RenderedImage mask(RenderedImage source, String propertyName) {
+    if (source.getProperty(propertyName) == Image.UndefinedProperty) {
+      return source;
+    }
+    return new PropertyMaskedRenderedImage(source, propertyName);

Review Comment:
   this is moot now anway



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #3312:
URL: https://github.com/apache/sedona/pull/3312#discussion_r3928028888


##########
spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/raster/RasterBandEditors.scala:
##########
@@ -22,13 +22,15 @@ import org.apache.sedona.common.raster.RasterBandEditors
 import org.apache.spark.sql.catalyst.expressions.Expression
 import org.apache.spark.sql.sedona_sql.expressions.InferrableFunctionConverter._
 import org.apache.spark.sql.sedona_sql.expressions.InferrableRasterTypes._
-import org.apache.spark.sql.sedona_sql.expressions.InferredExpression
+import org.apache.spark.sql.sedona_sql.expressions.{InferrableFunction, InferredExpression}
 
 private[apache] case class RS_SetBandNoDataValue(inputExpressions: Seq[Expression])
     extends InferredExpression(
       inferrableFunction4(RasterBandEditors.setBandNoDataValue),
-      inferrableFunction3(RasterBandEditors.setBandNoDataValue),
-      inferrableFunction2(RasterBandEditors.setBandNoDataValue)) {
+      // A null noDataValue removes the band's no-data value, so let it through instead of
+      // null-propagating the whole expression.
+      InferrableFunction.allowRightNull3(RasterBandEditors.setBandNoDataValue),
+      InferrableFunction.allowRightNull(RasterBandEditors.setBandNoDataValue)) {

Review Comment:
   I cleaned this story up + added a regression test



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-3311] Make RS_SetBandNoDataValue honor NULL to remove a band's no-data value in Spark SQL [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #3312:
URL: https://github.com/apache/sedona/pull/3312#discussion_r3939865839


##########
common/src/main/java/org/apache/sedona/common/utils/PropertyMaskedRenderedImage.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.sedona.common.utils;
+
+import java.awt.Image;
+import java.awt.Rectangle;
+import java.awt.image.ColorModel;
+import java.awt.image.Raster;
+import java.awt.image.RenderedImage;
+import java.awt.image.SampleModel;
+import java.awt.image.WritableRaster;
+import java.util.Arrays;
+import java.util.Vector;
+
+/**
+ * A {@link RenderedImage} that delegates everything to a source image but hides a single property.
+ *
+ * <p>Image properties cannot be removed in place: {@link javax.media.jai.RenderedImageAdapter}
+ * declares {@code getProperty} final and answers from the source, so {@code removeProperty} cannot
+ * mask an inherited value. Wrapping is also the only way to drop a property without touching pixels
+ * — copying the raster would materialize a lazily decoded image and lose a non-zero image origin.
+ */
+public class PropertyMaskedRenderedImage implements RenderedImage {
+  private final RenderedImage source;
+  private final String maskedProperty;
+
+  private PropertyMaskedRenderedImage(RenderedImage source, String maskedProperty) {
+    this.source = source;
+    this.maskedProperty = maskedProperty;
+  }
+
+  /**
+   * Wrap {@code source} so that {@code propertyName} reads back as {@link Image#UndefinedProperty}.
+   * Returns the source unchanged when it does not carry the property.
+   */
+  public static RenderedImage mask(RenderedImage source, String propertyName) {
+    if (source.getProperty(propertyName) == Image.UndefinedProperty) {
+      return source;
+    }
+    return new PropertyMaskedRenderedImage(source, propertyName);

Review Comment:
   Could we preserve `WritableRenderedImage` when the source is writable? Clearing NoData currently makes an editable coverage read-only. I reproduced this through the Java API:
   
   ```java
   import java.awt.image.WritableRenderedImage;
   import javax.media.jai.PlanarImage;
   import it.geosolutions.jaiext.range.NoDataContainer;
   import org.apache.sedona.common.raster.*;
   
   var raster = RasterConstructors.makeEmptyRaster(1, "d", 4, 3, 0, 0, 1);
   raster = RasterBandEditors.setBandNoDataValue(raster, 1, -9999.0);
   ((PlanarImage) raster.getRenderedImage()).setProperty(
       NoDataContainer.GC_NODATA, new NoDataContainer(-9999.0));
   
   System.out.println(raster.isDataEditable()); // true
   var cleared = RasterBandEditors.setBandNoDataValue(raster, 1, null);
   System.out.println(cleared.isDataEditable()); // false
   ((WritableRenderedImage) cleared.getRenderedImage()).getWritableTile(0, 0);
   // ClassCastException
   ```
   
   This affects Java/GeoTools callers using writable tiles; I haven't found a Spark SQL failure from it. A writable subclass selected only for writable sources, forwarding the eight `WritableRenderedImage` methods, looks like a small way to preserve that capability while keeping the lazy wrapper. A test that acquires, writes, and releases a tile after clearing would cover it.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org