You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "GGraziadei (via GitHub)" <gi...@apache.org> on 2026/08/15 11:48:54 UTC

[PR] Spark: Fix Z-order NPE on null booleans and case-insensitive column resolution [iceberg]

GGraziadei opened a new pull request, #17669:
URL: https://github.com/apache/iceberg/pull/17669

   Closes #17658.
   
   Two independent defects in the Spark Z-order clustering path, both spotted by @manuzhang while
   reviewing #16827. They are fixed here rather than in that PR because they change existing Z-order
   behaviour and deserve their own review. Both are present in `spark/v3.5`, `spark/v4.0` and
   `spark/v4.1`, so all three are fixed together.
   
   ### 1. NPE when a boolean clustering column contains nulls
   
   `SparkZOrderUDF.booleanToOrderedBytesUDF()` unboxed a possibly-null `Boolean` with no guard, so a
   single null aborted the whole rewrite with:
   
   ```
   [FAILED_EXECUTE_UDF] ... NullPointerException:
   Cannot invoke "java.lang.Boolean.booleanValue()" because "value" is null
   ```
   
   Boolean is the **only** converter with this problem — the seven numeric and temporal converters
   already return `PRIMITIVE_EMPTY` for null, and `stringToOrderedBytes` / `byteTruncateOrFill` handle
   it inside `ZOrderByteUtils`. The surrounding code therefore already establishes the semantics — a
   null sorts lowest, as all-zero bytes — and this change simply makes boolean follow it. No type that
   works today changes behaviour.
   
   ### 2. Case-insensitive validation kept the caller's spelling
   
   `validZOrderColNames` resolved the column case-insensitively when `spark.sql.caseSensitive` is
   `false` (the Spark default) but then stored the caller's spelling instead of the resolved field
   name. Validation passed, and the mis-cased name later reached the case-sensitive
   `df.schema().apply(...)` in `zValue`, failing with:
   
   ```
   [FIELD_NOT_FOUND] No such struct field `C2` in `c1`, `c2`, `c3`. SQLSTATE: 42704
   ```
   
   Z-ordering on `C2` in a table whose column is `c2` now works, which is what case-insensitive
   resolution promised.
   
   ### Tests
   
   Three regression tests were added to `TestRewriteDataFilesAction` in each Spark version. Each was
   written first and confirmed to fail for the expected reason before the fix:
   
   - `zOrderUDFEncodesNullValuesAsZeroBytes` — asserts that a null of every one of the 12 supported
     types produces all-zero ordered bytes, pinning the null-sorts-lowest convention across types
     rather than only checking that boolean no longer throws;
   - `zOrderSortWithNullBooleanValues` — end-to-end rewrite over a table with null booleans;
   - `zOrderSortWithMismatchedColumnCase` — end-to-end rewrite using `C2`/`C3` against columns
     `c2`/`c3`.
   
   Full `TestRewriteDataFilesAction` suite passes on all three versions (549 tests total, 0 failures),
   along with `checkstyleMain`, `checkstyleTest` and `spotlessCheck`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix Z-order NPE on null booleans and case-insensitive column resolution [iceberg]

Posted by "GGraziadei (via GitHub)" <gi...@apache.org>.
GGraziadei commented on PR #17669:
URL: https://github.com/apache/iceberg/pull/17669#issuecomment-5607772669

   To put this change in context: the NPE fix here is about correctness of the code path, not about making boolean Z-ordering more useful. A boolean contributes a single meaningful bit to the interleaved key, and since that bit lands at the top of the Z-value it effectively splits the data into two halves and then Z-orders the remaining columns within each half. That is a valid layout, but it is closer to `sort by flag` than to real multi-dimensional clustering, and the same is true for any very low-cardinality column.
   
   Would it make sense to also log a warning when a Z-order or Hilbert column is boolean, or more generally has a type whose cardinality is known to be tiny? It wouldn't change behavior, but it would steer users away from a configuration that does less than they probably expect.
   
   Happy to add it in this PR or as a follow-up, whichever you prefer.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix Z-order NPE on null booleans and case-insensitive column resolution [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #17669:
URL: https://github.com/apache/iceberg/pull/17669#issuecomment-5638973453

   @GGraziadei Could port the changes to 3.5, 4.0 and 4.2. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix Z-order NPE on null booleans and case-insensitive column resolution [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #17669:
URL: https://github.com/apache/iceberg/pull/17669#issuecomment-5637902194

   Thanks @GGraziadei for the PR! Thanks @nssalian @uros-b for the review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix Z-order NPE on null booleans and case-insensitive column resolution [iceberg]

Posted by "GGraziadei (via GitHub)" <gi...@apache.org>.
GGraziadei commented on PR #17669:
URL: https://github.com/apache/iceberg/pull/17669#issuecomment-5303005779

   Thanks for the feedback, @manuzhang ! I've reduced the scope of this PR to target Spark 4.1. Once this is merged, I'll follow up with PRs for 4.0 and 3.5.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix Z-order NPE on null booleans and case-insensitive column resolution [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao merged PR #17669:
URL: https://github.com/apache/iceberg/pull/17669


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix Z-order NPE on null booleans and case-insensitive column resolution [iceberg]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #17669:
URL: https://github.com/apache/iceberg/pull/17669#issuecomment-5302554188

   @GGraziadei Please submit your patch against Spark 4.1 first and back-port to previous versions when it's merged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix Z-order NPE on null booleans and case-insensitive column resolution [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17669:
URL: https://github.com/apache/iceberg/pull/17669#discussion_r3965295614


##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/SparkZOrderFileRewriteRunner.java:
##########
@@ -198,7 +198,8 @@ private List<String> validZOrderColNames(
       if (identityPartitionFieldIds.contains(field.fieldId())) {
         LOG.warn("Ignoring '{}' as such values are constant within a partition", colName);
       } else {
-        validZOrderColNames.add(colName);
+        // use the resolved name so a case-insensitive match is not carried through as-is
+        validZOrderColNames.add(field.name());

Review Comment:
   `field.name()` is the leaf name, so a dotted input loses its path: `nested.c2` is stored as `c2`. Usually harmless (it still fails downstream), but if the table also has a top-level `c2` it resolves to that column and Z-orders it instead of failing.
   
   ```suggestion
           validZOrderColNames.add(schema.findColumnName(field.fieldId()));
   ```



##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java:
##########
@@ -2209,6 +2212,105 @@ public void testZOrderUDFWithTimestampNTZType() {
     assertThat(zorderBytes).isNotNull().isNotEmpty();
   }
 
+  @TestTemplate
+  public void zOrderUDFEncodesNullValuesAsZeroBytes() {
+    Object[][] nullsByType = {
+      {"CAST(NULL AS BOOLEAN)", DataTypes.BooleanType},
+      {"CAST(NULL AS TINYINT)", DataTypes.ByteType},
+      {"CAST(NULL AS SMALLINT)", DataTypes.ShortType},
+      {"CAST(NULL AS INT)", DataTypes.IntegerType},
+      {"CAST(NULL AS BIGINT)", DataTypes.LongType},
+      {"CAST(NULL AS FLOAT)", DataTypes.FloatType},
+      {"CAST(NULL AS DOUBLE)", DataTypes.DoubleType},
+      {"CAST(NULL AS DATE)", DataTypes.DateType},
+      {"CAST(NULL AS TIMESTAMP)", DataTypes.TimestampType},
+      {"CAST(NULL AS TIMESTAMP_NTZ)", DataTypes.TimestampNTZType},
+      {"CAST(NULL AS STRING)", DataTypes.StringType},
+      {"CAST(NULL AS BINARY)", DataTypes.BinaryType},
+    };
+
+    for (Object[] nullByType : nullsByType) {
+      String literal = (String) nullByType[0];
+      DataType type = (DataType) nullByType[1];
+      SparkZOrderUDF zorderUDF = new SparkZOrderUDF(1, 16, 1024);
+      Dataset<Row> result =
+          spark
+              .sql("SELECT " + literal + " as test_col")
+              .withColumn(
+                  "zorder_result", zorderUDF.sortedLexicographically(col("test_col"), type));
+
+      byte[] zorderBytes = result.collectAsList().get(0).getAs("zorder_result");
+      assertThat(zorderBytes)
+          .as("A null %s must produce ordered bytes rather than failing", type.simpleString())
+          .isNotNull()
+          .isNotEmpty();
+      assertThat(zorderBytes)
+          .as("A null %s must sort lowest, as an all-zero byte sequence", type.simpleString())
+          .containsOnly((byte) 0);
+    }
+  }
+
+  @TestTemplate
+  public void zOrderSortWithNullBooleanValues() {
+    Schema schema =
+        new Schema(
+            optional(1, "id", Types.IntegerType.get()),
+            optional(2, "flag", Types.BooleanType.get()));
+    Table table =
+        TABLES.create(
+            schema,
+            PartitionSpec.unpartitioned(),
+            ImmutableMap.of(TableProperties.FORMAT_VERSION, String.valueOf(formatVersion)),
+            tableLocation);
+
+    for (int batch = 0; batch < 2; batch++) {
+      spark
+          .createDataFrame(
+              Lists.newArrayList(
+                  RowFactory.create(1, true),
+                  RowFactory.create(2, null),
+                  RowFactory.create(3, false)),
+              SparkSchemaUtil.convert(schema))
+          .write()
+          .format("iceberg")
+          .mode("append")
+          .save(tableLocation);
+    }
+    table.refresh();
+
+    long dataSizeBefore = testDataSize(table);
+    RewriteDataFiles.Result result =
+        basicRewrite(table)
+            .zOrder("id", "flag")
+            .option(SizeBasedFileRewritePlanner.MIN_INPUT_FILES, "1")
+            .execute();
+
+    assertThat(result.rewrittenBytesCount()).isEqualTo(dataSizeBefore);
+    assertThat(result.rewrittenDataFilesCount()).isGreaterThan(0);
+    assertThat(spark.read().format("iceberg").load(tableLocation).filter("flag IS NULL").count())
+        .isEqualTo(2);
+  }
+
+  @TestTemplate
+  public void zOrderSortWithMismatchedColumnCase() {
+    assertThat(spark.conf().get("spark.sql.caseSensitive"))
+        .as("This test covers the case-insensitive column resolution path")
+        .isEqualTo("false");

Review Comment:
   this asserts on whatever `spark.sql.caseSensitive` happens to be rather than setting it, so the test quietly depends on no earlier test in the shared session having changed it — and if one does, it fails on this assertion instead of testing the resolution path. `withSQLConf` pins it and lets the assertion go:
   
   ```java
   withSQLConf(
       ImmutableMap.of(SQLConf.CASE_SENSITIVE().key(), "false"),
       () -> {
         // existing body
       });
   ```
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix Z-order NPE on null booleans and case-insensitive column resolution [iceberg]

Posted by "GGraziadei (via GitHub)" <gi...@apache.org>.
GGraziadei commented on code in PR #17669:
URL: https://github.com/apache/iceberg/pull/17669#discussion_r3972430790


##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java:
##########
@@ -2209,6 +2212,105 @@ public void testZOrderUDFWithTimestampNTZType() {
     assertThat(zorderBytes).isNotNull().isNotEmpty();
   }
 
+  @TestTemplate
+  public void zOrderUDFEncodesNullValuesAsZeroBytes() {
+    Object[][] nullsByType = {
+      {"CAST(NULL AS BOOLEAN)", DataTypes.BooleanType},
+      {"CAST(NULL AS TINYINT)", DataTypes.ByteType},
+      {"CAST(NULL AS SMALLINT)", DataTypes.ShortType},
+      {"CAST(NULL AS INT)", DataTypes.IntegerType},
+      {"CAST(NULL AS BIGINT)", DataTypes.LongType},
+      {"CAST(NULL AS FLOAT)", DataTypes.FloatType},
+      {"CAST(NULL AS DOUBLE)", DataTypes.DoubleType},
+      {"CAST(NULL AS DATE)", DataTypes.DateType},
+      {"CAST(NULL AS TIMESTAMP)", DataTypes.TimestampType},
+      {"CAST(NULL AS TIMESTAMP_NTZ)", DataTypes.TimestampNTZType},
+      {"CAST(NULL AS STRING)", DataTypes.StringType},
+      {"CAST(NULL AS BINARY)", DataTypes.BinaryType},
+    };
+
+    for (Object[] nullByType : nullsByType) {
+      String literal = (String) nullByType[0];
+      DataType type = (DataType) nullByType[1];
+      SparkZOrderUDF zorderUDF = new SparkZOrderUDF(1, 16, 1024);
+      Dataset<Row> result =
+          spark
+              .sql("SELECT " + literal + " as test_col")
+              .withColumn(
+                  "zorder_result", zorderUDF.sortedLexicographically(col("test_col"), type));
+
+      byte[] zorderBytes = result.collectAsList().get(0).getAs("zorder_result");
+      assertThat(zorderBytes)
+          .as("A null %s must produce ordered bytes rather than failing", type.simpleString())
+          .isNotNull()
+          .isNotEmpty();
+      assertThat(zorderBytes)
+          .as("A null %s must sort lowest, as an all-zero byte sequence", type.simpleString())
+          .containsOnly((byte) 0);
+    }
+  }
+
+  @TestTemplate
+  public void zOrderSortWithNullBooleanValues() {
+    Schema schema =
+        new Schema(
+            optional(1, "id", Types.IntegerType.get()),
+            optional(2, "flag", Types.BooleanType.get()));
+    Table table =
+        TABLES.create(
+            schema,
+            PartitionSpec.unpartitioned(),
+            ImmutableMap.of(TableProperties.FORMAT_VERSION, String.valueOf(formatVersion)),
+            tableLocation);
+
+    for (int batch = 0; batch < 2; batch++) {
+      spark
+          .createDataFrame(
+              Lists.newArrayList(
+                  RowFactory.create(1, true),
+                  RowFactory.create(2, null),
+                  RowFactory.create(3, false)),
+              SparkSchemaUtil.convert(schema))
+          .write()
+          .format("iceberg")
+          .mode("append")
+          .save(tableLocation);
+    }
+    table.refresh();
+
+    long dataSizeBefore = testDataSize(table);
+    RewriteDataFiles.Result result =
+        basicRewrite(table)
+            .zOrder("id", "flag")
+            .option(SizeBasedFileRewritePlanner.MIN_INPUT_FILES, "1")
+            .execute();
+
+    assertThat(result.rewrittenBytesCount()).isEqualTo(dataSizeBefore);
+    assertThat(result.rewrittenDataFilesCount()).isGreaterThan(0);
+    assertThat(spark.read().format("iceberg").load(tableLocation).filter("flag IS NULL").count())
+        .isEqualTo(2);
+  }
+
+  @TestTemplate
+  public void zOrderSortWithMismatchedColumnCase() {
+    assertThat(spark.conf().get("spark.sql.caseSensitive"))
+        .as("This test covers the case-insensitive column resolution path")
+        .isEqualTo("false");

Review Comment:
   Hi @huaxingao,
   Fixed, thanks. Out of curiosity for future changes: since the session is shared across tests, would it make sense to reset the SQL conf in an @AfterEach in TestBase or directly close and restart the session, so that a test forgetting to use withSQLConf cannot leak config into the others? This point is related to test idempotency



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix Z-order NPE on null booleans and case-insensitive column resolution [iceberg]

Posted by "GGraziadei (via GitHub)" <gi...@apache.org>.
GGraziadei commented on code in PR #17669:
URL: https://github.com/apache/iceberg/pull/17669#discussion_r3972430790


##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java:
##########
@@ -2209,6 +2212,105 @@ public void testZOrderUDFWithTimestampNTZType() {
     assertThat(zorderBytes).isNotNull().isNotEmpty();
   }
 
+  @TestTemplate
+  public void zOrderUDFEncodesNullValuesAsZeroBytes() {
+    Object[][] nullsByType = {
+      {"CAST(NULL AS BOOLEAN)", DataTypes.BooleanType},
+      {"CAST(NULL AS TINYINT)", DataTypes.ByteType},
+      {"CAST(NULL AS SMALLINT)", DataTypes.ShortType},
+      {"CAST(NULL AS INT)", DataTypes.IntegerType},
+      {"CAST(NULL AS BIGINT)", DataTypes.LongType},
+      {"CAST(NULL AS FLOAT)", DataTypes.FloatType},
+      {"CAST(NULL AS DOUBLE)", DataTypes.DoubleType},
+      {"CAST(NULL AS DATE)", DataTypes.DateType},
+      {"CAST(NULL AS TIMESTAMP)", DataTypes.TimestampType},
+      {"CAST(NULL AS TIMESTAMP_NTZ)", DataTypes.TimestampNTZType},
+      {"CAST(NULL AS STRING)", DataTypes.StringType},
+      {"CAST(NULL AS BINARY)", DataTypes.BinaryType},
+    };
+
+    for (Object[] nullByType : nullsByType) {
+      String literal = (String) nullByType[0];
+      DataType type = (DataType) nullByType[1];
+      SparkZOrderUDF zorderUDF = new SparkZOrderUDF(1, 16, 1024);
+      Dataset<Row> result =
+          spark
+              .sql("SELECT " + literal + " as test_col")
+              .withColumn(
+                  "zorder_result", zorderUDF.sortedLexicographically(col("test_col"), type));
+
+      byte[] zorderBytes = result.collectAsList().get(0).getAs("zorder_result");
+      assertThat(zorderBytes)
+          .as("A null %s must produce ordered bytes rather than failing", type.simpleString())
+          .isNotNull()
+          .isNotEmpty();
+      assertThat(zorderBytes)
+          .as("A null %s must sort lowest, as an all-zero byte sequence", type.simpleString())
+          .containsOnly((byte) 0);
+    }
+  }
+
+  @TestTemplate
+  public void zOrderSortWithNullBooleanValues() {
+    Schema schema =
+        new Schema(
+            optional(1, "id", Types.IntegerType.get()),
+            optional(2, "flag", Types.BooleanType.get()));
+    Table table =
+        TABLES.create(
+            schema,
+            PartitionSpec.unpartitioned(),
+            ImmutableMap.of(TableProperties.FORMAT_VERSION, String.valueOf(formatVersion)),
+            tableLocation);
+
+    for (int batch = 0; batch < 2; batch++) {
+      spark
+          .createDataFrame(
+              Lists.newArrayList(
+                  RowFactory.create(1, true),
+                  RowFactory.create(2, null),
+                  RowFactory.create(3, false)),
+              SparkSchemaUtil.convert(schema))
+          .write()
+          .format("iceberg")
+          .mode("append")
+          .save(tableLocation);
+    }
+    table.refresh();
+
+    long dataSizeBefore = testDataSize(table);
+    RewriteDataFiles.Result result =
+        basicRewrite(table)
+            .zOrder("id", "flag")
+            .option(SizeBasedFileRewritePlanner.MIN_INPUT_FILES, "1")
+            .execute();
+
+    assertThat(result.rewrittenBytesCount()).isEqualTo(dataSizeBefore);
+    assertThat(result.rewrittenDataFilesCount()).isGreaterThan(0);
+    assertThat(spark.read().format("iceberg").load(tableLocation).filter("flag IS NULL").count())
+        .isEqualTo(2);
+  }
+
+  @TestTemplate
+  public void zOrderSortWithMismatchedColumnCase() {
+    assertThat(spark.conf().get("spark.sql.caseSensitive"))
+        .as("This test covers the case-insensitive column resolution path")
+        .isEqualTo("false");

Review Comment:
   Fixed, thanks. Out of curiosity for future changes: since the session is shared across tests, would it make sense to reset the SQL conf in an `@AfterEach` in `TestBase`, so that a test forgetting to use `withSQLConf` cannot leak config into the others?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix Z-order NPE on null booleans and case-insensitive column resolution [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17669:
URL: https://github.com/apache/iceberg/pull/17669#discussion_r3974703022


##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java:
##########
@@ -2209,6 +2212,105 @@ public void testZOrderUDFWithTimestampNTZType() {
     assertThat(zorderBytes).isNotNull().isNotEmpty();
   }
 
+  @TestTemplate
+  public void zOrderUDFEncodesNullValuesAsZeroBytes() {
+    Object[][] nullsByType = {
+      {"CAST(NULL AS BOOLEAN)", DataTypes.BooleanType},
+      {"CAST(NULL AS TINYINT)", DataTypes.ByteType},
+      {"CAST(NULL AS SMALLINT)", DataTypes.ShortType},
+      {"CAST(NULL AS INT)", DataTypes.IntegerType},
+      {"CAST(NULL AS BIGINT)", DataTypes.LongType},
+      {"CAST(NULL AS FLOAT)", DataTypes.FloatType},
+      {"CAST(NULL AS DOUBLE)", DataTypes.DoubleType},
+      {"CAST(NULL AS DATE)", DataTypes.DateType},
+      {"CAST(NULL AS TIMESTAMP)", DataTypes.TimestampType},
+      {"CAST(NULL AS TIMESTAMP_NTZ)", DataTypes.TimestampNTZType},
+      {"CAST(NULL AS STRING)", DataTypes.StringType},
+      {"CAST(NULL AS BINARY)", DataTypes.BinaryType},
+    };
+
+    for (Object[] nullByType : nullsByType) {
+      String literal = (String) nullByType[0];
+      DataType type = (DataType) nullByType[1];
+      SparkZOrderUDF zorderUDF = new SparkZOrderUDF(1, 16, 1024);
+      Dataset<Row> result =
+          spark
+              .sql("SELECT " + literal + " as test_col")
+              .withColumn(
+                  "zorder_result", zorderUDF.sortedLexicographically(col("test_col"), type));
+
+      byte[] zorderBytes = result.collectAsList().get(0).getAs("zorder_result");
+      assertThat(zorderBytes)
+          .as("A null %s must produce ordered bytes rather than failing", type.simpleString())
+          .isNotNull()
+          .isNotEmpty();
+      assertThat(zorderBytes)
+          .as("A null %s must sort lowest, as an all-zero byte sequence", type.simpleString())
+          .containsOnly((byte) 0);
+    }
+  }
+
+  @TestTemplate
+  public void zOrderSortWithNullBooleanValues() {
+    Schema schema =
+        new Schema(
+            optional(1, "id", Types.IntegerType.get()),
+            optional(2, "flag", Types.BooleanType.get()));
+    Table table =
+        TABLES.create(
+            schema,
+            PartitionSpec.unpartitioned(),
+            ImmutableMap.of(TableProperties.FORMAT_VERSION, String.valueOf(formatVersion)),
+            tableLocation);
+
+    for (int batch = 0; batch < 2; batch++) {
+      spark
+          .createDataFrame(
+              Lists.newArrayList(
+                  RowFactory.create(1, true),
+                  RowFactory.create(2, null),
+                  RowFactory.create(3, false)),
+              SparkSchemaUtil.convert(schema))
+          .write()
+          .format("iceberg")
+          .mode("append")
+          .save(tableLocation);
+    }
+    table.refresh();
+
+    long dataSizeBefore = testDataSize(table);
+    RewriteDataFiles.Result result =
+        basicRewrite(table)
+            .zOrder("id", "flag")
+            .option(SizeBasedFileRewritePlanner.MIN_INPUT_FILES, "1")
+            .execute();
+
+    assertThat(result.rewrittenBytesCount()).isEqualTo(dataSizeBefore);
+    assertThat(result.rewrittenDataFilesCount()).isGreaterThan(0);
+    assertThat(spark.read().format("iceberg").load(tableLocation).filter("flag IS NULL").count())
+        .isEqualTo(2);
+  }
+
+  @TestTemplate
+  public void zOrderSortWithMismatchedColumnCase() {
+    assertThat(spark.conf().get("spark.sql.caseSensitive"))
+        .as("This test covers the case-insensitive column resolution path")
+        .isEqualTo("false");

Review Comment:
   I would not reset config in `@AfterEach` because it runs after every test method, so a
   reset hook would wipe per-class setup. This class is a good example -- `setupSpark()`
   disables AQE in `@BeforeAll` because "tests assume that writes generate a particular
   number of files". Reset after each test and AQE comes back on for every method after the
   first.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix Z-order NPE on null booleans and case-insensitive column resolution [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #17669:
URL: https://github.com/apache/iceberg/pull/17669#issuecomment-5611509678

   > To put this change in context: the NPE fix here is about correctness of the code path, not about making boolean Z-ordering more useful. A boolean contributes a single meaningful bit to the interleaved key, and since that bit lands at the top of the Z-value it effectively splits the data into two halves and then Z-orders the remaining columns within each half. That is a valid layout, but it is closer to sort by flag than to real multi-dimensional clustering, and the same is true for any very low-cardinality column.
   > 
   > Would it make sense to also log a warning when a Z-order or Hilbert column is boolean, or more generally has a type whose cardinality is known to be tiny? It wouldn't change behavior, but it would steer users away from a configuration that does less than they probably expect.
   > 
   > Happy to add it in this PR or as a follow-up, whichever you prefer.
   
   No strong opinion, but I'd leave it out of this PR. The case that actually hurts is a low
   distinct-value count (e.g. a string with three values), which isn't visible from the type
   -- so a type-based warning would miss the common cases while firing on boolean columns
   that are a reasonable choice.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix Z-order NPE on null booleans and case-insensitive column resolution [iceberg]

Posted by "GGraziadei (via GitHub)" <gi...@apache.org>.
GGraziadei commented on code in PR #17669:
URL: https://github.com/apache/iceberg/pull/17669#discussion_r3972129381


##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/SparkZOrderFileRewriteRunner.java:
##########
@@ -198,7 +198,8 @@ private List<String> validZOrderColNames(
       if (identityPartitionFieldIds.contains(field.fieldId())) {
         LOG.warn("Ignoring '{}' as such values are constant within a partition", colName);
       } else {
-        validZOrderColNames.add(colName);
+        // use the resolved name so a case-insensitive match is not carried through as-is
+        validZOrderColNames.add(field.name());

Review Comment:
   Thanks for the clarification. I am applying.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org