(spark) branch master updated: [SPARK-51093][SQL][TESTS] Fix minor endianness issues in tests

[ma...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

maxgekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new f5f7c365d519 [SPARK-51093][SQL][TESTS] Fix minor endianness issues in tests
f5f7c365d519 is described below

commit f5f7c365d519c4f9d4b7a5dce2c8a047cf051899
Author: Jonathan Albrecht <jo...@ibm.com>
AuthorDate: Fri Feb 7 10:48:40 2025 +0100

    [SPARK-51093][SQL][TESTS] Fix minor endianness issues in tests
    
    ### What changes were proposed in this pull request?
    Fix minor endianness issues in the following tests.
    
    ArrayBasedMapBuilderSuite: The output of the UnsafeRow.toString() is based on the underlying bytes and is endian dependent. Add an expected value for big endian platforms. Add an expected value for big endian platforms.
    
    WriteDistributionAndOrderingSuite: Casting the id of type Int to Long doesn't work on big endian platforms because the BucketFunction calls UnsafeRow.getLong() for that column. That happens to work on little endian since an int field is stored in the first 4 bytes of the 8 byte field so positive ints are layed out the same as positive longs ie. little endian order. On big endian, the layout of UnsafeRow int fields does not happen to match the layout of long fields for the same number. [...]
    
    ### Why are the changes needed?
    Allow tests to pass on big endian platforms
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    Ran existing tests on amd64 (little endian) and s390x (big endian)
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No
    
    Closes #49812 from jonathan-albrecht-ibm/master-endian-testEndianness.
    
    Authored-by: Jonathan Albrecht <jo...@ibm.com>
    Signed-off-by: Max Gekk <ma...@gmail.com>
---
 .../apache/spark/sql/catalyst/util/ArrayBasedMapBuilderSuite.scala    | 2 +-
 .../spark/sql/connector/WriteDistributionAndOrderingSuite.scala       | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/ArrayBasedMapBuilderSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/ArrayBasedMapBuilderSuite.scala
index 0e872dcdb626..5352acf23c82 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/ArrayBasedMapBuilderSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/ArrayBasedMapBuilderSuite.scala
@@ -148,7 +148,7 @@ class ArrayBasedMapBuilderSuite extends SparkFunSuite with SQLHelper {
       exception = intercept[SparkRuntimeException](builder.put(unsafeRow, 3)),
       condition = "DUPLICATED_MAP_KEY",
       parameters = Map(
-        "key" -> "[0,1]",
+        "key" -> unsafeRow.toString(),
         "mapKeyDedupPolicy" -> "\"spark.sql.mapKeyDedupPolicy\"")
     )
 
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/WriteDistributionAndOrderingSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/WriteDistributionAndOrderingSuite.scala
index 12d5f13df01c..f62e092138a9 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/connector/WriteDistributionAndOrderingSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/WriteDistributionAndOrderingSuite.scala
@@ -70,7 +70,7 @@ class WriteDistributionAndOrderingSuite extends DistributionAndOrderingSuiteBase
   private val tableNameAsString = "testcat." + ident.toString
   private val emptyProps = Collections.emptyMap[String, String]
   private val columns = Array(
-    Column.create("id", IntegerType),
+    Column.create("id", LongType),
     Column.create("data", StringType),
     Column.create("day", DateType))
 
@@ -1122,7 +1122,7 @@ class WriteDistributionAndOrderingSuite extends DistributionAndOrderingSuiteBase
         Seq.empty
       ),
       catalyst.expressions.SortOrder(
-        ApplyFunctionExpression(BucketFunction, Seq(Literal(10), Cast(attr("id"), LongType))),
+        ApplyFunctionExpression(BucketFunction, Seq(Literal(10), attr("id"))),
         catalyst.expressions.Descending,
         catalyst.expressions.NullsFirst,
         Seq.empty


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org