(spark) branch master updated: [SPARK-51349][SQL][TESTS] Change precedence of null and "null" in sorting in QueryTest

[we...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new fa9dab054222 [SPARK-51349][SQL][TESTS] Change precedence of null and "null" in sorting in QueryTest
fa9dab054222 is described below

commit fa9dab05422235070d2ff586638331b2ab3cfd72
Author: Harsh Motwani <ha...@databricks.com>
AuthorDate: Tue Mar 18 13:53:41 2025 +0800

    [SPARK-51349][SQL][TESTS] Change precedence of null and "null" in sorting in QueryTest
    
    ### What changes were proposed in this pull request?
    
    Prior to this PR, it was difficult to evaluate tests where the resulting DataFrame would contain both "null" strings and null values since they had the same precedence in QueryTest sorting. This PR changes their precedence.
    
    ### Why are the changes needed?
    
    Currently DataFrames representing `["null", null]` and `[null, "null"]` are not considered equal in checkAnswer even though they are. This PR fixes it.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Unit test.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #50108 from harshmotw-db/harsh-motwani_data/variant_get_column.
    
    Authored-by: Harsh Motwani <ha...@databricks.com>
    Signed-off-by: Wenchen Fan <we...@databricks.com>
---
 sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala b/sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala
index 7ae29a6a1712..42b3b12efe39 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala
@@ -347,6 +347,8 @@ object QueryTest extends Assertions {
       // Convert array to Seq for easy equality check.
       case b: Array[_] => b.toSeq
       case r: Row => prepareRow(r)
+      // SPARK-51349: "null" and null had the same precedence in sorting
+      case "null" => "__null_string__"
       case o => o
     })
   }
@@ -479,4 +481,10 @@ class QueryTestSuite extends QueryTest with test.SharedSparkSession {
       checkAnswer(sql("SELECT 1"), Row(2) :: Nil)
     }
   }
+
+  test("SPARK-51349: null string and true null are distinguished") {
+    checkAnswer(sql("select case when id == 0 then struct('null') else struct(null) end s " +
+      "from range(2)"),
+      Seq(Row(Row(null)), Row(Row("null"))))
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org