(spark) branch branch-4.0 updated: [SPARK-50904][SQL] Fix collation expression walker query execution

[ma...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

maxgekk pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.0 by this push:
     new da41a255ba74 [SPARK-50904][SQL] Fix collation expression walker query execution
da41a255ba74 is described below

commit da41a255ba74a9e0e5c39560cbff95952907fed2
Author: Stefan Kandic <st...@databricks.com>
AuthorDate: Tue Jan 21 18:47:56 2025 +0200

    [SPARK-50904][SQL] Fix collation expression walker query execution
    
    ### What changes were proposed in this pull request?
    Changing when we collect results in `CollationExpressionWalkerSuite` on borders of changing session default collation.
    
    ### Why are the changes needed?
    Because we are analyzing the query and then setting a different default collation, this leads to problems for runtime replaceable expressions which will only be resolved when we call `collect` and by then the session default collation is changed. This is blocking the change in #49576.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    This is a test-only change.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No
    
    Closes #49586 from stefankandic/fixExpressionWalker.
    
    Authored-by: Stefan Kandic <st...@databricks.com>
    Signed-off-by: Max Gekk <ma...@gmail.com>
    (cherry picked from commit dabd21249f1eae38e8e866521caee6cd387c8851)
    Signed-off-by: Max Gekk <ma...@gmail.com>
---
 .../scala/org/apache/spark/sql/CollationExpressionWalkerSuite.scala | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/sql/core/src/test/scala/org/apache/spark/sql/CollationExpressionWalkerSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/CollationExpressionWalkerSuite.scala
index 1f9589c1c9ce..4602c1a9cc1a 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/CollationExpressionWalkerSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/CollationExpressionWalkerSuite.scala
@@ -740,10 +740,10 @@ class CollationExpressionWalkerSuite extends SparkFunSuite with SharedSparkSessi
     for (funInfo <- funInfos.filter(f => !toSkip.contains(f.getName))) {
       for (query <- "> .*;".r.findAllIn(funInfo.getExamples).map(s => s.substring(2))) {
         try {
-          val resultUTF8 = sql(query)
+          val resultUTF8 = sql(query).collect()
           withSQLConf(SqlApiConf.DEFAULT_COLLATION -> "UTF8_LCASE") {
-            val resultUTF8Lcase = sql(query)
-            assert(resultUTF8.collect() === resultUTF8Lcase.collect())
+            val resultUTF8Lcase = sql(query).collect()
+            assert(resultUTF8 === resultUTF8Lcase)
           }
         } catch {
           case e: SparkRuntimeException => assert(e.getCondition == "USER_RAISED_EXCEPTION")


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org