You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2025/08/26 20:09:41 UTC
(spark) branch branch-4.0 updated: [SPARK-52873][SQL][TESTS][FOLLOWUP] Fix test for non-ansi mode
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 83d5ff15f544 [SPARK-52873][SQL][TESTS][FOLLOWUP] Fix test for non-ansi mode
83d5ff15f544 is described below
commit 83d5ff15f544541166d6049d5f7414d5edd5ce05
Author: Bruce Robbins <be...@gmail.com>
AuthorDate: Tue Aug 26 13:05:24 2025 -0700
[SPARK-52873][SQL][TESTS][FOLLOWUP] Fix test for non-ansi mode
### What changes were proposed in this pull request?
In the `JoinSuite` test expecting `ignoreDuplicateKey=true`, don't include a query where a build-side key is a string used in a numeric expression.
### Why are the changes needed?
In non-ansi mode, casting of the string `t2.c1` to use in a numeric expression (`t2.c1 * 1000`) adds extra scaffolding around that expression in the build keys, but not in the condition. In the condition, `t2.c1 * 1000` is
`(cast(c1#299 as double) * 1000.0`
but in the build key, it is
`knownfloatingpointnormalized(normalizenanandzero((cast(c1#299 as double) * 1000.0)`
As a result, `t2.c1 * 1000` doesn't match between the build keys and the condition. Therefore, the optimization is not performed, and since the test checks for the optimization, the test fails in non-ansi mode.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manually ran the offending test both with and without `SPARK_ANSI_SQL_MODE=false`.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #52128 from bersprockets/broken_nonansi_test.
Authored-by: Bruce Robbins <be...@gmail.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit 20a6af7d00e567415d6f1c1ac73441d06bf602d5)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../src/test/scala/org/apache/spark/sql/JoinSuite.scala | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
index 2f27ac5c722a..aaac0ebc9aa0 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
@@ -1559,10 +1559,13 @@ class JoinSuite extends QueryTest with SharedSparkSession with AdaptiveSparkPlan
spark.range(10).map(i => (i.toString, i + 1)).toDF("c1", "c2").write.saveAsTable("t1")
spark.range(10).map(i => ((i % 5).toString, i % 3)).toDF("c1", "c2").write.saveAsTable("t2")
+ spark.range(10).map(i => (i, i + 1)).toDF("c1", "c2").write.saveAsTable("t1a")
+ spark.range(10).map(i => (i % 5, i % 3)).toDF("c1", "c2").write.saveAsTable("t2a")
+
val semiExpected1 = Seq(Row("0"), Row("1"), Row("2"), Row("3"), Row("4"))
val antiExpected1 = Seq(Row("5"), Row("6"), Row("7"), Row("8"), Row("9"))
- val semiExpected2 = Seq(Row("0"))
- val antiExpected2 = Seq.tabulate(9) { x => Row((x + 1).toString) }
+ val semiExpected2 = Seq(Row(0))
+ val antiExpected2 = Seq.tabulate(9) { x => Row(x + 1) }
val semiJoinQueries = Seq(
// No join condition, ignore duplicated key.
@@ -1587,9 +1590,9 @@ class JoinSuite extends QueryTest with SharedSparkSession with AdaptiveSparkPlan
// the same as t2.c2). In this case, ignoreDuplicatedKey should be false
(
s"""
- |SELECT /*+ SHUFFLE_HASH(t2) */ t1.c1 FROM t1 LEFT SEMI JOIN t2
- |ON CAST((t1.c2+10000)/1000 AS INT) = CAST((t2.c2+10000)/1000 AS INT)
- |AND t2.c2 >= t1.c2 + 1
+ |SELECT /*+ SHUFFLE_HASH(t2a) */ t1a.c1 FROM t1a LEFT SEMI JOIN t2a
+ |ON CAST((t1a.c2+10000)/1000 AS INT) = CAST((t2a.c2+10000)/1000 AS INT)
+ |AND t2a.c2 >= t1a.c2 + 1
|""".stripMargin,
false, semiExpected2, antiExpected2),
// SPARK-52873: Have a join condition that contains the same expression as the
@@ -1597,8 +1600,8 @@ class JoinSuite extends QueryTest with SharedSparkSession with AdaptiveSparkPlan
// In this case, ignoreDuplicatedKey should be true
(
s"""
- |SELECT /*+ SHUFFLE_HASH(t2) */ t1.c1 FROM t1 LEFT SEMI JOIN t2
- |ON t1.c1 * 10000 = t2.c1 * 1000 AND t2.c1 * 1000 >= t1.c1
+ |SELECT /*+ SHUFFLE_HASH(t2a) */ t1a.c1 FROM t1a LEFT SEMI JOIN t2a
+ |ON t1a.c1 * 10000 = t2a.c1 * 1000 AND t2a.c1 * 1000 >= t1a.c1
|""".stripMargin,
true, semiExpected2, antiExpected2)
)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org