(spark) branch branch-4.1 updated: [SPARK-52246][SQL][TEST] Add bucket transform regression test for one-side shuffle with join key tail of partition keys

[pt...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

peter-toth pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.1 by this push:
     new 030e87897492 [SPARK-52246][SQL][TEST] Add bucket transform regression test for one-side shuffle with join key tail of partition keys
030e87897492 is described below

commit 030e87897492d50e6ec9e95e84017dd2a4d6ed6c
Author: naveenp2708 <na...@gmail.com>
AuthorDate: Sun Jul 26 10:16:06 2026 +0200

    [SPARK-52246][SQL][TEST] Add bucket transform regression test for one-side shuffle with join key tail of partition keys
    
    ### What changes were proposed in this pull request?
    
    Add a regression test to `KeyGroupedPartitioningSuite` for the scenario reported in [SPARK-52246](https://issues.apache.org/jira/browse/SPARK-52246) (test adapted from the reproduction in the JIRA description): a storage-partitioned join where one side is auto-shuffled, the partitioned side uses a bucket transform, and the partition key is the tail of the join keys.
    
    ### Why are the changes needed?
    
    SPARK-52246 reported silent incorrect results (empty join output) for this scenario on 4.0.0/4.0.1. The defect was fixed by SPARK-54439, but the tests added there cover identity and years transforms only, not bucket. This locks in coverage for the bucket case.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    New test passes on master; verified it fails on v4.0.1 (pre-SPARK-54439), reproducing the original wrong results.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #57442 from naveenp2708/repro-SPARK-52246.
    
    Authored-by: naveenp2708 <na...@gmail.com>
    Signed-off-by: Peter Toth <pe...@gmail.com>
    (cherry picked from commit d2693b5baf5cd7c449a521e93564fac4b296a167)
    Signed-off-by: Peter Toth <pe...@gmail.com>
---
 .../connector/KeyGroupedPartitioningSuite.scala    | 26 ++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
index 122c511bf835..e8d51dedcda3 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
@@ -2438,6 +2438,32 @@ class KeyGroupedPartitioningSuite extends DistributionAndOrderingSuiteBase {
    }
   }
 
+  test("SPARK-52246: one-side shuffle with join key tail part of the partition keys") {
+    val items_partitions = Array(bucket(2, "id"))
+    createTable(items, itemsColumns, items_partitions)
+
+    sql(s"INSERT INTO testcat.ns.$items VALUES " +
+      "(1, 'aa', 40.0, cast('2020-01-01' as timestamp)), " +
+      "(1, 'aa', 30.0, cast('2020-01-02' as timestamp)), " +
+      "(3, 'bb', 10.0, cast('2020-01-01' as timestamp)), " +
+      "(4, 'cc', 15.5, cast('2020-02-01' as timestamp))")
+
+    createTable(purchases, purchasesColumns, Array.empty)
+    sql(s"INSERT INTO testcat.ns.$purchases VALUES " +
+      "(1, 42.0, cast('2020-01-01' as timestamp)), " +
+      "(1, 89.0, cast('2020-01-03' as timestamp)), " +
+      "(3, 19.5, cast('2020-02-01' as timestamp)), " +
+      "(5, 26.0, cast('2023-01-01' as timestamp)), " +
+      "(6, 50.0, cast('2023-02-01' as timestamp))")
+
+    withSQLConf(SQLConf.V2_BUCKETING_SHUFFLE_ENABLED.key -> "true") {
+      val df = createJoinTestDF(Seq("arrive_time" -> "time", "id" -> "item_id"))
+      val shuffles = collectShuffles(df.queryExecution.executedPlan)
+      assert(shuffles.size == 1, "SPJ should be triggered")
+      checkAnswer(df, Seq(Row(1, "aa", 40.0, 42.0)))
+    }
+  }
+
   test("SPARK-48949: test partition filters inner join") {
     val items_partitions = Array(bucket(8, "id"), days("arrive_time"))
     createTable(items, itemsColumns, items_partitions)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org