[jira] [Updated] (SPARK-60382) CoalesceBucketsInJoin uses a broadcast join's build-side bucket count

["Yuming Wang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-60382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yuming Wang updated SPARK-60382:
--------------------------------
    Priority: Critical  (was: Major)

> CoalesceBucketsInJoin uses a broadcast join's build-side bucket count
> ---------------------------------------------------------------------
>
>                 Key: SPARK-60382
>                 URL: https://issues.apache.org/jira/browse/SPARK-60382
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Yuming Wang
>            Priority: Critical
>
> When spark.sql.bucketing.coalesceBucketsInJoin.enabled is true, a sort-merge join whose child is a BuildLeft broadcast hash join, broadcast nested-loop join, or broadcast range join can coalesce the streamed bucketed scan using the build side's bucket count.
> ExtractJoinWithBuckets.getBucketSpec uses collectFirst, which walks the left child first. For BuildLeft the left child is the broadcast build side, so a bucketed scan there is reported as the join input's bucket count. updateNumCoalescedBucketsInScan then stamps every bucketed scan in that subtree, including the streamed scan whose partitioning the parent join actually uses.
> Bucket files are grouped by the original bucket id modulo the coalesced count, while the scan advertises HashPartitioning of that count. If the streamed table's bucket count is not divisible by the build side's count, hash(key) % streamedBuckets and hash(key) % coalescedBuckets land in different partitions. The other side of the sort-merge join is a real table of the coalesced bucket count, EnsureRequirements inserts no shuffle, and an inner join drops matching rows. An outer join null-pads them.
> The count that reaches the parent join is the streamed side's. The lookup should follow the same streamed-side walk as hasScanOperation (Filter, Project, then the streamed child of a broadcast hash, nested-loop, or range join), and only that scan should be rewritten.
> The config defaults to false, so this does not affect plans unless bucket coalescing is enabled.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org