You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "naveenp2708 (via GitHub)" <gi...@apache.org> on 2026/03/11 03:21:36 UTC
[PR] [SPARK-55848][SQL][4.1] Fix incorrect dedup results with SPJ partial clustering [spark]
naveenp2708 opened a new pull request, #54749:
URL: https://github.com/apache/spark/pull/54749
### What changes were proposed in this pull request?
Backport fix for SPARK-55848 to branch-4.1. This branch does not have the KeyGroupedPartitioning refactor (#54330) from master.
The fix adds an `isPartiallyClustered` flag to `KeyGroupedPartitioning` and restructures `satisfies0()` to check `ClusteredDistribution` first, returning `false` when partially clustered. `EnsureRequirements` then inserts the necessary Exchange.
### Why are the changes needed?
SPJ with partial clustering produces incorrect results for post-join dedup operations (dropDuplicates, Window row_number). The partially-clustered partitioning is incorrectly treated as satisfying `ClusteredDistribution`, so no Exchange is inserted before dedup operators.
### Does this PR introduce any user-facing change?
Yes. Queries using SPJ with partial clustering followed by dedup operations will now return correct results.
### How was this patch tested?
Three regression tests added to KeyGroupedPartitioningSuite with data correctness checks and plan assertions verifying shuffle Exchange presence. All 95 tests pass.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55848][SQL][4.1] Fix incorrect dedup results with SPJ partial clustering [spark]
Posted by "naveenp2708 (via GitHub)" <gi...@apache.org>.
naveenp2708 closed pull request #54749: [SPARK-55848][SQL][4.1] Fix incorrect dedup results with SPJ partial clustering
URL: https://github.com/apache/spark/pull/54749
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org