You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2026/09/18 14:04:00 UTC

[jira] [Resolved] (SPARK-59289) Insert a co-partitioned child's GroupPartitionsExec once, where the join's merged keys are known

     [ https://issues.apache.org/jira/browse/SPARK-59289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun resolved SPARK-59289.
-----------------------------------
    Fix Version/s: 4.4.0
       Resolution: Fixed

Issue resolved by pull request 58659
[https://github.com/apache/spark/pull/58659]

> Insert a co-partitioned child's GroupPartitionsExec once, where the join's merged keys are known
> ------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-59289
>                 URL: https://issues.apache.org/jira/browse/SPARK-59289
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.4.0
>            Reporter: Peter Toth
>            Assignee: Peter Toth
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.4.0
>
>
> `EnsureRequirements`' first per-child loop wraps every child whose `KeyedPartitioning` is not grouped in a `GroupPartitionsExec`, and only afterwards does `checkKeyGroupCompatible` discover that the parent is a join and rewrite or replace that node. At the point the node is inserted the rule cannot yet tell a join child about to be re-aligned from a single-child operator that genuinely needs its input grouped, because `childrenIndexes` and `checkKeyGroupCompatible` run after it.
> The change is to decide a co-partitioned child's node once, at the site that already holds the join's merged keys, instead of inserting a placeholder and rewriting it.
> Two consequences beyond the tidying:
> * That site can ask whether the regrouping keeps the keyed claim *before* committing to the pairing, and fall back to a shuffle instead of leaving a plan that `ValidateRequirements` rejects. SPARK-59272 is that question, and [~ulysses] and I agreed on apache/spark#58339 that it belongs here rather than as a consumer-side guard.
> * It retires `rewriteGroupPartitions`' "drop what sits above" and `unwrapGroupPartitions`, both of which exist only because the placeholder has to be undone.
> A cheap version does not work, measured: normalizing the join children at the top of `checkKeyGroupCompatible` and planning from the raw children fails 44 of 126 `KeyGroupedPartitioningSuite` tests, because `createKeyedShuffleSpec` gates on `satisfies`, which gates on `isGrouped`, and the inserted node is what makes a non-grouped child satisfy. Admitting `keysMaySatisfy` there takes it down to 7 failures, which says the "child partitionings not modified" fast path and the merged-key and expected-count computation have to be reworked together with the gate.
> Master only.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org