[jira] [Commented] (SPARK-59272) Revalidate the storage-partitioned join pairing when a regrouped side drops the keyed claim

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-59272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116163#comment-18116163 ] 

Dongjoon Hyun commented on SPARK-59272:
---------------------------------------

This is resolved back via the follow-up.
- https://github.com/apache/spark/pull/58858

> Revalidate the storage-partitioned join pairing when a regrouped side drops the keyed claim
> -------------------------------------------------------------------------------------------
>
>                 Key: SPARK-59272
>                 URL: https://issues.apache.org/jira/browse/SPARK-59272
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.4.0
>            Reporter: XiDuo You
>            Assignee: Peter Toth
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.4.0
>
>
> ## Problem
> In a storage-partitioned join (SPJ), `GroupPartitionsExec` deliberately gives up a marked layout's keyed claim - reporting `UnknownPartitioning` - when the regrouping is not the identity: a marked side realigned onto differently-ordered merged keys, or a grouping that applies a reducer or a non-identity projection (`identityGrouping` now requires `keysRewritten` to be false). This is the intended response to a claim the regrouping invalidates, and results stay correct because the give-up forces downstream consumers to stop trusting the layout.
> The gap is that the give-up is evaluated inside `GroupPartitionsExec.outputPartitioning`, which `checkKeyGroupCompatible` reaches through `applyGroupPartitions` after it has decided the pairing is compatible and before it commits it. `EnsureRequirements` never re-checks the rebuilt children, so a committed join can carry a child that no longer satisfies its required distribution. `ValidateRequirements` then rejects the stage, vetoing AQE rules (partition coalescing, local read, skew join) on it. The regrouping test on SPARK-59050 exercises this state today. No wrong results are known on the current head; the cost is planning quality and lost AQE optimizations.
> ## Why a plain recheck is wrong
> The direct fix - re-check `newLeft.outputPartitioning.satisfies(distribution)` after `applyGroupPartitions` - is not viable: partial clustering deliberately produces ungrouped yet value-aligned children, both aligned onto one `mergedPartitionKeys` by `alignToExpectedKeys`. A `KeyedPartitioning` only satisfies a `ClusteredDistribution` when grouped, so a `satisfies` gate rejects the whole partial-clustering family (measured: the partial-clustering suites regress).
> ## Proposed direction
> Add a pairwise co-location check at the pairing site: whether the two rebuilt children still declare the same aligned partition key sequence (duplicates and order included) - the guarantee `alignToExpectedKeys` constructs - and whether either side gave up its keyed layout. This should be reconciled with `ValidateRequirements`' semantics, which reason per side from `satisfies` and cannot express the constructed twin spread.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org