You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "XiDuo You (Jira)" <ji...@apache.org> on 2026/08/25 07:44:00 UTC
[jira] [Updated] (SPARK-58988) Fix PartitioningCollection invariant violation exception when allowKeysSubsetOfPartitionKeys
[ https://issues.apache.org/jira/browse/SPARK-58988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
XiDuo You updated SPARK-58988:
------------------------------
Summary: Fix PartitioningCollection invariant violation exception when allowKeysSubsetOfPartitionKeys (was: Fix PartitioningCollection invariant violation in subset-key storage-partitioned join)
> Fix PartitioningCollection invariant violation exception when allowKeysSubsetOfPartitionKeys
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-58988
> URL: https://issues.apache.org/jira/browse/SPARK-58988
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: XiDuo You
> Priority: Major
>
> This is a follow-up to SPARK-56877, which added a check in `PartitioningCollection.fromPartitionings` requiring all `KeyedPartitioning`s to share equal `partitionKeys`.
> When a storage-partitioned join has join keys that are a subset of the partition keys (e.g. a v2 table partitioned by `[dt, bucket(16, c1)]` joined on `c1` only), and `spark.sql.sources.v2.bucketing.shuffle.enabled` is on so only the non-keyed side is re-shuffled, `KeyedPartitioning.createShuffleSpec` projects the keyed side to the join keys but keeps them in first-occurrence order (`distinct` without sorting). The keyed side, however, goes through `GroupPartitionsExec`, which sorts the projected keys. The two sides then carry the same keys in a different order, and `PartitioningCollection.fromPartitionings` throws:
> ```
> java.lang.IllegalArgumentException: requirement failed: All KeyedPartitionings in a PartitioningCollection must have equal partitionKeys
> ```
> The fix sorts the distinct projected keys in `createShuffleSpec` (using `toGrouped`, which sorts by the same natural ascending ordering as `GroupPartitionsExec`), aligning the two sides.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org