[jira] [Updated] (SPARK-60277) Preserve keyed partitioning through unions with unknown keys

["XiDuo You (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-60277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XiDuo You updated SPARK-60277:
------------------------------
    Summary: Preserve keyed partitioning through unions with unknown keys  (was: SPARK-60277)

> Preserve keyed partitioning through unions with unknown keys
> ------------------------------------------------------------
>
>                 Key: SPARK-60277
>                 URL: https://issues.apache.org/jira/browse/SPARK-60277
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: XiDuo You
>            Priority: Major
>              Labels: pull-request-available
>
> A `UnionExec` whose children all report a `KeyedPartitioning` merges their keys into one
> layout, so a storage-partitioned join or a keyed aggregate above it needs no exchange. Today
> that merge is given up whenever one child's layout carries
> `mayContainUnknownPartitionKeys`: such a child routes a row whose key it does not declare to
> the partition of some declared key, and another child may declare exactly that key, so the
> merged claim could promise co-location the marked child cannot honor.
> The refusal is wider than it has to be. When the marked child declares every key the other
> children declare, no child can claim a key the marked one routes by hash, and the keys no child
> declares reach the same declared key from every marked child as long as they declare the same
> keys in the same order. Each key's rows then share a group, which is what a clustering asks of
> the merged layout, and the merge can be kept with the marker carried onto it so pairing keeps
> reading it.
> This matters for a plan whose union leg comes out of a one-side-shuffled join: the leg is marked
> although its keys are accurate for every key the other legs hold, and the operator above the
> union pays an exchange it does not need.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org