You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Peter Toth (Jira)" <ji...@apache.org> on 2026/09/02 20:36:00 UTC

[jira] [Commented] (SPARK-59187) Carry the partition key data types on KeyedPartitioning

    [ https://issues.apache.org/jira/browse/SPARK-59187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18110865#comment-18110865 ] 

Peter Toth commented on SPARK-59187:
------------------------------------

Working on it.

> Carry the partition key data types on KeyedPartitioning
> -------------------------------------------------------
>
>                 Key: SPARK-59187
>                 URL: https://issues.apache.org/jira/browse/SPARK-59187
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Peter Toth
>            Priority: Major
>
> {{KeyedPartitioning}} carries its partition keys as {{InternalRowComparableWrapper}} rows and no types. Every reader that needs the types samples them from the first key row, via {{keyDataTypes}}:
> {code:scala}
> @transient lazy val keyDataTypes: Seq[DataType] =
>   partitionKeys.headOption.map(_.dataTypes).getOrElse(expressionDataTypes)
> {code}
> Two problems follow from that.
> * With no key row there is nothing to sample, so it falls back to the partition expressions' own types. Those describe the keys only while the expressions do, and after a join reduced both sides' keys they no longer do (SPARK-59121). SPARK-59176 was a query that failed because one caller held that fallback against a real answer.
> * Nothing checks that the remaining key rows agree with the first one.
> The types are known where the keys are built, and are already computed and dropped there:
> {code:scala}
> def apply(expressions: Seq[Expression], partitionKeys: Seq[InternalRow]): KeyedPartitioning = {
>   val dataTypes = expressions.map(_.dataType)
>   val comparableKeyWrapperFactory =
>     InternalRowComparableWrapper.getInternalRowComparableWrapperFactory(dataTypes)
>   ...
> {code}
> The proposal is to carry them on the partitioning beside {{expressions}}, and to have {{projectKeys}}, {{reduceKeys}}, {{KeyedShuffleSpec.createPartitioning}} and {{GroupPartitionsExec}} carry them through the transformations they apply to the keys. Then no reader samples a key row, and a partitioning with no key still reports what its keys would hold.
> This subsumes SPARK-59176's fix, which leaves a keyless side out of one comparison rather than giving it an answer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org