You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Peter Toth (Jira)" <ji...@apache.org> on 2026/09/02 20:36:00 UTC
[jira] [Commented] (SPARK-59187) Carry the partition key data types on KeyedPartitioning
[ https://issues.apache.org/jira/browse/SPARK-59187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18110865#comment-18110865 ]
Peter Toth commented on SPARK-59187:
------------------------------------
Working on it.
> Carry the partition key data types on KeyedPartitioning
> -------------------------------------------------------
>
> Key: SPARK-59187
> URL: https://issues.apache.org/jira/browse/SPARK-59187
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Peter Toth
> Priority: Major
>
> {{KeyedPartitioning}} carries its partition keys as {{InternalRowComparableWrapper}} rows and no types. Every reader that needs the types samples them from the first key row, via {{keyDataTypes}}:
> {code:scala}
> @transient lazy val keyDataTypes: Seq[DataType] =
> partitionKeys.headOption.map(_.dataTypes).getOrElse(expressionDataTypes)
> {code}
> Two problems follow from that.
> * With no key row there is nothing to sample, so it falls back to the partition expressions' own types. Those describe the keys only while the expressions do, and after a join reduced both sides' keys they no longer do (SPARK-59121). SPARK-59176 was a query that failed because one caller held that fallback against a real answer.
> * Nothing checks that the remaining key rows agree with the first one.
> The types are known where the keys are built, and are already computed and dropped there:
> {code:scala}
> def apply(expressions: Seq[Expression], partitionKeys: Seq[InternalRow]): KeyedPartitioning = {
> val dataTypes = expressions.map(_.dataType)
> val comparableKeyWrapperFactory =
> InternalRowComparableWrapper.getInternalRowComparableWrapperFactory(dataTypes)
> ...
> {code}
> The proposal is to carry them on the partitioning beside {{expressions}}, and to have {{projectKeys}}, {{reduceKeys}}, {{KeyedShuffleSpec.createPartitioning}} and {{GroupPartitionsExec}} carry them through the transformations they apply to the keys. Then no reader samples a key row, and a partitioning with no key still reports what its keys would hold.
> This subsumes SPARK-59176's fix, which leaves a keyless side out of one comparison rather than giving it an answer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org