[jira] [Updated] (SPARK-59285) Hold a KeyedPartitioning's shared partition layout in one value

["Peter Toth (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Peter Toth updated SPARK-59285:
-------------------------------
    Description: 
{{KeyedPartitioning}} carries four things: the partition expressions, the partition key rows, whether those rows are unique ({{isGrouped}}), and whether one of them stands for several keys of a partitioning it was derived from ({{isCollapsed}}). Only the first is per instance. The members of a {{PartitioningCollection}} name one physical layout with their own expressions, so everything else is shared, and {{PartitioningCollection}} spends a clause per shared field to say so:

{code:scala}
require(rep.expressions.length == first.expressions.length, ...)
require(rep.partitionKeys eq first.partitionKeys, ...)
require(rep.isCollapsed == first.isCollapsed, ...)
{code}

{{isGrouped}} is not in that list, so two members over one key list can disagree about whether those keys are unique, and {{satisfies0}} is an {{exists}} over the members, so it would answer from whichever one won.

h3. Proposal

{{KeyLayout(partitionKeys, dataTypes, isGrouped, isCollapsed)}}, shared by reference, with {{KeyedPartitioning = (expressions, layout)}}:

* the collection's invariant becomes one {{eq}} on the layout plus the expressions' arity, whatever fields the layout later grows;
* {{fromPartitionings}} merges one canonical layout instead of interning the key reference and ORing a flag;
* {{KeyedShuffleSpec.createPartitioning}} has nothing to decide, since a copy that only replaces the expressions keeps the layout;
* the key types move onto the layout, so {{keyDataTypes}} is one read rather than a sample of the first key row with a fallback to the partition expressions. A partitioning whose partitions were all pruned then still answers for its key space, which is what SPARK-59176 needed an exception in {{EnsureRequirements}} for. That exception goes.

h3. Sequencing

Stacked on SPARK-59187, which erases the naming from the types the key rows are compared at. Without that the types are not shared: two sides of one join name the same key space differently.

It should also wait for SPARK-59050, which adds {{mayContainUnknownPartitionKeys}}, another field of the shared layout. Doing this first means doing it twice.

> Hold a KeyedPartitioning's shared partition layout in one value
> ---------------------------------------------------------------
>
>                 Key: SPARK-59285
>                 URL: https://issues.apache.org/jira/browse/SPARK-59285
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Peter Toth
>            Priority: Major
>
> {{KeyedPartitioning}} carries four things: the partition expressions, the partition key rows, whether those rows are unique ({{isGrouped}}), and whether one of them stands for several keys of a partitioning it was derived from ({{isCollapsed}}). Only the first is per instance. The members of a {{PartitioningCollection}} name one physical layout with their own expressions, so everything else is shared, and {{PartitioningCollection}} spends a clause per shared field to say so:
> {code:scala}
> require(rep.expressions.length == first.expressions.length, ...)
> require(rep.partitionKeys eq first.partitionKeys, ...)
> require(rep.isCollapsed == first.isCollapsed, ...)
> {code}
> {{isGrouped}} is not in that list, so two members over one key list can disagree about whether those keys are unique, and {{satisfies0}} is an {{exists}} over the members, so it would answer from whichever one won.
> h3. Proposal
> {{KeyLayout(partitionKeys, dataTypes, isGrouped, isCollapsed)}}, shared by reference, with {{KeyedPartitioning = (expressions, layout)}}:
> * the collection's invariant becomes one {{eq}} on the layout plus the expressions' arity, whatever fields the layout later grows;
> * {{fromPartitionings}} merges one canonical layout instead of interning the key reference and ORing a flag;
> * {{KeyedShuffleSpec.createPartitioning}} has nothing to decide, since a copy that only replaces the expressions keeps the layout;
> * the key types move onto the layout, so {{keyDataTypes}} is one read rather than a sample of the first key row with a fallback to the partition expressions. A partitioning whose partitions were all pruned then still answers for its key space, which is what SPARK-59176 needed an exception in {{EnsureRequirements}} for. That exception goes.
> h3. Sequencing
> Stacked on SPARK-59187, which erases the naming from the types the key rows are compared at. Without that the types are not shared: two sides of one join name the same key space differently.
> It should also wait for SPARK-59050, which adds {{mayContainUnknownPartitionKeys}}, another field of the shared layout. Doing this first means doing it twice.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org