[jira] [Updated] (SPARK-59249) Take the grouped key-row ordering from the shared InternalRowComparableWrapper cache

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-59249:
-----------------------------------
    Labels: pull-request-available  (was: )

> Take the grouped key-row ordering from the shared InternalRowComparableWrapper cache
> ------------------------------------------------------------------------------------
>
>                 Key: SPARK-59249
>                 URL: https://issues.apache.org/jira/browse/SPARK-59249
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Peter Toth
>            Priority: Major
>              Labels: pull-request-available
>
> `KeyedPartitioning.groupedKeyRowOrdering` builds the ordering that grouped partition keys are laid out by:
> {code:scala}
> def groupedKeyRowOrdering(dataTypes: Seq[DataType]): BaseOrdering =
>   RowOrdering.createNaturalAscendingOrdering(dataTypes)
> {code}
> That call is byte-for-byte the {{loadFunc}} of {{InternalRowComparableWrapper}}'s private {{orderingCache}}, a 1024-entry {{NonFateSharingCache}}:
> {code:scala}
> private val orderingCache = {
>   val loadFunc = (dataTypes: Seq[DataType]) => {
>     RowOrdering.createNaturalAscendingOrdering(dataTypes)
>   }
>   NonFateSharingCache(loadFunc, MAX_CACHE_ENTRIES)
> }
> {code}
> So every call regenerates an ordering the cache already holds, and that every partition key wrapper already exposes as {{ordering}}. {{GenerateOrdering}} has its own code cache, so nothing is recompiled, but the {{SortOrder}} list and the generated instance are rebuilt each time. The callers are {{KeyedPartitioning.keyRowOrdering}}, {{GroupPartitionsExec.groupAndSortByKeys}} and the reduced-key ordering in {{EnsureRequirements}}.
> The more useful half is not the cost. {{InternalRowComparableWrapper.equals}} is
> {code:scala}
> ordering.compare(row, otherWrapper.row) == 0
> {code}
> over that same ordering, so "two partition keys are equal" and "two partition keys sort together" already come from one definition, written down in two places. Making them share it explicitly is what this ticket is about.
> The proposal is a public {{InternalRowComparableWrapper.orderingFor(dataTypes)}} that returns the cached ordering, with {{groupedKeyRowOrdering}} delegating to it.
> {{DataSourceV2ScanExecBase.outputPartitioning}} is a second site. It calls {{RowOrdering.createNaturalAscendingOrdering}} directly to sort the partition keys it is about to hand to {{KeyedPartitioning}}, which is exactly the grouped key layout order, so it should name that contract rather than rebuild it.
> No behaviour change.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org