You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Chao Sun (Jira)" <ji...@apache.org> on 2024/04/30 04:27:00 UTC
[jira] [Resolved] (SPARK-48030) InternalRowComparableWrapper should cache rowOrdering to improve performace
[ https://issues.apache.org/jira/browse/SPARK-48030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chao Sun resolved SPARK-48030.
------------------------------
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 46265
[https://github.com/apache/spark/pull/46265]
> InternalRowComparableWrapper should cache rowOrdering to improve performace
> ---------------------------------------------------------------------------
>
> Key: SPARK-48030
> URL: https://issues.apache.org/jira/browse/SPARK-48030
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.5.1, 3.4.3
> Reporter: YE
> Assignee: YE
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: screenshot-1.png
>
>
> InternalRowComparableWrapper recreates row ordering for each output partition when SPJ is enabled. The row ordering is generated via codegen which is quite expensive and the output partitions might be quite large for production table such as hundreds of thousands partitions. We encountered this issue when applying SPJ with multiple large Iceberg tables and the plan phase took tens of minutes to complete.
> Attaching a screenshot to provide related stack trace:
> !screenshot-1.png!
> A simple fix for this would be caching the rowOrdering for InternalRowComparableWrapper as the datatype of the InternalRow is immutable
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org