You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wan Kun (Jira)" <ji...@apache.org> on 2022/12/02 14:49:00 UTC
[jira] [Updated] (SPARK-41368) Reorder the window partition expressions by expression stats
[ https://issues.apache.org/jira/browse/SPARK-41368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wan Kun updated SPARK-41368:
----------------------------
Description:
We can reorder window partition expressions by the distinct values stats.
Sorting with high cardinality will be faster.
Benchmark result:
{code:java}
Window: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------
Disable reorder window partition expressions 13338 14035 1138 0.0 1333756.2 1.0X
Enable reorder window partition expressions 5981 6030 77 0.0 598102.1 2.2X
{code}
was:
We can reorder window partition expressions by the distinct values stats.
Sorting with high cardinality will be faster.
Benchmark result:
{code:java}
Window: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------
Without reorder window partition expressions 13338 14035 1138 0.0 1333756.2 1.0X
With reorder window partition expressions 5981 6030 77 0.0 598102.1 2.2X
{code}
> Reorder the window partition expressions by expression stats
> ------------------------------------------------------------
>
> Key: SPARK-41368
> URL: https://issues.apache.org/jira/browse/SPARK-41368
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Wan Kun
> Priority: Major
>
> We can reorder window partition expressions by the distinct values stats.
> Sorting with high cardinality will be faster.
> Benchmark result:
> {code:java}
> Window: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
> ----------------------------------------------------------------------------------------------------------------------------
> Disable reorder window partition expressions 13338 14035 1138 0.0 1333756.2 1.0X
> Enable reorder window partition expressions 5981 6030 77 0.0 598102.1 2.2X
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org