You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "zhuml (Jira)" <ji...@apache.org> on 2023/02/27 02:48:00 UTC
[jira] [Updated] (SPARK-42588) collapse two adjacent windows with the equivalent partition/order expression
[ https://issues.apache.org/jira/browse/SPARK-42588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhuml updated SPARK-42588:
--------------------------
Description:
Extend the CollapseWindow rule to collapse Window nodes with the equivalent partition/order expressions
{code:java}
Seq((1, 1), (2, 2)).toDF("a", "b")
.withColumn("max_b", expr("max(b) OVER (PARTITION BY abs(a))"))
.withColumn("min_b", expr("min(b) OVER (PARTITION BY abs(a))"))
== Optimized Logical Plan ==
before
Project [a#7, b#8, max_b#11, min_b#17]
+- Window [min(b#8) windowspecdefinition(_w0#19, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS min_b#17], [_w0#19]
+- Project [a#7, b#8, max_b#11, abs(a#7) AS _w0#19]
+- Window [max(b#8) windowspecdefinition(_w0#13, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS max_b#11], [_w0#13]
+- Project [_1#2 AS a#7, _2#3 AS b#8, abs(_1#2) AS _w0#13]
+- LocalRelation [_1#2, _2#3]
after
Project [a#7, b#8, max_b#11, min_b#17]
+- Window [max(b#8) windowspecdefinition(_w0#13, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS max_b#11, min(b#8) windowspecdefinition(_w0#13, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS min_b#17], [_w0#13]
+- Project [_1#2 AS a#7, _2#3 AS b#8, abs(_1#2) AS _w0#13]
+- LocalRelation [_1#2, _2#3]
{code}
> collapse two adjacent windows with the equivalent partition/order expression
> ----------------------------------------------------------------------------
>
> Key: SPARK-42588
> URL: https://issues.apache.org/jira/browse/SPARK-42588
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.2.3, 3.3.2
> Reporter: zhuml
> Priority: Major
>
> Extend the CollapseWindow rule to collapse Window nodes with the equivalent partition/order expressions
> {code:java}
> Seq((1, 1), (2, 2)).toDF("a", "b")
> .withColumn("max_b", expr("max(b) OVER (PARTITION BY abs(a))"))
> .withColumn("min_b", expr("min(b) OVER (PARTITION BY abs(a))"))
> == Optimized Logical Plan ==
> before
> Project [a#7, b#8, max_b#11, min_b#17]
> +- Window [min(b#8) windowspecdefinition(_w0#19, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS min_b#17], [_w0#19]
> +- Project [a#7, b#8, max_b#11, abs(a#7) AS _w0#19]
> +- Window [max(b#8) windowspecdefinition(_w0#13, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS max_b#11], [_w0#13]
> +- Project [_1#2 AS a#7, _2#3 AS b#8, abs(_1#2) AS _w0#13]
> +- LocalRelation [_1#2, _2#3]
> after
> Project [a#7, b#8, max_b#11, min_b#17]
> +- Window [max(b#8) windowspecdefinition(_w0#13, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS max_b#11, min(b#8) windowspecdefinition(_w0#13, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS min_b#17], [_w0#13]
> +- Project [_1#2 AS a#7, _2#3 AS b#8, abs(_1#2) AS _w0#13]
> +- LocalRelation [_1#2, _2#3]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org