You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2020/07/07 23:48:00 UTC
[jira] [Assigned] (SPARK-32216) Remove redundant ProjectExec
[ https://issues.apache.org/jira/browse/SPARK-32216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-32216:
------------------------------------
Assignee: Apache Spark
> Remove redundant ProjectExec
> ----------------------------
>
> Key: SPARK-32216
> URL: https://issues.apache.org/jira/browse/SPARK-32216
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Allison Wang
> Assignee: Apache Spark
> Priority: Major
>
> Currently Spark executed plan can have redundant `ProjectExec` node. For example:
> After Filter:
> {code:java}
> == Physical Plan ==
> *(1) Project [a#14L, b#15L, c#16, key#17]
> +- *(1) Filter (isnotnull(a#14L) AND (a#14L > 5))
> +- *(1) ColumnarToRow
> +- FileScan parquet [a#14L,b#15L,c#16,key#17] {code}
> The `Project [a#14L, b#15L, c#16, key#17]` is redundant because its output is exactly the same as filter's output.
> Before Aggregate:
> {code:java}
> == Physical Plan ==
> *(2) HashAggregate(keys=[key#17], functions=[sum(a#14L), last(b#15L, false)], output=[sum_a#39L, key#17, last_b#41L])
> +- Exchange hashpartitioning(key#17, 5), true, [id=#77]
> +- *(1) HashAggregate(keys=[key#17], functions=[partial_sum(a#14L), partial_last(b#15L, false)], output=[key#17, sum#49L, last#50L, valueSet#51])
> +- *(1) Project [key#17, a#14L, b#15L]
> +- *(1) Filter (isnotnull(a#14L) AND (a#14L > 100))
> +- *(1) ColumnarToRow
> +- FileScan parquet [a#14L,b#15L,key#17] {code}
> The `Project [key#17, a#14L, b#15L]` is redundant because hash aggregate doesn't require child plan's output to be in a specific order.
>
> In general, a project is redundant when
> # It has the same output attributes and order as its child's output when ordering of these attributes is required.
> # It has the same output attributes as its child's output when attribute output ordering is not required.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org