You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2016/09/24 00:17:20 UTC
[jira] [Updated] (SPARK-17653) Optimizer should remove unnecessary
distincts (in multiple unions)
[ https://issues.apache.org/jira/browse/SPARK-17653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reynold Xin updated SPARK-17653:
--------------------------------
Description:
Query:
{code}
select 1 a union select 2 b union select 3 c
{code}
Explain plan:
{code}
== Physical Plan ==
*HashAggregate(keys=[a#13], functions=[])
+- Exchange hashpartitioning(a#13, 200)
+- *HashAggregate(keys=[a#13], functions=[])
+- Union
:- *HashAggregate(keys=[a#13], functions=[])
: +- Exchange hashpartitioning(a#13, 200)
: +- *HashAggregate(keys=[a#13], functions=[])
: +- Union
: :- *Project [1 AS a#13]
: : +- Scan OneRowRelation[]
: +- *Project [2 AS b#14]
: +- Scan OneRowRelation[]
+- *Project [3 AS c#15]
+- Scan OneRowRelation[]
{code}
Only one distinct should be necessary. This makes a bunch of unions slower than a bunch of union alls followed by a distinct.
was:
Query:
{code}
select 1 a union select 2 b union select 3 c
{code}
Explain plan:
{code}
== Physical Plan ==
*HashAggregate(keys=[a#13], functions=[])
+- Exchange hashpartitioning(a#13, 200)
+- *HashAggregate(keys=[a#13], functions=[])
+- Union
:- *HashAggregate(keys=[a#13], functions=[])
: +- Exchange hashpartitioning(a#13, 200)
: +- *HashAggregate(keys=[a#13], functions=[])
: +- Union
: :- *Project [1 AS a#13]
: : +- Scan OneRowRelation[]
: +- *Project [2 AS b#14]
: +- Scan OneRowRelation[]
+- *Project [3 AS c#15]
+- Scan OneRowRelation[]
{code}
Only one distinct should be necessary.
> Optimizer should remove unnecessary distincts (in multiple unions)
> ------------------------------------------------------------------
>
> Key: SPARK-17653
> URL: https://issues.apache.org/jira/browse/SPARK-17653
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Reynold Xin
>
> Query:
> {code}
> select 1 a union select 2 b union select 3 c
> {code}
> Explain plan:
> {code}
> == Physical Plan ==
> *HashAggregate(keys=[a#13], functions=[])
> +- Exchange hashpartitioning(a#13, 200)
> +- *HashAggregate(keys=[a#13], functions=[])
> +- Union
> :- *HashAggregate(keys=[a#13], functions=[])
> : +- Exchange hashpartitioning(a#13, 200)
> : +- *HashAggregate(keys=[a#13], functions=[])
> : +- Union
> : :- *Project [1 AS a#13]
> : : +- Scan OneRowRelation[]
> : +- *Project [2 AS b#14]
> : +- Scan OneRowRelation[]
> +- *Project [3 AS c#15]
> +- Scan OneRowRelation[]
> {code}
> Only one distinct should be necessary. This makes a bunch of unions slower than a bunch of union alls followed by a distinct.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org