You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2016/04/28 21:55:12 UTC
[jira] [Updated] (SPARK-14939) Improve EliminateSorts optimizer to
handle Ordinal/Alias SortOrders
[ https://issues.apache.org/jira/browse/SPARK-14939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-14939:
----------------------------------
Description:
This issue aims to improve `EliminateSorts` optimizer to handle ordinal (case 2) and alias (case 3). Case 1 is the current implementation.
1. Literals and foldable expression, e.g. "ORDER BY 1.0, 'abc', Now()"
2. Foldable ordinals, e.g. "SELECT 1.0, 'abc', Now() ORDER BY 1, 2, 3"
3. Foldable aliases, e.g. "SELECT 1.0 x, 'abc' y, Now() z ORDER BY x, y, z"
**Before**
{code}
scala> sql("SELECT 1.0, Now() x ORDER BY 1, x").explain
== Physical Plan ==
WholeStageCodegen
: +- Sort [1.0#5 ASC,x#0 ASC], true, 0
: +- INPUT
+- Exchange rangepartitioning(1.0#5 ASC, x#0 ASC, 200), None
+- WholeStageCodegen
: +- Project [1.0 AS 1.0#5,1461873043577000 AS x#0]
: +- INPUT
+- Scan OneRowRelation[]
{code}
**After**
{code}
scala> sql("SELECT 1.0, Now() x ORDER BY 1, x").explain
== Physical Plan ==
WholeStageCodegen
: +- Project [1.0 AS 1.0#5,1461873079484000 AS x#0]
: +- INPUT
+- Scan OneRowRelation[]
{code}
was:
This issue aims to delete No-op **Alias** SortOrders in Sort. Currently, `EliminateSorts` removes direct foldable expressions, but cannot handle aliases.
**Before**
{code}
scala> sql("select a, year(current_date()) as y from (select explode(array(1,2)) a) order by y").explain()
== Physical Plan ==
WholeStageCodegen
: +- Sort [y#8 ASC], true, 0
: +- INPUT
+- Exchange rangepartitioning(y#8 ASC, 200), None
+- WholeStageCodegen
: +- Project [a#13,2016 AS y#8]
: +- INPUT
+- Generate explode([1,2]), false, false, [a#13]
+- Scan OneRowRelation[]
{code}
**After**
{code}
scala> sql("select a, year(current_date()) as y from (select explode(array(1,2)) a) order by y").explain()
== Physical Plan ==
WholeStageCodegen
: +- Project [a#13,2016 AS y#8]
: +- INPUT
+- Generate explode([1,2]), false, false, [a#13]
+- Scan OneRowRelation[]
{code}
Summary: Improve EliminateSorts optimizer to handle Ordinal/Alias SortOrders (was: Eliminate No-op Alias SortOrder in Sort)
> Improve EliminateSorts optimizer to handle Ordinal/Alias SortOrders
> -------------------------------------------------------------------
>
> Key: SPARK-14939
> URL: https://issues.apache.org/jira/browse/SPARK-14939
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Dongjoon Hyun
>
> This issue aims to improve `EliminateSorts` optimizer to handle ordinal (case 2) and alias (case 3). Case 1 is the current implementation.
> 1. Literals and foldable expression, e.g. "ORDER BY 1.0, 'abc', Now()"
> 2. Foldable ordinals, e.g. "SELECT 1.0, 'abc', Now() ORDER BY 1, 2, 3"
> 3. Foldable aliases, e.g. "SELECT 1.0 x, 'abc' y, Now() z ORDER BY x, y, z"
> **Before**
> {code}
> scala> sql("SELECT 1.0, Now() x ORDER BY 1, x").explain
> == Physical Plan ==
> WholeStageCodegen
> : +- Sort [1.0#5 ASC,x#0 ASC], true, 0
> : +- INPUT
> +- Exchange rangepartitioning(1.0#5 ASC, x#0 ASC, 200), None
> +- WholeStageCodegen
> : +- Project [1.0 AS 1.0#5,1461873043577000 AS x#0]
> : +- INPUT
> +- Scan OneRowRelation[]
> {code}
> **After**
> {code}
> scala> sql("SELECT 1.0, Now() x ORDER BY 1, x").explain
> == Physical Plan ==
> WholeStageCodegen
> : +- Project [1.0 AS 1.0#5,1461873079484000 AS x#0]
> : +- INPUT
> +- Scan OneRowRelation[]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org