[jira] [Updated] (SPARK-15076) Add ReorderAssociativeOperator optimizer

["Wenchen Fan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan updated SPARK-15076:
--------------------------------
    Assignee: Dongjoon Hyun

> Add ReorderAssociativeOperator optimizer
> ----------------------------------------
>
>                 Key: SPARK-15076
>                 URL: https://issues.apache.org/jira/browse/SPARK-15076
>             Project: Spark
>          Issue Type: Improvement
>          Components: Optimizer
>            Reporter: Dongjoon Hyun
>            Assignee: Dongjoon Hyun
>             Fix For: 2.1.0
>
>
> This issue add a new optimizer  `ReorderAssociativeOperator` by taking advantage of integral associative property. Currently, Spark works like the following.
> 1) Can optimize `1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + a` into `45 + a`.
> 2) Cannot optimize `a + 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9`.
> This issue can handle Case 2 for *Add/Multiply* expression whose data types are `ByteType`, `ShortType`, `IntegerType`, and `LongType`. The followings are the plan comparison between `before` and `after` this issue.
> **Before**
> {code}
> scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1)) a)").explain
> == Physical Plan ==
> WholeStageCodegen
> :  +- Project [(((((((((a#7 + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9) AS (((((((((a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9)#8]
> :     +- INPUT
> +- Generate explode([1]), false, false, [a#7]
>    +- Scan OneRowRelation[]
> {code}
> **After**
> {code}
> scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1)) a)").explain
> == Physical Plan ==
> WholeStageCodegen
> :  +- Project [(a#7 + 45) AS (((((((((a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9)#8]
> :     +- INPUT
> +- Generate explode([1]), false, false, [a#7]
>    +- Scan OneRowRelation[]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org