[jira] [Commented] (SPARK-18395) Evaluate common subexpression like lazy variable with a function approach

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-18395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653351#comment-15653351 ] 

Apache Spark commented on SPARK-18395:
--------------------------------------

User 'viirya' has created a pull request for this issue:
https://github.com/apache/spark/pull/15837

> Evaluate common subexpression like lazy variable with a function approach
> -------------------------------------------------------------------------
>
>                 Key: SPARK-18395
>                 URL: https://issues.apache.org/jira/browse/SPARK-18395
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Liang-Chi Hsieh
>
> As per the discussion at pr 15807, we need to change the way of subexpression elimination.
> In current approach, common subexpressions are evaluated no matter they are really used or not later. E.g., in the following generated codes, {{subexpr2}} is evaluated even only the {{if}} branch is run.
> {code}
>     if (isNull(subexpr)) {
>       ...
>     } else {
>       AssertNotNull(subexpr)  // subexpr2
>       ....
>       SomeExpr(AssertNotNull(subexpr)) // SomeExpr(subexpr2)
>     }
> {code}
> Besides possible performance regression, the expression like {{AssertNotNull}} can throw exception. So wrongly evaluating {{subexpr2}} will throw exception unexceptedly..
> With this patch, now common subexpressions are not evaluated until they are used. We create a function for each common subexpression which evaluates and stores the result as a member variable. We have an initialization status variable to record whether the subexpression is evaluated.
> Thus, when an expression using the subexpression is going to be evaluated, we check if the subexpression is initialized, if yes directly returning the result, if no call the function to evaluate it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org