You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by viirya <gi...@git.apache.org> on 2016/10/31 09:08:54 UTC

[GitHub] spark pull request #15693: [SPARK-18125][SQL] Fix a compilation error in cod...

GitHub user viirya opened a pull request:

    https://github.com/apache/spark/pull/15693

    [SPARK-18125][SQL] Fix a compilation error in codegen due to splitExpression

    ## What changes were proposed in this pull request?
    
    As reported in the jira, sometimes the generated java code in codegen will cause compilation error.
    
    Code snippet to test it:
    
        case class Route(src: String, dest: String, cost: Int)
        case class GroupedRoutes(src: String, dest: String, routes: Seq[Route])
    
        val ds = sc.parallelize(Array(
          Route("a", "b", 1),
          Route("a", "b", 2),
          Route("a", "c", 2),
          Route("a", "d", 10),
          Route("b", "a", 1),
          Route("b", "a", 5),
          Route("b", "c", 6))
        ).toDF.as[Route]
    
        val grped = ds.map(r => GroupedRoutes(r.src, r.dest, Seq(r)))
          .groupByKey(r => (r.src, r.dest))
          .reduceGroups { (g1: GroupedRoutes, g2: GroupedRoutes) =>
            GroupedRoutes(g1.src, g1.dest, g1.routes ++ g2.routes)
          }.map(_._2)
    
    The problem here is, in `ReferenceToExpressions` we evaluate the children vars to local variables. Then the result expression is evaluated to use those children variables. In the above case, the result expression code is too long and will be split by `CodegenContext.splitExpression`. So those local variables cannot be accessed and cause compilation error.
    
    ## How was this patch tested?
    
    Jenkins tests.
    
    Please review https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark before opening a pull request.
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/viirya/spark-1 fix-codege-compilation-error

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/15693.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #15693
    
----
commit cb876e46cae1b90aaa0c681b4c6dbc66480e2950
Author: Liang-Chi Hsieh <vi...@gmail.com>
Date:   2016-10-31T08:59:00Z

    Fix a compilation error in codegen due to splitExpression.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    **[Test build #68173 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68173/consoleFull)** for PR 15693 at commit [`448abfa`](https://github.com/apache/spark/commit/448abfac34d5454a3bc74e0a4df11736713179b9).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    **[Test build #67808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67808/consoleFull)** for PR 15693 at commit [`0b660e0`](https://github.com/apache/spark/commit/0b660e02480bb3d193daf4acc997c1c0ca040930).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #15693: [SPARK-18125][SQL] Fix a compilation error in cod...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/15693#discussion_r86656043
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ReferenceToExpressions.scala ---
    @@ -63,15 +63,33 @@ case class ReferenceToExpressions(result: Expression, children: Seq[Expression])
     
       override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
         val childrenGen = children.map(_.genCode(ctx))
    -    val childrenVars = childrenGen.zip(children).map {
    -      case (childGen, child) => LambdaVariable(childGen.value, childGen.isNull, child.dataType)
    -    }
    +    val (childrenVars, classChildrenVars) = childrenGen.zip(children).map {
    +      case (childGen, child) =>
    +        val childVar = LambdaVariable(childGen.value, childGen.isNull, child.dataType)
    +
    +        // SPARK-18125: The children vars are local variables. If the result expression uses
    +        // splitExpression, those variables cannot be accessed so compilation fails.
    +        // To fix it, we use class variables to hold those local variables.
    +        val classChildVarName = ctx.freshName("classChildVar")
    +        val classChildVarIsNull = ctx.freshName("classChildVarIsNull")
    +        ctx.addMutableState(ctx.javaType(childVar.dataType), classChildVarName, "")
    +        ctx.addMutableState("boolean", classChildVarIsNull, "")
    +        val classChildVar =
    +          LambdaVariable(classChildVarName, classChildVarIsNull, childVar.dataType)
    +
    +        (childVar, classChildVar)
    +    }.unzip
    +
    +    val initClassChildrenVars = classChildrenVars.zipWithIndex.map { case (classChildrenVar, i) =>
    --- End diff --
    
    Just want to avoid tangling all codes together. I made the change to generate the code in previous block. Please see if it is better or not.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67808/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    **[Test build #67850 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67850/consoleFull)** for PR 15693 at commit [`5006bcb`](https://github.com/apache/spark/commit/5006bcb58d17e63aebeca6b9c62bf1e6ffaeb529).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    ping @hvanhovell Please let me know if you have other more comments. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    **[Test build #68173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68173/consoleFull)** for PR 15693 at commit [`448abfa`](https://github.com/apache/spark/commit/448abfac34d5454a3bc74e0a4df11736713179b9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #15693: [SPARK-18125][SQL] Fix a compilation error in cod...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/15693#discussion_r85740867
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ReferenceToExpressions.scala ---
    @@ -67,11 +67,27 @@ case class ReferenceToExpressions(result: Expression, children: Seq[Expression])
           case (childGen, child) => LambdaVariable(childGen.value, childGen.isNull, child.dataType)
         }
     
    +    // SPARK-18125: The children vars are local variables. If the result expression uses
    +    // splitExpression, those variables cannot be accessed so compilation fails.
    +    // To fix it, we use class variables to hold those local variables.
    +    val initClassChildVars = childrenVars.map { childVar =>
    --- End diff --
    
    Let me refactor it a bit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    @hvanhovell any more thoughts?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    LGTM. Merging to master/2.1. Thanks!
    
    Could you open a backport for 2.0?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68173/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    **[Test build #67811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67811/consoleFull)** for PR 15693 at commit [`0b660e0`](https://github.com/apache/spark/commit/0b660e02480bb3d193daf4acc997c1c0ca040930).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68269/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    **[Test build #67850 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67850/consoleFull)** for PR 15693 at commit [`5006bcb`](https://github.com/apache/spark/commit/5006bcb58d17e63aebeca6b9c62bf1e6ffaeb529).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    @hvanhovell OK. I will open a backport later.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    **[Test build #67807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67807/consoleFull)** for PR 15693 at commit [`cb876e4`](https://github.com/apache/spark/commit/cb876e46cae1b90aaa0c681b4c6dbc66480e2950).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #15693: [SPARK-18125][SQL] Fix a compilation error in cod...

Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/15693#discussion_r86601596
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ReferenceToExpressions.scala ---
    @@ -63,15 +63,33 @@ case class ReferenceToExpressions(result: Expression, children: Seq[Expression])
     
       override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
         val childrenGen = children.map(_.genCode(ctx))
    -    val childrenVars = childrenGen.zip(children).map {
    -      case (childGen, child) => LambdaVariable(childGen.value, childGen.isNull, child.dataType)
    -    }
    +    val (childrenVars, classChildrenVars) = childrenGen.zip(children).map {
    +      case (childGen, child) =>
    +        val childVar = LambdaVariable(childGen.value, childGen.isNull, child.dataType)
    +
    +        // SPARK-18125: The children vars are local variables. If the result expression uses
    +        // splitExpression, those variables cannot be accessed so compilation fails.
    +        // To fix it, we use class variables to hold those local variables.
    +        val classChildVarName = ctx.freshName("classChildVar")
    +        val classChildVarIsNull = ctx.freshName("classChildVarIsNull")
    +        ctx.addMutableState(ctx.javaType(childVar.dataType), classChildVarName, "")
    +        ctx.addMutableState("boolean", classChildVarIsNull, "")
    +        val classChildVar =
    +          LambdaVariable(classChildVarName, classChildVarIsNull, childVar.dataType)
    +
    +        (childVar, classChildVar)
    +    }.unzip
    +
    +    val initClassChildrenVars = classChildrenVars.zipWithIndex.map { case (classChildrenVar, i) =>
    --- End diff --
    
    Why not generate this code in the previous block?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    Thanks, good to see generated code for ease of understanding.
    While a local variable `value4` at line 239 cannot be accessed before this PR, an instance variable `classChildVar1` at line 306 can be accessed within this class after this PR.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    @kiszk the part of generated codes are posted as follows.
    
    Before this patch:
    
    Notice that line 239 `funcResult3 = value4._2();`, `value4` cannot be accessed.
    
        /* 288 */   public java.lang.Object apply(java.lang.Object _i) {
        /* 289 */     InternalRow i = (InternalRow) _i;
        /* 290 */
        /* 291 */
        /* 292 */
        /* 293 */     Object obj = ((Expression) references[0]).eval(null);
        /* 294 */     scala.Tuple2 value1 = (scala.Tuple2) obj;
        /* 295 */
        /* 296 */     boolean isNull2 = false;
        /* 297 */     boolean value2 = false;
        /* 298 */     if (!isNull2) {
        /* 299 */
        /* 300 */       Object funcResult = null;
        /* 301 */       funcResult = value1._1();
        /* 302 */       if (funcResult == null) {
        /* 303 */         isNull2 = true;
        /* 304 */       } else {
        /* 305 */         value2 = (Boolean) funcResult;
        /* 306 */       }
        /* 307 */
        /* 308 */     }
        /* 309 */     this.isNull_0 = isNull2;
        /* 310 */     this.value_0 = value2;
        /* 311 */
        /* 312 */
        /* 313 */     Object obj1 = ((Expression) references[1]).eval(null);
        /* 314 */     scala.Tuple2 value4 = (scala.Tuple2) obj1;
        /* 315 */
        /* 316 */     boolean isNull8 = false;
        /* 317 */     org.apache.spark.sql.GroupedRoutes value8 = null;
        /* 318 */     if (!isNull8) {
        /* 319 */
        /* 320 */       Object funcResult1 = null;
        /* 321 */       funcResult1 = value4._2();
        /* 322 */       if (funcResult1 == null) {
        /* 323 */         isNull8 = true;
        /* 324 */       } else {
        /* 325 */         value8 = (org.apache.spark.sql.GroupedRoutes) funcResult1;
        /* 326 */       }
        /* 327 */
        /* 328 */     }
        ...
        /* 232 */   private void apply1_0(InternalRow i) {
        /* 233 */
        /* 234 */     boolean isNull17 = false;
        /* 235 */     org.apache.spark.sql.GroupedRoutes value17 = null;
        /* 236 */     if (!isNull17) {
        /* 237 */
        /* 238 */       Object funcResult3 = null;
        /* 239 */       funcResult3 = value4._2();
        /* 240 */       if (funcResult3 == null) {
        /* 241 */         isNull17 = true;
        /* 242 */       } else {
        /* 243 */         value17 = (org.apache.spark.sql.GroupedRoutes) funcResult3;
        /* 244 */       }
        /* 245 */
        /* 246 */     }
        /* 247 */     isNull17 = value17 == null;
        /* 248 */
        /* 249 */     if (isNull17) {
        /* 250 */       throw new RuntimeException(((java.lang.String) references[3]));
        /* 251 */     }
        /* 252 */
    
        
    
    
    After this patch:
    
        /* 292 */   public java.lang.Object apply(java.lang.Object _i) {
        /* 293 */     InternalRow i = (InternalRow) _i;
        /* 294 */
        /* 295 */
        /* 296 */
        /* 297 */     Object obj = ((Expression) references[0]).eval(null);
        /* 298 */     scala.Tuple2 value1 = (scala.Tuple2) obj;
        /* 299 */     classChildVar = value1;
        /* 300 */     classChildVarIsNull = false;
        /* 301 */     boolean isNull2 = classChildVarIsNull;
        /* 302 */     boolean value2 = false;
        /* 303 */     if (!isNull2) {
        /* 304 */
        /* 305 */       Object funcResult = null;
        /* 306 */       funcResult = classChildVar._1();
        /* 307 */       if (funcResult == null) {
        /* 308 */         isNull2 = true;
        /* 309 */       } else {
        /* 310 */         value2 = (Boolean) funcResult;
        /* 311 */       }
        /* 312 */
        /* 313 */     }
        /* 314 */     this.isNull_0 = isNull2;
        /* 315 */     this.value_0 = value2;
        /* 316 */
        /* 317 */
        /* 318 */     Object obj1 = ((Expression) references[1]).eval(null);
        /* 319 */     scala.Tuple2 value4 = (scala.Tuple2) obj1;
        /* 320 */     classChildVar1 = value4;
        /* 321 */     classChildVarIsNull1 = false;
        /* 322 */     boolean isNull8 = classChildVarIsNull1;
        /* 323 */     org.apache.spark.sql.GroupedRoutes value8 = null;
        /* 324 */     if (!isNull8) {
        /* 325 */
        /* 326 */       Object funcResult1 = null;
        /* 327 */       funcResult1 = classChildVar1._2();
        /* 328 */       if (funcResult1 == null) {
        /* 329 */         isNull8 = true;
        /* 330 */       } else {
        /* 331 */         value8 = (org.apache.spark.sql.GroupedRoutes) funcResult1;
        /* 332 */       }
        /* 333 */
        /* 334 */     }
        /* 335 */     isNull8 = value8 == null;
        ...
        /* 236 */   private void apply1_0(InternalRow i) {
        /* 237 */
        /* 238 */     boolean isNull17 = classChildVarIsNull1;
        /* 239 */     org.apache.spark.sql.GroupedRoutes value17 = null;
        /* 240 */     if (!isNull17) {
        /* 241 */
        /* 242 */       Object funcResult3 = null;
        /* 243 */       funcResult3 = classChildVar1._2();
        /* 244 */       if (funcResult3 == null) {
        /* 245 */         isNull17 = true;
        /* 246 */       } else {
        /* 247 */         value17 = (org.apache.spark.sql.GroupedRoutes) funcResult3;
        /* 248 */       }
        /* 249 */
        /* 250 */     }
        /* 251 */     isNull17 = value17 == null;
        /* 252 */
        /* 253 */     if (isNull17) {
        /* 254 */       throw new RuntimeException(((java.lang.String) references[3]));
        /* 255 */     }
        /* 256 */
        /* 257 */     boolean isNull15 = false;
        /* 258 */     java.lang.String value15 = null;
        /* 259 */     if (!isNull15) {
        /* 260 */
        /* 261 */       Object funcResult4 = null;
        /* 262 */       funcResult4 = value17.src();
        /* 263 */       if (funcResult4 == null) {
        /* 264 */         isNull15 = true;
        /* 265 */       } else {
        /* 266 */         value15 = (java.lang.String) funcResult4;
        /* 267 */       }
        /* 268 */
        /* 269 */     }
        /* 270 */     isNull15 = value15 == null;
        /* 271 */     boolean isNull14 = isNull15;
        /* 272 */     final UTF8String value14 = isNull14 ? null : org.apache.spark.unsafe.types.UTF8String.fromString(value15);
        /* 273 */     isNull14 = value14 == null;
        /* 274 */     if (isNull14) {
        /* 275 */       values[0] = null;
        /* 276 */     } else {
        /* 277 */       values[0] = value14;
        /* 278 */     }
        /* 279 */   }
        /* 280 */
    
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #15693: [SPARK-18125][SQL] Fix a compilation error in cod...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/15693#discussion_r85739963
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ReferenceToExpressions.scala ---
    @@ -67,11 +67,27 @@ case class ReferenceToExpressions(result: Expression, children: Seq[Expression])
           case (childGen, child) => LambdaVariable(childGen.value, childGen.isNull, child.dataType)
         }
     
    +    // SPARK-18125: The children vars are local variables. If the result expression uses
    +    // splitExpression, those variables cannot be accessed so compilation fails.
    +    // To fix it, we use class variables to hold those local variables.
    +    val initClassChildVars = childrenVars.map { childVar =>
    --- End diff --
    
    We could. Not sure if merging them will reduce readability or not.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #15693: [SPARK-18125][SQL] Fix a compilation error in cod...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/15693


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67850/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    Would it be possible to put a part of generated code before and after this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    **[Test build #68269 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68269/consoleFull)** for PR 15693 at commit [`8b71061`](https://github.com/apache/spark/commit/8b7106111db93c7939c3f68c84f327bba3cda84b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67807/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67811/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    **[Test build #67808 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67808/consoleFull)** for PR 15693 at commit [`0b660e0`](https://github.com/apache/spark/commit/0b660e02480bb3d193daf4acc997c1c0ca040930).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    **[Test build #67807 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67807/consoleFull)** for PR 15693 at commit [`cb876e4`](https://github.com/apache/spark/commit/cb876e46cae1b90aaa0c681b4c6dbc66480e2950).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #15693: [SPARK-18125][SQL] Fix a compilation error in cod...

Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/15693#discussion_r85738758
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ReferenceToExpressions.scala ---
    @@ -67,11 +67,27 @@ case class ReferenceToExpressions(result: Expression, children: Seq[Expression])
           case (childGen, child) => LambdaVariable(childGen.value, childGen.isNull, child.dataType)
         }
     
    +    // SPARK-18125: The children vars are local variables. If the result expression uses
    +    // splitExpression, those variables cannot be accessed so compilation fails.
    +    // To fix it, we use class variables to hold those local variables.
    +    val initClassChildVars = childrenVars.map { childVar =>
    --- End diff --
    
    Shouldn't we merge this with `childrenVars`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    **[Test build #68269 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68269/consoleFull)** for PR 15693 at commit [`8b71061`](https://github.com/apache/spark/commit/8b7106111db93c7939c3f68c84f327bba3cda84b).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/15693
  
    **[Test build #67811 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67811/consoleFull)** for PR 15693 at commit [`0b660e0`](https://github.com/apache/spark/commit/0b660e02480bb3d193daf4acc997c1c0ca040930).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org