You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by kiszk <gi...@git.apache.org> on 2017/12/02 06:09:31 UTC

[GitHub] spark pull request #19865: [SPARK-22668][SQL] Exclude global variables from ...

GitHub user kiszk opened a pull request:

    https://github.com/apache/spark/pull/19865

    [SPARK-22668][SQL] Exclude global variables from arguments of method split by CodegenContext.splitExpressions()

    ## What changes were proposed in this pull request?
    
    This PR fixes to make incorrect result when split method by `CodegenContext.splitExpressions() ` has global variables in its arguments. When variables in arguments are declared as local variables, to assign a value to the variable, which has been originally declared as a global varible, updates a local variable.
    
    To fix this problem, this PR excludes global variables from arguments of the split method.
    
    Before this PR
    ```
    class Test {
      int globalInt;
    
      void splittedFunction(int globalInt) {
        ...
        globalInt = 2;
      }
    
      void apply() {
        globalInt = 1;
        ...
        splittedFunction(globalInt);
        // globalInt should be 2 here since it is 2 if statemetns are not split
      }
    }
    ```
    
    After this PR.
    ```
    class Test {
      int globalInt;
    
      void splittedFunction() {
        ...
        globalInt = 2;
      }
    
      void apply() {
        globalInt = 1;
        ...
        splittedFunction();
        // globalInt is 2
      }
    }
    ```
    
    ## How was this patch tested?
    
    Added test case into `CodeGenerationSuite`

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kiszk/spark SPARK-22668

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19865.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19865
    
----
commit 3b25b5f2fdaec7b1e97732f1b8a0bf4c588a44f2
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date:   2017-12-02T06:07:03Z

    initial commit

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84688/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19865: [SPARK-22668][SQL] Ensure no global variables in ...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19865#discussion_r157122860
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ---
    @@ -217,6 +217,18 @@ class CodegenContext {
         splitExpressions(expressions = initCodes, funcName = "init", arguments = Nil)
       }
     
    +  /**
    +   * Return true if a given variable has been described as a global variable
    +   */
    +  def isDeclaredMutableState(varName: String): Boolean = {
    --- End diff --
    
    I think that it is a good way at caller side since this function is valid at debug and production environments.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84687/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Do not pass global variables to argum...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Thank you. I think so for this case.  
    In general, to make global `ev.value` or to pass a global variable to `consume()` may potentially cause this problem.  
    As @cloud-fan suggested, I identified these patterns and am fixing them.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    thanks for your answer @kiszk . Maybe I have not understood correctly, thus I apologize in advance if my guessing is wrong, but if this is a problem introduced by #19811, why aren't we fixing it there?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    I am neutral how to fix this problem in the current master. What I am saying from the beginning is that this problem does not only exists in #19811, but also in the current master.
    
    I am happy to agree that we fix the invalid case in the current master.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19865: [SPARK-22668][SQL] Ensure no global variables in ...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19865#discussion_r156703303
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ---
    @@ -842,7 +856,10 @@ class CodegenContext {
           blocks.head
         } else {
           val func = freshName(funcName)
    -      val argString = arguments.map { case (t, name) => s"$t $name" }.mkString(", ")
    +      val argString = arguments.map { case (t, name) =>
    +        assert(!isDeclaredMutableState(name),
    --- End diff --
    
    Got it. I did not know #19082 has own split routine. While it is under review, there is no way to add assertion by this PR.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    **[Test build #84391 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84391/testReport)** for PR 19865 at commit [`d04544c`](https://github.com/apache/spark/commit/d04544cc06216f2d38fa36b5f3f8ecc3882b448c).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    I think for this case, shouldn't we fix it and not pass in a global variable into `splitExpressions`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    **[Test build #84401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84401/testReport)** for PR 19865 at commit [`ad86986`](https://github.com/apache/spark/commit/ad869866f4aa28eceaf30a1fa09f3d5930a726ad).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    **[Test build #84402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84402/testReport)** for PR 19865 at commit [`f63cd3a`](https://github.com/apache/spark/commit/f63cd3a37f828f86aa2a991bd611813a155645d5).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Do not pass global variables to argum...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    hopefully https://github.com/apache/spark/pull/19878 can fix the problem.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    @mgaido91 Since I realized this issue through the work of #19811, I do not have an real example now.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84387/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    I would like to hear options from others, too.
    WDYT? @cloud-fan @viirya @maropu @gatorsmile 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84386/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    **[Test build #84391 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84391/testReport)** for PR 19865 at commit [`d04544c`](https://github.com/apache/spark/commit/d04544cc06216f2d38fa36b5f3f8ecc3882b448c).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    **[Test build #84688 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84688/testReport)** for PR 19865 at commit [`c37311f`](https://github.com/apache/spark/commit/c37311f8d0385c979054fb800bbc5b1a784d199a).
     * This patch **fails PySpark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    @mgaido91 @viirya  As you see, we see an assertion failure. Here is an evidence that we pass a global variable to arguments of split function.
    In practice, we did not guarantee that we do not pass a global variable.
    
    
    An [value](github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala) was declared as a global variable. Then, it is passed as `ExprCode.value`. Finally, `value` is passed as an argument in `CodeGenContext.splitFunction`. Fortunally, this `expressions` did not update the global variable. Thus, it worked fuctionally correct. 
    In general, it is hard to ensure there is no update in the `expressions`. Of course, we do not like to use regular expressions to detect it.
    
    As you said, how do we ensure that we do not pass a global variable?
    
    
    
    ```
    **********************************************************************
    File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/ml/feature.py", line 1205, in __main__.MinHashLSH
    Failed example:
    ...
        Caused by: java.lang.AssertionError: assertion failed: smj_value16 in arguments should not be declared as a global variable
        	at scala.Predef$.assert(Predef.scala:170)
        	at org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext.org$apache$spark$sql$catalyst$expressions$codegen$CodegenContext$$isDeclaredMutableState(CodeGenerator.scala:226)
        	at org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext$$anonfun$9.apply(CodeGenerator.scala:854)
        	at org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext$$anonfun$9.apply(CodeGenerator.scala:854)
        	at scala.collection.TraversableLike$$anonfun$filterImpl$1.apply(TraversableLike.scala:248)
        	at scala.collection.immutable.List.foreach(List.scala:381)
        	at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
        	at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
        	at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
        	at org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext.splitExpressions(CodeGenerator.scala:853)
        	at org.apache.spark.sql.catalyst.expressions.HashExpression.genHashForStruct(hash.scala:395)
        	at org.apache.spark.sql.catalyst.expressions.HashExpression.computeHashWithTailRec(hash.scala:421)
        	at org.apache.spark.sql.catalyst.expressions.HashExpression.computeHash(hash.scala:429)
        	at org.apache.spark.sql.catalyst.expressions.HashExpression$$anonfun$1.apply(hash.scala:276)
        	at org.apache.spark.sql.catalyst.expressions.HashExpression$$anonfun$1.apply(hash.scala:273)
        	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
        	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        	at org.apache.spark.sql.catalyst.expressions.HashExpression.doGenCode(hash.scala:273)
        	at org.apache.spark.sql.catalyst.expressions.Expression$$anonfun$genCode$2.apply(Expression.scala:107)
        	at org.apache.spark.sql.catalyst.expressions.Expression$$anonfun$genCode$2.apply(Expression.scala:104)
        	at scala.Option.getOrElse(Option.scala:121)
        	at org.apache.spark.sql.catalyst.expressions.Expression.genCode(Expression.scala:104)
        	at org.apache.spark.sql.execution.aggregate.HashAggregateExec.doConsumeWithKeys(HashAggregateExec.scala:772)
        	at org.apache.spark.sql.execution.aggregate.HashAggregateExec.doConsume(HashAggregateExec.scala:173)
        	at org.apache.spark.sql.execution.CodegenSupport$class.consume(WholeStageCodegenExec.scala:162)
        	at org.apache.spark.sql.execution.ProjectExec.consume(basicPhysicalOperators.scala:35)
        	at org.apache.spark.sql.execution.ProjectExec.doConsume(basicPhysicalOperators.scala:65)
        	at org.apache.spark.sql.execution.CodegenSupport$class.consume(WholeStageCodegenExec.scala:162)
        	at org.apache.spark.sql.execution.joins.SortMergeJoinExec.consume(SortMergeJoinExec.scala:36)
        	at org.apache.spark.sql.execution.joins.SortMergeJoinExec.doProduce(SortMergeJoinExec.scala:626)
        	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:85)
        	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:80)
        	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:141)
        	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:138)
        	at org.apache.spark.sql.execution.CodegenSupport$class.produce(WholeStageCodegenExec.scala:80)
        	at org.apache.spark.sql.execution.joins.SortMergeJoinExec.produce(SortMergeJoinExec.scala:36)
        	at org.apache.spark.sql.execution.ProjectExec.doProduce(basicPhysicalOperators.scala:45)
        	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:85)
        	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:80)
        	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:141)
        	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:138)
        	at org.apache.spark.sql.execution.CodegenSupport$class.produce(WholeStageCodegenExec.scala:80)
        	at org.apache.spark.sql.execution.ProjectExec.produce(basicPhysicalOperators.scala:35)
        	at org.apache.spark.sql.execution.aggregate.HashAggregateExec.doProduceWithKeys(HashAggregateExec.scala:647)
        	at org.apache.spark.sql.execution.aggregate.HashAggregateExec.doProduce(HashAggregateExec.scala:165)
        	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:85)
        	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:80)
        	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:141)
        	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:138)
        	at org.apache.spark.sql.execution.CodegenSupport$class.produce(WholeStageCodegenExec.scala:80)
        	at org.apache.spark.sql.execution.aggregate.HashAggregateExec.produce(HashAggregateExec.scala:39)
        	at org.apache.spark.sql.execution.WholeStageCodegenExec.doCodeGen(WholeStageCodegenExec.scala:374)
        	at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:422)
        	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
        	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:113)
        	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:141)
        	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:138)
        	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
        	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.prepareShuffleDependency(ShuffleExchangeExec.scala:89)
        	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$doExecute$1.apply(ShuffleExchangeExec.scala:125)
        	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$doExecute$1.apply(ShuffleExchangeExec.scala:116)
        	at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
    ...```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    It would be good to hear opinions of others.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    I think we have to guarantee this. The case you are dealing with is a wrong situation that should be avoided. Indeed, in such a case, it is not even clear to me which is the right thing to do: I mean, which was the intention? Was it to use the global variable or was it to use the passed argument?
    
    Here you are assuming that the intention was to use the global variable, which actually is an interpretation I would question, since up to me the right thing to do is to use the passed variable as Java does on his own, because anybody creating this situation hopefully knows as Java works, thus he/she might also rely on that.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19865: [SPARK-22668][SQL] Ensure no global variables in ...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/19865


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    cc @cloud-fan @viirya @maropu


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84391/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19865: [SPARK-22668][SQL] Ensure no global variables in ...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19865#discussion_r156866827
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ---
    @@ -217,6 +217,18 @@ class CodegenContext {
         splitExpressions(expressions = initCodes, funcName = "init", arguments = Nil)
       }
     
    +  /**
    +   * Return true if a given variable has been described as a global variable
    +   */
    +  def isDeclaredMutableState(varName: String): Boolean = {
    +    val j = varName.indexOf("[")
    --- End diff --
    
    I see.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    I see, my point is: I cannot understand how this can happen in the current codebase. Is there any place where this problem is present now? Because at the moment it looks to me more like a potential problem and not a real problem. Not sure if I was clear enough.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Assert to ensure no global variables ...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    @cloud-fan unfortunately, #19878 did not fix this issue. #19937 will fix this issue.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19865: [SPARK-22668][SQL] Ensure no global variables in ...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19865#discussion_r156862608
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ---
    @@ -217,6 +217,18 @@ class CodegenContext {
         splitExpressions(expressions = initCodes, funcName = "init", arguments = Nil)
       }
     
    +  /**
    +   * Return true if a given variable has been described as a global variable
    +   */
    +  def isDeclaredMutableState(varName: String): Boolean = {
    --- End diff --
    
    let's only enable this check in test environment, in case it has bugs and break production jobs.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19865: [SPARK-22668][SQL] Ensure no global variables in ...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19865#discussion_r157110926
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ---
    @@ -217,6 +217,18 @@ class CodegenContext {
         splitExpressions(expressions = initCodes, funcName = "init", arguments = Nil)
       }
     
    +  /**
    +   * Return true if a given variable has been described as a global variable
    +   */
    +  def isDeclaredMutableState(varName: String): Boolean = {
    --- End diff --
    
    How about checking Utils.isTesting and throwing an exception in tests?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    **[Test build #84387 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84387/testReport)** for PR 19865 at commit [`cf796fe`](https://github.com/apache/spark/commit/cf796fe8ad0da85fa397371fa26d9b850047b3da).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `      class CodeGenSuiteTest extends $`
      * `abstract class CodeGenSuiteAbstract `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Do not pass global variables to argum...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Since other PRs addresses several operations to reduce usage of global variables, this PR will address `SortMergeJoinExec` and `HashAggregateExec`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19865: [SPARK-22668][SQL] Ensure no global variables in ...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19865#discussion_r156838026
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ---
    @@ -842,7 +856,10 @@ class CodegenContext {
           blocks.head
         } else {
           val func = freshName(funcName)
    -      val argString = arguments.map { case (t, name) => s"$t $name" }.mkString(", ")
    +      val argString = arguments.map { case (t, name) =>
    +        assert(!isDeclaredMutableState(name),
    --- End diff --
    
    IMHO we'd better to check arguments in all the registered functions via `addFunction` if you check the case described in the description?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19865: [SPARK-22668][SQL] Ensure no global variables in ...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19865#discussion_r156868316
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ---
    @@ -217,6 +217,18 @@ class CodegenContext {
         splitExpressions(expressions = initCodes, funcName = "init", arguments = Nil)
       }
     
    +  /**
    +   * Return true if a given variable has been described as a global variable
    +   */
    +  def isDeclaredMutableState(varName: String): Boolean = {
    --- End diff --
    
    hmm, even only in `assert`, it still can fail compilation, doesn't it?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    @kiszk I think that in the case you hit them, this might have also been done appositely and relying on the way Java behaves, ie. that it uses the local variable and the global one is not used there. It can also be something which has been designed like that.  In this way instead you are forcing a behavior which is not the expected one.
    
    I totally agree with @cloud-fan, that we should fix the problem where they are created, if there is any. We can also decide that such situation should be avoided for clarity, and therefore we can change the point where you find this behavior to be present. I am neutral to that. But I disagree in creating a situation which is counterintuitive.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    **[Test build #84386 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84386/testReport)** for PR 19865 at commit [`3b25b5f`](https://github.com/apache/spark/commit/3b25b5f2fdaec7b1e97732f1b8a0bf4c588a44f2).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84401/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    **[Test build #84402 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84402/testReport)** for PR 19865 at commit [`f63cd3a`](https://github.com/apache/spark/commit/f63cd3a37f828f86aa2a991bd611813a155645d5).
     * This patch **fails PySpark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84402/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    **[Test build #84387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84387/testReport)** for PR 19865 at commit [`cf796fe`](https://github.com/apache/spark/commit/cf796fe8ad0da85fa397371fa26d9b850047b3da).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    @kiszk do you have any real example where this problem can happen now? If so, can we add an end-to-end test case?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Thank you for your comment. This problem potentially exists regard less #19811. I added a test with end-to-end code generation regarding this.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    **[Test build #84687 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84687/testReport)** for PR 19865 at commit [`4e83f9f`](https://github.com/apache/spark/commit/4e83f9f40e514600ce4e56b40c7e41e74e998ba8).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    I confirmed that this failure does not occur after merging #19937 in my environment.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19865: [SPARK-22668][SQL] Ensure no global variables in ...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19865#discussion_r156866697
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ---
    @@ -217,6 +217,18 @@ class CodegenContext {
         splitExpressions(expressions = initCodes, funcName = "init", arguments = Nil)
       }
     
    +  /**
    +   * Return true if a given variable has been described as a global variable
    +   */
    +  def isDeclaredMutableState(varName: String): Boolean = {
    --- End diff --
    
    Of course, this PR uses the method only in `assert`.  


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19865: [SPARK-22668][SQL] Ensure no global variables in ...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19865#discussion_r156863124
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ---
    @@ -217,6 +217,18 @@ class CodegenContext {
         splitExpressions(expressions = initCodes, funcName = "init", arguments = Nil)
       }
     
    +  /**
    +   * Return true if a given variable has been described as a global variable
    +   */
    +  def isDeclaredMutableState(varName: String): Boolean = {
    +    val j = varName.indexOf("[")
    --- End diff --
    
    we don't need to deal with `[]` here, using them as parameter name will fail to compile.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    **[Test build #84688 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84688/testReport)** for PR 19865 at commit [`c37311f`](https://github.com/apache/spark/commit/c37311f8d0385c979054fb800bbc5b1a784d199a).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    **[Test build #84386 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84386/testReport)** for PR 19865 at commit [`3b25b5f`](https://github.com/apache/spark/commit/3b25b5f2fdaec7b1e97732f1b8a0bf4c588a44f2).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19865: [SPARK-22668][SQL] Ensure no global variables in ...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19865#discussion_r156889936
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ---
    @@ -217,6 +217,18 @@ class CodegenContext {
         splitExpressions(expressions = initCodes, funcName = "init", arguments = Nil)
       }
     
    +  /**
    +   * Return true if a given variable has been described as a global variable
    +   */
    +  def isDeclaredMutableState(varName: String): Boolean = {
    +    val j = varName.indexOf("[")
    +    val qualifiedName = if (j < 0) varName else varName.substring(0, j)
    +    mutableStates.find { s =>
    --- End diff --
    
    nit: find -> exists


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    **[Test build #84687 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84687/testReport)** for PR 19865 at commit [`4e83f9f`](https://github.com/apache/spark/commit/4e83f9f40e514600ce4e56b40c7e41e74e998ba8).
     * This patch **fails PySpark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    We think that we normally would not pass a global variable. My question is: Did we guarantee it?  
    If so, I would recommend to insert assertion.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    @cloud-fan I see. As I pointed out, there are several places to set a global variable `ExprCode.value` that is passed to successor operations.  
    Should we make lifetime of global time local in an operation?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    making a variable global need to be done manually(call `ctx.mutableState`), splitting the code into methods also need to be done manually(call `ctx.splitExpressions`). If we hit a problem here, it's probably due to misuse. Can we just check the codebase, find these invalid cases and fix them? We may probably add document to `ctx.splitExpression` that global variables should not be in the parameter.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    **[Test build #84401 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84401/testReport)** for PR 19865 at commit [`ad86986`](https://github.com/apache/spark/commit/ad869866f4aa28eceaf30a1fa09f3d5930a726ad).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19865: [SPARK-22668][SQL] Ensure no global variables in ...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19865#discussion_r156243726
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ---
    @@ -842,7 +856,10 @@ class CodegenContext {
           blocks.head
         } else {
           val func = freshName(funcName)
    -      val argString = arguments.map { case (t, name) => s"$t $name" }.mkString(", ")
    +      val argString = arguments.map { case (t, name) =>
    +        assert(!isDeclaredMutableState(name),
    --- End diff --
    
    I'm neutral about this pr though, I feel this is not the best place for this assertion cuz this is not only the place where the suggested case happens, e.g., my pr splits aggregate functions in `HashAggregate`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/19865
  
    If the parameters to `splitExpressions` are given by us, I think normally we won't pass in a global variable?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19865: [SPARK-22668][SQL] Ensure no global variables in ...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19865#discussion_r156866988
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala ---
    @@ -842,7 +856,10 @@ class CodegenContext {
           blocks.head
         } else {
           val func = freshName(funcName)
    -      val argString = arguments.map { case (t, name) => s"$t $name" }.mkString(", ")
    +      val argString = arguments.map { case (t, name) =>
    +        assert(!isDeclaredMutableState(name),
    --- End diff --
    
    Yeah, this problem occurs not only in split method also in normal method. Let us check this in `addNewFunction`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org