You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/06/03 05:12:43 UTC

[GitHub] [spark] karuppayya opened a new pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

karuppayya opened a new pull request #28715:
URL: https://github.com/apache/spark/pull/28715


   ### What changes were proposed in this pull request?
   Enabling codegen for GenerateExec
   
   ### Why are the changes needed?
   To leverage code geneartion for Generators
   
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   ### How was this patch tested?
   - UT tests added
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642882468


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123862/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-791799869


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135819/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-645008292






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r440609422



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,93 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("SPARK-31897: GenerateExec should be included in WholeStageCodegen") {
+    import testImplicits._
+    val arrayData = Seq(("James", Seq("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown")))
+    val df = arrayData.toDF("name", "knownLanguages", "properties")
+
+    // Array - explode
+    var expDF = df.select($"name", explode($"knownLanguages"), $"properties")
+    var plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ => false
+    }.isDefined)
+    var results = expDF.collect()
+    assert(results ===
+      Array(Row("James", "Java", Map("hair" -> "black", "eye" -> "brown")),
+        Row("James", "Scala", Map("hair" -> "black", "eye" -> "brown"))))

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] github-actions[bot] commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-748546359


   We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-644871681


   **[Test build #124132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124132/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com/apache/spark/commit/d6f8459556d864b3d2249f45f73332812659dba0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642450214






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-796278694


   Added the benchmark using the benchmark infra. It tests with codegen on and off .
   Since there is no way to disable codegen only for GenerateExec, disabling codegen will disable code generation for all operators. In the added benchmark, Project operator will be affected, but this should not cause a huge deviation in the benchmark. Please let me know otherwise.
   Benchmark results:
   ```
   [info] GenerateExec Benchmark:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   [info] ------------------------------------------------------------------------------------------------------------------------
   [info] GenerateExec Benchmark wholestage off             85625          87134        2134          1.2         856.3       1.0X
   [info] GenerateExec Benchmark wholestage on              43547          44003         377          2.3         435.5       2.0X
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on a change in pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r438915561



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -124,7 +124,7 @@ case class GenerateExec(
     }
   }
 
-  override def supportCodegen: Boolean = false
+  override def supportCodegen: Boolean = true

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r440525221



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -226,18 +226,33 @@ case class GenerateExec(
       "0"
     }
     val numOutput = metricTerm(ctx, "numOutputRows")
+    val requiredInput =
+      evaluateRequiredVariablesExpr(child.output,
+        input, parent.inputSet) ++ position ++ values
     s"""
        |${data.code}
        |$initMapData
        |int $numElements = ${data.isNull} ? 0 : ${data.value}.numElements();
        |for (int $index = $init; $index < $numElements; $index++) {
        |  $numOutput.add(1);
        |  $updateRowData
-       |  ${consume(ctx, input ++ position ++ values)}
+       |  ${consume(ctx, requiredInput)}
        |}
      """.stripMargin
   }
 
+  /**
+   * Returns [ExprCode] for required attributes
+   */
+  private def evaluateRequiredVariablesExpr(

Review comment:
       How about inlining this method in the caller side? Btw, why do we need this method? It seems https://github.com/apache/spark/commit/b5c5bd98ea5e8dbfebcf86c5459bdf765f5ceb53 doesn't have this logic though.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642882462


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-657163572






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667706833


   **[Test build #126942 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126942/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com/apache/spark/commit/d6f8459556d864b3d2249f45f73332812659dba0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642325105


   **[Test build #123798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123798/testReport)** for PR 28715 at commit [`a9bd9db`](https://github.com/apache/spark/commit/a9bd9dbc8b2dfc67bc9bd41f773c68f427e6a309).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-644574974


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124100/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r440525322



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -226,18 +226,33 @@ case class GenerateExec(
       "0"
     }
     val numOutput = metricTerm(ctx, "numOutputRows")
+    val requiredInput =
+      evaluateRequiredVariablesExpr(child.output,
+        input, parent.inputSet) ++ position ++ values

Review comment:
       nit format;
   ```
       val numOutput = metricTerm(ctx, "numOutputRows")
       val requiredInput = evaluateRequiredVariablesExpr(child.output, input, parent.inputSet) ++
         position ++ values
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-791620200


   **[Test build #135819 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135819/testReport)** for PR 28715 at commit [`7635542`](https://github.com/apache/spark/commit/7635542a68f95d0ba41bdb4f2e39b2558e7192a7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642325105


   **[Test build #123798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123798/testReport)** for PR 28715 at commit [`a9bd9db`](https://github.com/apache/spark/commit/a9bd9dbc8b2dfc67bc9bd41f773c68f427e6a309).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642317273


   **[Test build #123791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123791/testReport)** for PR 28715 at commit [`caf0ce4`](https://github.com/apache/spark/commit/caf0ce401722c2e197e9e4bc4a11832f698acd37).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-791620200


   **[Test build #135819 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135819/testReport)** for PR 28715 at commit [`7635542`](https://github.com/apache/spark/commit/7635542a68f95d0ba41bdb4f2e39b2558e7192a7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-796437217


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135949/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-793271717






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642449582


   **[Test build #123828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123828/testReport)** for PR 28715 at commit [`bdbaa6b`](https://github.com/apache/spark/commit/bdbaa6b2becc311bde4a342668e4501325b6aa28).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-791784128


   **[Test build #135819 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135819/testReport)** for PR 28715 at commit [`7635542`](https://github.com/apache/spark/commit/7635542a68f95d0ba41bdb4f2e39b2558e7192a7).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-790306323


   **[Test build #135725 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135725/testReport)** for PR 28715 at commit [`ce9cd00`](https://github.com/apache/spark/commit/ce9cd000565497af71e189ad83f6d8f9d50ac271).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-788163539


   **[Test build #135603 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135603/testReport)** for PR 28715 at commit [`d6640de`](https://github.com/apache/spark/commit/d6640ded2604294cd58b37b6c9f2f14b05676a66).
    * This patch **fails to build**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r589256580



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -226,14 +229,15 @@ case class GenerateExec(
       "0"
     }
     val numOutput = metricTerm(ctx, "numOutputRows")
+    val requiredInput = input ++ position ++ values

Review comment:
       Unnecessary change. We can just inline it as before.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802051267


   @viirya Thanks for triggering the tests. @maropu I have handled the comment and the test are passing now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-788454084


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135608/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-657163441


   **[Test build #125703 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125703/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com/apache/spark/commit/d6f8459556d864b3d2249f45f73332812659dba0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r440603564



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -226,18 +226,33 @@ case class GenerateExec(
       "0"
     }
     val numOutput = metricTerm(ctx, "numOutputRows")
+    val requiredInput =
+      evaluateRequiredVariablesExpr(child.output,
+        input, parent.inputSet) ++ position ++ values
     s"""
        |${data.code}
        |$initMapData
        |int $numElements = ${data.isNull} ? 0 : ${data.value}.numElements();
        |for (int $index = $init; $index < $numElements; $index++) {
        |  $numOutput.add(1);
        |  $updateRowData
-       |  ${consume(ctx, input ++ position ++ values)}
+       |  ${consume(ctx, requiredInput)}
        |}
      """.stripMargin
   }
 
+  /**
+   * Returns [ExprCode] for required attributes
+   */
+  private def evaluateRequiredVariablesExpr(

Review comment:
       This method is required when we either project all column or project none of the columns(except for the exploded columns).
   There were couple of test failure
   https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123751/testReport/
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r589255729



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -144,9 +144,12 @@ case class GenerateExec(
       Seq.empty
     }
 
+    val requiredInput = input.zip(child.output).filter {

Review comment:
       Why do we need to look at `parent.input`? How about
   ```
   val requiredAttrSet = AttributeSet(requiredChildOutput)
   val requiredInput = child.output.zip(input).filter {
     case (attr, _) => requiredAttrSet.contains(attr)
   }.map(_._2)
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-788149703


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40184/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-801479973


   @karuppayya could you address the last @viirya comment? After that, I'll check and merge it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642388028






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-801519545


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40763/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-796299529


   **[Test build #135949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135949/testReport)** for PR 28715 at commit [`9a34cfa`](https://github.com/apache/spark/commit/9a34cfae0376bf7d6873a9e63ee020243e048387).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642383840






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r437985630



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,77 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("GenerateExec should be included in WholeStageCodegen") {
+    val arrayData1 = Seq(
+      Row("James", List("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown"))
+    )
+
+    val arraySchema1 = new StructType()
+      .add("name", StringType)
+      .add("knownLanguages", ArrayType(StringType))
+      .add("properties", MapType(StringType, StringType))
+
+    val df = spark.createDataFrame(spark.sparkContext.parallelize(arrayData1), arraySchema1)

Review comment:
       nit:
   ```
   val df = Seq(("James"), Seq("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown"))
     .toDF("name", "knownLanguages", "properties")
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642454680






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642517025






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642679308






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-644872445






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642999598


   **[Test build #123873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123873/testReport)** for PR 28715 at commit [`8cc5394`](https://github.com/apache/spark/commit/8cc5394e812ab33854f6be27e4469eadfc4d5419).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-645572648


   Ready for review @maropu @cloud-fan 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642382760


   **[Test build #123812 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123812/testReport)** for PR 28715 at commit [`bdbaa6b`](https://github.com/apache/spark/commit/bdbaa6b2becc311bde4a342668e4501325b6aa28).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642325411






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667625572






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642793344






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-657183513


   **[Test build #125703 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125703/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com/apache/spark/commit/d6f8459556d864b3d2249f45f73332812659dba0).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642319332


   **[Test build #123792 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123792/testReport)** for PR 28715 at commit [`869330d`](https://github.com/apache/spark/commit/869330d23eb4e159ad8e98bb9656e3de3d9ba31e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642439230


   The test failure doesnt seem related to this change.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-644574965


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-796315176


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40533/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-788180196


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135603/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-787728721


   @karuppayya Are you still here? Could you fix the build failure?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-796315176


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40533/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-797167922


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40568/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on a change in pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r438569824



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -226,18 +226,33 @@ case class GenerateExec(
       "0"
     }
     val numOutput = metricTerm(ctx, "numOutputRows")
+    val requiredInput =
+      evaluateRequiredVariablesExpr(child.output,
+        input, parent.inputSet) ++ position ++ values
     s"""
        |${data.code}
        |$initMapData
        |int $numElements = ${data.isNull} ? 0 : ${data.value}.numElements();
        |for (int $index = $init; $index < $numElements; $index++) {
        |  $numOutput.add(1);
        |  $updateRowData
-       |  ${consume(ctx, input ++ position ++ values)}
+       |  ${consume(ctx, requiredInput)}
        |}
      """.stripMargin
   }
 
+  /**
+   * Returns [ExprCode] for required attributes
+   */
+  private def evaluateRequiredVariablesExpr(

Review comment:
       @cloud-fan Should this method be in introduced in WholeStageCodeGenExec.scala? Please advise




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642320102


   **[Test build #123792 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123792/testReport)** for PR 28715 at commit [`869330d`](https://github.com/apache/spark/commit/869330d23eb4e159ad8e98bb9656e3de3d9ba31e).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-801648211


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40772/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-788159133


   **[Test build #135603 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135603/testReport)** for PR 28715 at commit [`d6640de`](https://github.com/apache/spark/commit/d6640ded2604294cd58b37b6c9f2f14b05676a66).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r592825867



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/GenerateExecBenchmark.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import org.apache.spark.sql.execution.benchmark.SqlBasedBenchmark
+import org.apache.spark.sql.functions.explode
+
+/**
+ * Benchmark to measure performance for generate exec operator(with codegen).
+ * To run this benchmark:
+ * {{{
+ *   1. without sbt: bin/spark-submit --class <this class> <spark sql test jar>
+ *   2. build/sbt "sql/test:runMain <this class>"
+ *   3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain <this class>"
+ *      Results will be written to "benchmarks/GenerateExecBenchmark-results.txt".
+ * }}}
+ */
+case class Data(value1: Float, value2: Map[String, String], value3: String)
+
+object GenerateExecBenchmark extends SqlBasedBenchmark {
+
+  override def runBenchmarkSuite(mainArgs: Array[String]): Unit = {
+    runBenchmark("GenerateExec benchmark") {
+      import spark.implicits._
+      val numRecords = 100000000
+      codegenBenchmark("GenerateExec Benchmark", numRecords) {
+            val srcDF = spark.range(numRecords).map {

Review comment:
       nit: wrong indents.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-791172587


   can you address https://github.com/apache/spark/pull/28715#discussion_r585746818 ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802204367


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40799/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642449582


   **[Test build #123828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123828/testReport)** for PR 28715 at commit [`bdbaa6b`](https://github.com/apache/spark/commit/bdbaa6b2becc311bde4a342668e4501325b6aa28).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667707033






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667637956


   **[Test build #126929 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126929/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com/apache/spark/commit/d6f8459556d864b3d2249f45f73332812659dba0).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-690591988






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642353999


   **[Test build #123791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123791/testReport)** for PR 28715 at commit [`caf0ce4`](https://github.com/apache/spark/commit/caf0ce401722c2e197e9e4bc4a11832f698acd37).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802128348


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136217/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-641875211






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-796299529


   **[Test build #135949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135949/testReport)** for PR 28715 at commit [`9a34cfa`](https://github.com/apache/spark/commit/9a34cfae0376bf7d6873a9e63ee020243e048387).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-791648531


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40401/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-644557745


   **[Test build #124100 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124100/testReport)** for PR 28715 at commit [`a85f511`](https://github.com/apache/spark/commit/a85f51121ede087d6a943275e832ba290caa0966).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642319657






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642999857






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667625572






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-647234069


   @maropu @cloud-fan Can you please help review the changes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r440984034



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -226,14 +226,19 @@ case class GenerateExec(
       "0"
     }
     val numOutput = metricTerm(ctx, "numOutputRows")
+    val requiredInput = {
+      input.zipWithIndex.flatMap { case (ev, i) =>
+        if ((parent.inputSet).contains(child.output(i))) Seq(ev) else Seq()
+      }
+    } ++ position ++ values

Review comment:
       Done with slight modification.. Please check

##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,86 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("SPARK-31897: GenerateExec should be included in WholeStageCodegen") {

Review comment:
       done

##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,86 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("SPARK-31897: GenerateExec should be included in WholeStageCodegen") {
+    import testImplicits._
+    val arrayData = Seq(("James", Seq("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown")))
+    val df = arrayData.toDF("name", "knownLanguages", "properties")
+
+    // Array - explode
+    var expDF = df.select($"name", explode($"knownLanguages"), $"properties")
+    var plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ => false
+    }.isDefined)
+    checkAnswer(expDF, Array(Row("James", "Java", Map("hair" -> "black", "eye" -> "brown")),
+      Row("James", "Scala", Map("hair" -> "black", "eye" -> "brown"))))
+
+
+    // Map - explode
+    expDF = df.select($"name", $"knownLanguages", explode($"properties"))
+    plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ => false
+    }.isDefined)
+    checkAnswer(expDF,
+      Array(Row("James", List("Java", "Scala"), "hair", "black"),
+        Row("James", List("Java", "Scala"), "eye", "brown")))
+
+    // Array - posexplode
+    expDF = df.select($"name", posexplode($"knownLanguages"))
+    plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ => false
+    }.isDefined)
+    checkAnswer(expDF,
+      Array(Row("James", 0, "Java"), Row("James", 1, "Scala")))
+
+    // Map - posexplode
+    expDF = df.select($"name", posexplode($"properties"))
+    plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ => false
+    }.isDefined)
+    checkAnswer(expDF,
+      Array(Row("James", 0, "hair", "black"), Row("James", 1, "eye", "brown")))
+

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-644574864


   **[Test build #124100 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124100/testReport)** for PR 28715 at commit [`a85f511`](https://github.com/apache/spark/commit/a85f51121ede087d6a943275e832ba290caa0966).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642320112


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-641926223


   **[Test build #123751 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123751/testReport)** for PR 28715 at commit [`55c13c2`](https://github.com/apache/spark/commit/55c13c2260aabb010bd9a08f7b14a27f209bce8f).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r440609572



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -226,18 +226,33 @@ case class GenerateExec(
       "0"
     }
     val numOutput = metricTerm(ctx, "numOutputRows")
+    val requiredInput =
+      evaluateRequiredVariablesExpr(child.output,
+        input, parent.inputSet) ++ position ++ values
     s"""
        |${data.code}
        |$initMapData
        |int $numElements = ${data.isNull} ? 0 : ${data.value}.numElements();
        |for (int $index = $init; $index < $numElements; $index++) {
        |  $numOutput.add(1);
        |  $updateRowData
-       |  ${consume(ctx, input ++ position ++ values)}
+       |  ${consume(ctx, requiredInput)}
        |}
      """.stripMargin
   }
 
+  /**
+   * Returns [ExprCode] for required attributes
+   */
+  private def evaluateRequiredVariablesExpr(

Review comment:
       made the code inline.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on a change in pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r438458045



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,77 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("GenerateExec should be included in WholeStageCodegen") {
+    val arrayData1 = Seq(
+      Row("James", List("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown"))
+    )
+
+    val arraySchema1 = new StructType()
+      .add("name", StringType)
+      .add("knownLanguages", ArrayType(StringType))
+      .add("properties", MapType(StringType, StringType))
+
+    val df = spark.createDataFrame(spark.sparkContext.parallelize(arrayData1), arraySchema1)
+
+    // Array - explode
+    var expDF = df.select($"name", explode($"knownLanguages"), $"properties")
+    var plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ =>
+        false

Review comment:
       Fixed

##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,77 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("GenerateExec should be included in WholeStageCodegen") {
+    val arrayData1 = Seq(
+      Row("James", List("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown"))
+    )
+
+    val arraySchema1 = new StructType()
+      .add("name", StringType)
+      .add("knownLanguages", ArrayType(StringType))
+      .add("properties", MapType(StringType, StringType))
+
+    val df = spark.createDataFrame(spark.sparkContext.parallelize(arrayData1), arraySchema1)

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642454630


   **[Test build #123828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123828/testReport)** for PR 28715 at commit [`bdbaa6b`](https://github.com/apache/spark/commit/bdbaa6b2becc311bde4a342668e4501325b6aa28).
    * This patch **fails due to an unknown error code, -9**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802173457


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136222/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642436045






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642999598


   **[Test build #123873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123873/testReport)** for PR 28715 at commit [`8cc5394`](https://github.com/apache/spark/commit/8cc5394e812ab33854f6be27e4469eadfc4d5419).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r588877871



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -144,9 +144,12 @@ case class GenerateExec(
       Seq.empty
     }
 
+    val requiredInput = input.zip(child.output).filter {

Review comment:
       ```
       val requiredInput = if (requiredChildOutput.nonEmpty) {
           input.zip(child.output).filter {
           case(_, output) => parent.inputSet.contains(output)
         }.map(_._1)
       } else {
         Nil
       }
   ```
   ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-796425123


   **[Test build #135949 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135949/testReport)** for PR 28715 at commit [`9a34cfa`](https://github.com/apache/spark/commit/9a34cfae0376bf7d6873a9e63ee020243e048387).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `case class Data(value1: Float, value2: Map[String, String], value3: String)`
     * `class GenerateExecBenchmark extends SqlBasedBenchmark `


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r596211064



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/GenerateExecBenchmark.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution

Review comment:
       I think benchmark suite should be in there.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642320117


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123792/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-641926524


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] tanelk commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
tanelk commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-787077864


   @maropu or @cloud-fan , could you consider reopeing this PR?
   It has minimal code change with possibly major performance improvements. 
   You two already have reviewed it quite thoroughly and @karuppayya has fixed all the issues.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-657163441


   **[Test build #125703 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125703/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com/apache/spark/commit/d6f8459556d864b3d2249f45f73332812659dba0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642315444






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-643078708






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-690175382


   **[Test build #128512 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128512/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com/apache/spark/commit/d6f8459556d864b3d2249f45f73332812659dba0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r440636308



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,86 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("SPARK-31897: GenerateExec should be included in WholeStageCodegen") {
+    import testImplicits._
+    val arrayData = Seq(("James", Seq("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown")))
+    val df = arrayData.toDF("name", "knownLanguages", "properties")
+
+    // Array - explode
+    var expDF = df.select($"name", explode($"knownLanguages"), $"properties")
+    var plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ => false
+    }.isDefined)
+    checkAnswer(expDF, Array(Row("James", "Java", Map("hair" -> "black", "eye" -> "brown")),
+      Row("James", "Scala", Map("hair" -> "black", "eye" -> "brown"))))
+
+
+    // Map - explode
+    expDF = df.select($"name", $"knownLanguages", explode($"properties"))
+    plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ => false
+    }.isDefined)
+    checkAnswer(expDF,
+      Array(Row("James", List("Java", "Scala"), "hair", "black"),
+        Row("James", List("Java", "Scala"), "eye", "brown")))
+
+    // Array - posexplode
+    expDF = df.select($"name", posexplode($"knownLanguages"))
+    plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ => false
+    }.isDefined)
+    checkAnswer(expDF,
+      Array(Row("James", 0, "Java"), Row("James", 1, "Scala")))
+
+    // Map - posexplode
+    expDF = df.select($"name", posexplode($"properties"))
+    plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ => false
+    }.isDefined)
+    checkAnswer(expDF,
+      Array(Row("James", 0, "hair", "black"), Row("James", 1, "eye", "brown")))
+

Review comment:
       nit the single blank is enough?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-644260009


   @cloud-fan @maropu I have fixed the tests. Can you please take a look again. Thanks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r440609352



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -226,18 +226,33 @@ case class GenerateExec(
       "0"
     }
     val numOutput = metricTerm(ctx, "numOutputRows")
+    val requiredInput =
+      evaluateRequiredVariablesExpr(child.output,
+        input, parent.inputSet) ++ position ++ values

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-787693106






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-788454084


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135608/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-788238766


   **[Test build #135608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135608/testReport)** for PR 28715 at commit [`214d699`](https://github.com/apache/spark/commit/214d6993af9d69b6a9d3a6b83833a866150a6184).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-797160168


   **[Test build #135984 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135984/testReport)** for PR 28715 at commit [`f2d8de5`](https://github.com/apache/spark/commit/f2d8de590bedd4b60364c563c220e129564e5b20).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642354095


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123791/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642792690


   **[Test build #123862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123862/testReport)** for PR 28715 at commit [`4eb2195`](https://github.com/apache/spark/commit/4eb219527afcf3924d8f6b5934364e3e41061b35).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667638045


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642315444






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642454687


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123828/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-790208898


   **[Test build #135725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135725/testReport)** for PR 28715 at commit [`ce9cd00`](https://github.com/apache/spark/commit/ce9cd000565497af71e189ad83f6d8f9d50ac271).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-790224635


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40307/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-790315786


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135725/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-801637594


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40772/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-657183621






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r437987463



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,77 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("GenerateExec should be included in WholeStageCodegen") {
+    val arrayData1 = Seq(
+      Row("James", List("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown"))
+    )
+
+    val arraySchema1 = new StructType()
+      .add("name", StringType)
+      .add("knownLanguages", ArrayType(StringType))
+      .add("properties", MapType(StringType, StringType))
+
+    val df = spark.createDataFrame(spark.sparkContext.parallelize(arrayData1), arraySchema1)
+
+    // Array - explode
+    var expDF = df.select($"name", explode($"knownLanguages"), $"properties")
+    var plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ =>
+        false
+    }.isDefined)
+    var results = expDF.collect()
+    assert(results.size ==2 && results ===

Review comment:
       you don't need to check the size, the `===` already checks it.

##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,77 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("GenerateExec should be included in WholeStageCodegen") {
+    val arrayData1 = Seq(
+      Row("James", List("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown"))
+    )
+
+    val arraySchema1 = new StructType()
+      .add("name", StringType)
+      .add("knownLanguages", ArrayType(StringType))
+      .add("properties", MapType(StringType, StringType))
+
+    val df = spark.createDataFrame(spark.sparkContext.parallelize(arrayData1), arraySchema1)
+
+    // Array - explode
+    var expDF = df.select($"name", explode($"knownLanguages"), $"properties")
+    var plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ =>
+        false
+    }.isDefined)
+    var results = expDF.collect()
+    assert(results.size ==2 && results ===
+      Array(Row("James", "Java", Map("hair" -> "black", "eye" -> "brown")),
+        Row("James", "Scala", Map("hair" -> "black", "eye" -> "brown"))))
+
+    // Map - explode
+    expDF = df.select($"name", $"knownLanguages", explode($"properties"))
+    plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ =>
+        false
+    }.isDefined)
+    results = expDF.collect()
+    assert(results.size ==2 && results ===

Review comment:
       ditto




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya edited a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
viirya edited a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802140598


   <del>Jenkins seems unstable. @cloud-fan @maropu Do you know if the Scala 2.13 SBT build failure is known issue? If so, as GA passed all others, I think this should be okay to merge?</del>
   
   The Scala 2.13 SBT build error seems fixed now. Re-triggered GA.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802174714


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40804/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802075305


   Don't see Jenkins test. GA passed all except for Scala 2.13 build with SBT but it looks unrelated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r592826875



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/GenerateExecBenchmark.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import org.apache.spark.sql.execution.benchmark.SqlBasedBenchmark
+import org.apache.spark.sql.functions.explode
+
+/**
+ * Benchmark to measure performance for generate exec operator(with codegen).

Review comment:
       nit: remove `(with codegen)`? This code runs benchmarks both with/without codegen.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r584484987



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -124,7 +124,7 @@ case class GenerateExec(
     }
   }
 
-  override def supportCodegen: Boolean = false
+  override def supportCodegen: Boolean = super.supportCodegen && generator.supportCodegen

Review comment:
       Why we need `super.supportCodegen`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-644871681


   **[Test build #124132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124132/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com/apache/spark/commit/d6f8459556d864b3d2249f45f73332812659dba0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642516021


   **[Test build #123840 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123840/testReport)** for PR 28715 at commit [`bdbaa6b`](https://github.com/apache/spark/commit/bdbaa6b2becc311bde4a342668e4501325b6aa28).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642382760


   **[Test build #123812 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123812/testReport)** for PR 28715 at commit [`bdbaa6b`](https://github.com/apache/spark/commit/bdbaa6b2becc311bde4a342668e4501325b6aa28).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-787676740


   **[Test build #135578 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135578/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com/apache/spark/commit/d6f8459556d864b3d2249f45f73332812659dba0).
    * This patch **fails to build**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r585743800



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -124,7 +124,7 @@ case class GenerateExec(
     }
   }
 
-  override def supportCodegen: Boolean = false
+  override def supportCodegen: Boolean = super.supportCodegen && generator.supportCodegen

Review comment:
       same question




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-644546701


   @maropu Updated the description with benchmarks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642793344






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802174714


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40804/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on a change in pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r438569824



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -226,18 +226,33 @@ case class GenerateExec(
       "0"
     }
     val numOutput = metricTerm(ctx, "numOutputRows")
+    val requiredInput =
+      evaluateRequiredVariablesExpr(child.output,
+        input, parent.inputSet) ++ position ++ values
     s"""
        |${data.code}
        |$initMapData
        |int $numElements = ${data.isNull} ? 0 : ${data.value}.numElements();
        |for (int $index = $init; $index < $numElements; $index++) {
        |  $numOutput.add(1);
        |  $updateRowData
-       |  ${consume(ctx, input ++ position ++ values)}
+       |  ${consume(ctx, requiredInput)}
        |}
      """.stripMargin
   }
 
+  /**
+   * Returns [ExprCode] for required attributes
+   */
+  private def evaluateRequiredVariablesExpr(

Review comment:
       @cloud-fan Should this method be in introduced in WholeStageCodeGenExec.scala?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642325411






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-690589477


   **[Test build #128512 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128512/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com/apache/spark/commit/d6f8459556d864b3d2249f45f73332812659dba0).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-690148236






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on a change in pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r438457836



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,77 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("GenerateExec should be included in WholeStageCodegen") {
+    val arrayData1 = Seq(
+      Row("James", List("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown"))
+    )
+
+    val arraySchema1 = new StructType()
+      .add("name", StringType)
+      .add("knownLanguages", ArrayType(StringType))
+      .add("properties", MapType(StringType, StringType))
+
+    val df = spark.createDataFrame(spark.sparkContext.parallelize(arrayData1), arraySchema1)
+
+    // Array - explode
+    var expDF = df.select($"name", explode($"knownLanguages"), $"properties")
+    var plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ =>
+        false
+    }.isDefined)
+    var results = expDF.collect()
+    assert(results.size ==2 && results ===
+      Array(Row("James", "Java", Map("hair" -> "black", "eye" -> "brown")),
+        Row("James", "Scala", Map("hair" -> "black", "eye" -> "brown"))))
+
+    // Map - explode
+    expDF = df.select($"name", $"knownLanguages", explode($"properties"))
+    plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ =>
+        false
+    }.isDefined)
+    results = expDF.collect()
+    assert(results.size ==2 && results ===
+      Array(Row("James", List("Java", "Scala"), "hair", "black"),
+        Row("James", List("Java", "Scala"), "eye", "brown")))
+
+    // Array - posexplode
+    expDF = df.select($"name", posexplode($"knownLanguages"))
+    plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ =>
+        false
+    }.isDefined)
+    results = expDF.collect()
+    assert(results.size ==2 && results ===
+      Array(Row("James", 0, "Java"),
+        Row("James", 1, "Scala")))
+
+    // Map - posexplode
+    expDF = df.select($"name", posexplode($"properties"))
+    plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ =>
+        false
+    }.isDefined)
+    results = expDF.collect()
+    assert(results.size ==2 && results ===
+      Array(Row("James", 0, "hair", "black"),
+        Row("James", 1, "eye", "brown")))
+
+

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802172532


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40799/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-657163572






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-797145478


   @tanelk  I have added the benchmark results.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-641875211






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-788319196


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40188/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r589256202



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -144,9 +144,12 @@ case class GenerateExec(
       Seq.empty
     }
 
+    val requiredInput = input.zip(child.output).filter {
+      case(_, output) => parent.inputSet.contains(output)
+    }.map(_._1)
     boundGenerator match {
-      case e: CollectionGenerator => codeGenCollection(ctx, e, values, row)
-      case g => codeGenTraversableOnce(ctx, g, values, row)
+      case e: CollectionGenerator => codeGenCollection(ctx, e, requiredInput, row)

Review comment:
       can we also update the definition of `codeGenCollection` and rename the parameter `input` to `requiredInput`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-641874474


   **[Test build #123751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123751/testReport)** for PR 28715 at commit [`55c13c2`](https://github.com/apache/spark/commit/55c13c2260aabb010bd9a08f7b14a27f209bce8f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-790315786


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135725/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642997347


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642999857






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-788149680


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40184/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-797160168


   **[Test build #135984 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135984/testReport)** for PR 28715 at commit [`f2d8de5`](https://github.com/apache/spark/commit/f2d8de590bedd4b60364c563c220e129564e5b20).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-801605838


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136190/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-790247352


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40307/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802077141


   retest this please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642446974


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-793271717






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642792690


   **[Test build #123862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123862/testReport)** for PR 28715 at commit [`4eb2195`](https://github.com/apache/spark/commit/4eb219527afcf3924d8f6b5934364e3e41061b35).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642679308






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802138771


   retest this please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-788159133


   **[Test build #135603 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135603/testReport)** for PR 28715 at commit [`d6640de`](https://github.com/apache/spark/commit/d6640ded2604294cd58b37b6c9f2f14b05676a66).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802530100


   Oh, the same Scala 2.13 SBT build error. I think it's because this doesn't sync up with latest master. 
   
   It is unrelated so I'm going to merge this to master.
   
   Thanks @karuppayya for working on this and all for the review!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-641872311


   ok to test


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642436049


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123812/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-637957469


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-644872445






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642435548


   **[Test build #123812 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123812/testReport)** for PR 28715 at commit [`bdbaa6b`](https://github.com/apache/spark/commit/bdbaa6b2becc311bde4a342668e4501325b6aa28).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667731207


   **[Test build #126942 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126942/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com/apache/spark/commit/d6f8459556d864b3d2249f45f73332812659dba0).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-801605838


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136190/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-797168556


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40567/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-801244983


   LGTM, @maropu can you do the final sign-off?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667706833


   **[Test build #126942 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126942/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com/apache/spark/commit/d6f8459556d864b3d2249f45f73332812659dba0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642882127


   **[Test build #123862 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123862/testReport)** for PR 28715 at commit [`4eb2195`](https://github.com/apache/spark/commit/4eb219527afcf3924d8f6b5934364e3e41061b35).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-790208898


   **[Test build #135725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135725/testReport)** for PR 28715 at commit [`ce9cd00`](https://github.com/apache/spark/commit/ce9cd000565497af71e189ad83f6d8f9d50ac271).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-657183621






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] kiszk commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667625363


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-643078161


   **[Test build #123873 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123873/testReport)** for PR 28715 at commit [`8cc5394`](https://github.com/apache/spark/commit/8cc5394e812ab33854f6be27e4469eadfc4d5419).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-639899181


   @viirya @cloud-fan Can you please help review this PR


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642319332


   **[Test build #123792 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123792/testReport)** for PR 28715 at commit [`869330d`](https://github.com/apache/spark/commit/869330d23eb4e159ad8e98bb9656e3de3d9ba31e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-644555570






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-641926524






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667707033






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642436045


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r437986642



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,77 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("GenerateExec should be included in WholeStageCodegen") {
+    val arrayData1 = Seq(
+      Row("James", List("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown"))
+    )
+
+    val arraySchema1 = new StructType()
+      .add("name", StringType)
+      .add("knownLanguages", ArrayType(StringType))
+      .add("properties", MapType(StringType, StringType))
+
+    val df = spark.createDataFrame(spark.sparkContext.parallelize(arrayData1), arraySchema1)
+
+    // Array - explode
+    var expDF = df.select($"name", explode($"knownLanguages"), $"properties")
+    var plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ =>
+        false

Review comment:
       nit: `case _ => false`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-801648211


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40772/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802128348


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136217/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642387587


   **[Test build #123798 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123798/testReport)** for PR 28715 at commit [`a9bd9db`](https://github.com/apache/spark/commit/a9bd9dbc8b2dfc67bc9bd41f773c68f427e6a309).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642517025






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-641874474


   **[Test build #123751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123751/testReport)** for PR 28715 at commit [`55c13c2`](https://github.com/apache/spark/commit/55c13c2260aabb010bd9a08f7b14a27f209bce8f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642354085






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r437987683



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,77 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("GenerateExec should be included in WholeStageCodegen") {
+    val arrayData1 = Seq(
+      Row("James", List("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown"))
+    )
+
+    val arraySchema1 = new StructType()
+      .add("name", StringType)
+      .add("knownLanguages", ArrayType(StringType))
+      .add("properties", MapType(StringType, StringType))
+
+    val df = spark.createDataFrame(spark.sparkContext.parallelize(arrayData1), arraySchema1)
+
+    // Array - explode
+    var expDF = df.select($"name", explode($"knownLanguages"), $"properties")
+    var plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ =>
+        false
+    }.isDefined)
+    var results = expDF.collect()
+    assert(results.size ==2 && results ===
+      Array(Row("James", "Java", Map("hair" -> "black", "eye" -> "brown")),
+        Row("James", "Scala", Map("hair" -> "black", "eye" -> "brown"))))
+
+    // Map - explode
+    expDF = df.select($"name", $"knownLanguages", explode($"properties"))
+    plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ =>
+        false
+    }.isDefined)
+    results = expDF.collect()
+    assert(results.size ==2 && results ===
+      Array(Row("James", List("Java", "Scala"), "hair", "black"),
+        Row("James", List("Java", "Scala"), "eye", "brown")))
+
+    // Array - posexplode
+    expDF = df.select($"name", posexplode($"knownLanguages"))
+    plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ =>
+        false
+    }.isDefined)
+    results = expDF.collect()
+    assert(results.size ==2 && results ===
+      Array(Row("James", 0, "Java"),
+        Row("James", 1, "Scala")))
+
+    // Map - posexplode
+    expDF = df.select($"name", posexplode($"properties"))
+    plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ =>
+        false
+    }.isDefined)
+    results = expDF.collect()
+    assert(results.size ==2 && results ===
+      Array(Row("James", 0, "hair", "black"),
+        Row("James", 1, "eye", "brown")))
+
+

Review comment:
       nit: remove blank lines




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-791799869


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135819/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-797164953


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40567/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-788450177


   **[Test build #135608 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135608/testReport)** for PR 28715 at commit [`214d699`](https://github.com/apache/spark/commit/214d6993af9d69b6a9d3a6b83833a866150a6184).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] tanelk commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
tanelk commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-796980570


   @karuppayya , could you add the benchmark results to the PR as a file in `sql/core/benchmarks/`.
   You can generate it like this (copied form the `JoinBenchmark` docstring):
   ``` 
   *   3. generate result:
    *      SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain <this class>"
    *      Results will be written to "benchmarks/JoinBenchmark-results.txt".
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-790240134


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40307/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-793435653


   BTW, although there is performance benchmark in the PR description, it will not be tracked later. I think it may be good to have a benchmark suite for Generate codegen on/off too. We can follow other benchmark suite.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642383840






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r440636071



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,86 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("SPARK-31897: GenerateExec should be included in WholeStageCodegen") {
+    import testImplicits._
+    val arrayData = Seq(("James", Seq("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown")))
+    val df = arrayData.toDF("name", "knownLanguages", "properties")
+
+    // Array - explode
+    var expDF = df.select($"name", explode($"knownLanguages"), $"properties")
+    var plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ => false
+    }.isDefined)
+    checkAnswer(expDF, Array(Row("James", "Java", Map("hair" -> "black", "eye" -> "brown")),
+      Row("James", "Scala", Map("hair" -> "black", "eye" -> "brown"))))
+
+

Review comment:
       nit : plz remove the one blank.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-797243480


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135984/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r440635929



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,86 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("SPARK-31897: GenerateExec should be included in WholeStageCodegen") {

Review comment:
       nit: please the prefix `SPARK-31897` because this is not a bug.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-644466372


   Could you show us performance gains by enabling this codegen?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642319657






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-797152266


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135983/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-797149047


   Looks okay otherwise.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-644574965






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r584503039



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,84 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("GenerateExec should be included in WholeStageCodegen") {
+    import testImplicits._
+    val arrayData = Seq(("James", Seq("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown")))
+    val df = arrayData.toDF("name", "knownLanguages", "properties")
+
+    // Array - explode
+    var expDF = df.select($"name", explode($"knownLanguages"), $"properties")
+    var plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ => false
+    }.isDefined)
+    checkAnswer(expDF, Array(Row("James", "Java", Map("hair" -> "black", "eye" -> "brown")),

Review comment:
       Could you use `testWithWholeStageCodegenOnAndOff` to check if queries work correctly with/without codegen?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802140598


   Jenkins seems unstable. @cloud-fan @maropu Do you know if the Scala 2.13 SBT build failure is known issue? If so, as GA passed all others, I think this should be okay to merge?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-644555570






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-644557745


   **[Test build #124100 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124100/testReport)** for PR 28715 at commit [`a85f511`](https://github.com/apache/spark/commit/a85f51121ede087d6a943275e832ba290caa0966).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-788238766


   **[Test build #135608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135608/testReport)** for PR 28715 at commit [`214d699`](https://github.com/apache/spark/commit/214d6993af9d69b6a9d3a6b83833a866150a6184).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802204367


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40799/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r588877950



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -144,9 +144,12 @@ case class GenerateExec(
       Seq.empty
     }

Review comment:
       The code in L141-145 is not needed now.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] kiszk commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-657162781


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-801233015


   @cloud-fan @maropu @tanelk PR is ready to merge.  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-801514289


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136181/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-788180196


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135603/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r437987021



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,77 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("GenerateExec should be included in WholeStageCodegen") {
+    val arrayData1 = Seq(
+      Row("James", List("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown"))
+    )
+
+    val arraySchema1 = new StructType()
+      .add("name", StringType)
+      .add("knownLanguages", ArrayType(StringType))
+      .add("properties", MapType(StringType, StringType))
+
+    val df = spark.createDataFrame(spark.sparkContext.parallelize(arrayData1), arraySchema1)
+
+    // Array - explode
+    var expDF = df.select($"name", explode($"knownLanguages"), $"properties")
+    var plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ =>
+        false
+    }.isDefined)
+    var results = expDF.collect()
+    assert(results.size ==2 && results ===

Review comment:
       please use `checkAnswer` to check result.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-787652913


   @maropu can you take another look?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-643078708






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802191695


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40799/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-790247352


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40307/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-797152266






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-791648531


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40401/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667625475


   **[Test build #126929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126929/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com/apache/spark/commit/d6f8459556d864b3d2249f45f73332812659dba0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-670251884


   @maropu @cloud-fan @kiszk
   Can you please help review this change?
   Do you see any issues with the PR
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642516021


   **[Test build #123840 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123840/testReport)** for PR 28715 at commit [`bdbaa6b`](https://github.com/apache/spark/commit/bdbaa6b2becc311bde4a342668e4501325b6aa28).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642882462






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] viirya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-801594072


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667731372






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-638637169


   @dongjoon-hyun @gatorsmile @cloud-fan can anyone help review this change? Thanks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r440526114



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
##########
@@ -51,6 +51,93 @@ class WholeStageCodegenSuite extends QueryTest with SharedSparkSession
     assert(df.collect() === Array(Row(9, 4.5)))
   }
 
+  test("SPARK-31897: GenerateExec should be included in WholeStageCodegen") {
+    import testImplicits._
+    val arrayData = Seq(("James", Seq("Java", "Scala"), Map("hair" -> "black", "eye" -> "brown")))
+    val df = arrayData.toDF("name", "knownLanguages", "properties")
+
+    // Array - explode
+    var expDF = df.select($"name", explode($"knownLanguages"), $"properties")
+    var plan = expDF.queryExecution.executedPlan
+    assert(plan.find {
+      case stage: WholeStageCodegenExec =>
+        stage.find(_.isInstanceOf[GenerateExec]).isDefined
+      case _ => false
+    }.isDefined)
+    var results = expDF.collect()
+    assert(results ===
+      Array(Row("James", "Java", Map("hair" -> "black", "eye" -> "brown")),
+        Row("James", "Scala", Map("hair" -> "black", "eye" -> "brown"))))

Review comment:
       Plz use `checkAnswer` instead.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667625475


   **[Test build #126929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126929/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com/apache/spark/commit/d6f8459556d864b3d2249f45f73332812659dba0).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-637957469


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642317273


   **[Test build #123791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123791/testReport)** for PR 28715 at commit [`caf0ce4`](https://github.com/apache/spark/commit/caf0ce401722c2e197e9e4bc4a11832f698acd37).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642354085


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-802173457


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136222/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-788149703


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40184/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r438816770



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -124,7 +124,7 @@ case class GenerateExec(
     }
   }
 
-  override def supportCodegen: Boolean = false
+  override def supportCodegen: Boolean = true

Review comment:
       shall we also check `generator.supportCodegen`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-637957851


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642320112






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-637957851


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642450214






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642388033


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123798/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] kiszk commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-690147782


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-645008292






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642677935


   **[Test build #123840 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123840/testReport)** for PR 28715 at commit [`bdbaa6b`](https://github.com/apache/spark/commit/bdbaa6b2becc311bde4a342668e4501325b6aa28).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-642511892


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667638045






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28715:
URL: https://github.com/apache/spark/pull/28715#discussion_r440635513



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala
##########
@@ -226,14 +226,19 @@ case class GenerateExec(
       "0"
     }
     val numOutput = metricTerm(ctx, "numOutputRows")
+    val requiredInput = {
+      input.zipWithIndex.flatMap { case (ev, i) =>
+        if ((parent.inputSet).contains(child.output(i))) Seq(ev) else Seq()
+      }
+    } ++ position ++ values

Review comment:
       more style nit: how about reformatting it ilke this?
   ```
       val requiredOutputVars = input.zip(child.output).filter(ev => parent.inputSet.contains(ev._2))
         .map(_._1) ++ position ++ values
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-645007466


   **[Test build #124132 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124132/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com/apache/spark/commit/d6f8459556d864b3d2249f45f73332812659dba0).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] karuppayya commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
karuppayya commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-637958350


   @rxin The codegen was disabled in https://github.com/apache/spark/commit/b5c5bd98ea5e8dbfebcf86c5459bdf765f5ceb53. If you happen to remember the scenario for workload breaking, I can check if it works now and add a test case as well. Thanks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667638049


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126929/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] kiszk commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-667706719


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28715:
URL: https://github.com/apache/spark/pull/28715#issuecomment-690591988






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org