You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by viirya <gi...@git.apache.org> on 2018/05/23 03:26:38 UTC
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/21405
[SPARK-24361][SQL] Polish code block manipulation API
## What changes were proposed in this pull request?
Current code block manipulation API is immature and hacky. We need a formal API to manipulate code blocks.
The basic idea is making `JavaCode` as `TreeNode`. So we can use familiar `transform` API to manipulate code blocks and expressions in code blocks.
For example, we can replace `SimpleExprValue` in a code block like this:
```scala
code.transformExprValues {
case SimpleExprValue("1 + 1", _) => aliasedParam
}
```
## How was this patch tested?
Added unite tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 codeblock-api
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21405.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21405
----
commit 1df9943915b2c3df0945d4def8282a3159f584bf
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-04-30T13:29:07Z
Add API for handling expression code generation.
commit 5fe425c2d2837f00bdfe9ba5e6f446829fba32c1
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-01T15:06:32Z
Add new abstraction for expression codegen.
commit 00bef6bb2559d93e68266d2605ef4403f4d8d2af
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-03T11:13:15Z
Add basic tests.
commit 5d9c454a34401f1a0c2648a17e3a811b1f452024
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-03T14:53:42Z
Merge remote-tracking branch 'upstream/master' into SPARK-24121
commit 162deb2f4de0b4d6fee3641ce79fbab424dd8e9b
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-03T14:54:10Z
Deal merging conflict.
commit d138ee05541bc1976442c5f1170ba7a7f3f5114c
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-04T01:12:46Z
Address comments and add more tests.
commit ee9a4c03a2da8a01f588c1283500f3412e356875
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-05T04:33:21Z
Address comments.
commit e7cfa286b6c4f812bcbae3d59ec24ae956d9e330
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-05T13:44:44Z
Remove JavaCode.block. We should always use code string interpolator for constructing code block.
commit 5945c156b2df7aa2f61df57d86f07279b6959e2a
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-05T14:00:43Z
We should not implicitly convert code block to string. Otherwise we may not aware of necessary block inputs.
commit 2b30654bc50a39a8af597df68ba288280299defe
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-05T22:54:54Z
Address comment and trim expected code string.
commit aff411bbf21f484c20588997d98682f2ec77191a
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-08T03:24:04Z
Remove unused import.
commit 53b329a3b4e72d423cf503af2982d545848bece8
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-08T05:33:35Z
Address comments.
commit 72faac3209beb8bc38938f8788de6338e9b2ffae
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-09T13:37:29Z
Address some comments.
commit ffbf4abcd4212a24bed6ebeebab8bd8d86beb699
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-17T09:37:57Z
Merge remote-tracking branch 'upstream/master' into SPARK-24121
commit d04067635ca5785e262afb65781faab5ca737d93
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-17T09:38:59Z
Use code block for newly merged codegen.
commit c378ce2a1982b2c5453b15fbc1de06851033ac1d
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-17T12:55:31Z
Use Set as method exprValues method returning type.
commit 2ca974190527f3ac1ade1feddf9bd2b953fabdd7
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-19T09:31:51Z
Address comments.
commit d91f11199cc8b3016465962a8bc4cc886512c484
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-19T13:10:32Z
Merge remote-tracking branch 'upstream/master' into SPARK-24121
commit 5c4023353b74d1c28b8ab3caf22498ca8b05d097
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-21T06:20:58Z
Merge remote-tracking branch 'upstream/master' into codeblock-api
commit 8d0b1b9d9dc6631ea787389bb9d3f561950c3d0b
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-21T09:19:09Z
Add API to manipulate blocks and expressions.
commit 2d77332d77d133be1ed1c2c155c9020462628c55
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-23T02:45:31Z
Merge remote-tracking branch 'upstream/master' into codeblock-api
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91021/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3493/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r190228283
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeBlockSuite.scala ---
@@ -120,11 +120,11 @@ class CodeBlockSuite extends SparkFunSuite {
|}""".stripMargin
val aliasedParam = JavaCode.variable("aliased", expr.javaType)
- val aliasedInputs = code.asInstanceOf[CodeBlock].blockInputs.map {
- case _: SimpleExprValue => aliasedParam
- case other => other
+
+ // We want to replace all occurrences of `expr` with the variable `aliasedParam`.
+ val aliasedCode = code.transformExprValues {
+ case SimpleExprValue("1 + 1", _) => aliasedParam
--- End diff --
nit: I know the current code works correctly. How about replacing `_` with `CodeGenerator.javaClass(IntegerType)`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21405
**[Test build #92642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92642/testReport)** for PR 21405 at commit [`c9dee44`](https://github.com/apache/spark/commit/c9dee44c6b4e9fdd41b2460fbd331f8f0e9235f4).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r200244488
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/javaCode.scala ---
@@ -118,10 +119,10 @@ object JavaCode {
/**
* A trait representing a block of java code.
*/
-trait Block extends JavaCode {
+trait Block extends TreeNode[Block] with JavaCode {
import Block._
- // The expressions to be evaluated inside this block.
+ // All expressions to be evaluated inside this block and underlying blocks.
def exprValues: Set[ExprValue]
--- End diff --
do we still need it?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21405
@cloud-fan I found that the way to collect method parameters in the use case can be simplified. Updated in the PR description. Please take a look when you have time. Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r200244308
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/javaCode.scala ---
@@ -291,5 +328,8 @@ class LiteralValue(val value: String, val javaType: Class[_]) extends ExprValue
override def hashCode(): Int = value.hashCode() * 31 + javaType.hashCode()
}
+case class LiteralExpr(override val value: String, override val javaType: Class[_])
--- End diff --
do we still need this change?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21405
**[Test build #91021 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91021/testReport)** for PR 21405 at commit [`2d77332`](https://github.com/apache/spark/commit/2d77332d77d133be1ed1c2c155c9020462628c55).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21405
**[Test build #91031 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91031/testReport)** for PR 21405 at commit [`2d77332`](https://github.com/apache/spark/commit/2d77332d77d133be1ed1c2c155c9020462628c55).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21405
**[Test build #91014 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91014/testReport)** for PR 21405 at commit [`2d77332`](https://github.com/apache/spark/commit/2d77332d77d133be1ed1c2c155c9020462628c55).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21405
ping @cloud-fan @hvanhovell @kiszk
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21405
**[Test build #91251 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91251/testReport)** for PR 21405 at commit [`e30be7a`](https://github.com/apache/spark/commit/e30be7a3da9061a71e8ac222006e20b4e173cb74).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r200245418
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeBlockSuite.scala ---
@@ -133,4 +133,55 @@ class CodeBlockSuite extends SparkFunSuite {
|}""".stripMargin
assert(aliasedCode.toString == expected.toString)
}
+
+ test ("transform expr in nested blocks") {
+ val expr = JavaCode.expression("1 + 1", IntegerType)
+ val isNull = JavaCode.isNullVariable("expr1_isNull")
--- End diff --
not related to this PR, shall we remove `isNullVariable` and always use `expression(..., BooleanType)`? Or we can add a bunch of `intExpression(...)`, `booleanExpression(...)` etc.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r200259117
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeBlockSuite.scala ---
@@ -133,4 +133,55 @@ class CodeBlockSuite extends SparkFunSuite {
|}""".stripMargin
assert(aliasedCode.toString == expected.toString)
}
+
+ test ("transform expr in nested blocks") {
+ val expr = JavaCode.expression("1 + 1", IntegerType)
+ val isNull = JavaCode.isNullVariable("expr1_isNull")
--- End diff --
I think it is fine. I was thinking the name of `isNullVariable` is a bit confusing at the first look actually.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r200236745
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/javaCode.scala ---
@@ -22,14 +22,17 @@ import java.lang.{Boolean => JBool}
import scala.collection.mutable.ArrayBuffer
import scala.language.{existentials, implicitConversions}
+import org.apache.spark.sql.catalyst.trees.TreeNode
import org.apache.spark.sql.types.{BooleanType, DataType}
/**
* Trait representing an opaque fragments of java code.
*/
-trait JavaCode {
+trait JavaCode extends TreeNode[JavaCode] {
--- End diff --
Will update in next commit.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r200236723
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/javaCode.scala ---
@@ -22,14 +22,17 @@ import java.lang.{Boolean => JBool}
import scala.collection.mutable.ArrayBuffer
import scala.language.{existentials, implicitConversions}
+import org.apache.spark.sql.catalyst.trees.TreeNode
import org.apache.spark.sql.types.{BooleanType, DataType}
/**
* Trait representing an opaque fragments of java code.
*/
-trait JavaCode {
+trait JavaCode extends TreeNode[JavaCode] {
--- End diff --
I think it is ok. Currently `ExprValue` doesn't have to be `TreeNode`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r190497595
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/javaCode.scala ---
@@ -149,6 +152,34 @@ trait Block extends JavaCode {
// Concatenates this block with other block.
def + (other: Block): Block
+
+ /**
+ * Apply a map function to each java expression codes present in this java code, and return a new
+ * java code based on the mapped java expression codes.
+ */
+ def transformExprValues(f: PartialFunction[ExprValue, ExprValue]): this.type = {
--- End diff --
am I wrong or we are not updating the `exprValues` here?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r190502024
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/javaCode.scala ---
@@ -149,6 +152,34 @@ trait Block extends JavaCode {
// Concatenates this block with other block.
def + (other: Block): Block
+
+ /**
+ * Apply a map function to each java expression codes present in this java code, and return a new
+ * java code based on the mapped java expression codes.
+ */
+ def transformExprValues(f: PartialFunction[ExprValue, ExprValue]): this.type = {
--- End diff --
I see, can we add some tests for this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91251/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21405
@cloud-fan Thanks. I give a use case of splitting code into method in the PR description. I think it can show the basic idea.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21405
**[Test build #91014 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91014/testReport)** for PR 21405 at commit [`2d77332`](https://github.com/apache/spark/commit/2d77332d77d133be1ed1c2c155c9020462628c55).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21405
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21405
retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21405
we can do it later, when we do need `transformAllExprValues`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3489/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21405
can we give a concrete use case in the PR description? I think splitting code into methods is a good one, but we need to make it completed, e.g. how the method is generated.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r200247398
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeBlockSuite.scala ---
@@ -133,4 +133,55 @@ class CodeBlockSuite extends SparkFunSuite {
|}""".stripMargin
assert(aliasedCode.toString == expected.toString)
}
+
+ test ("transform expr in nested blocks") {
+ val expr = JavaCode.expression("1 + 1", IntegerType)
+ val isNull = JavaCode.isNullVariable("expr1_isNull")
+ val exprInFunc = JavaCode.variable("expr1", IntegerType)
+
+ val funcs = Seq("callFunc1", "callFunc2", "callFunc3")
+ val subBlocks = funcs.map { funcName =>
+ code"""
+ |$funcName(int $expr) {
+ | boolean $isNull = false;
+ | int $exprInFunc = $expr + 1;
+ |}""".stripMargin
+ }
+
+ val aliasedParam = JavaCode.variable("aliased", expr.javaType)
+
+ val block = code"${subBlocks(0)}\n${subBlocks(1)}\n${subBlocks(2)}"
+ val transformedBlock = block.transform {
+ case b: Block => b.transformExprValues {
--- End diff --
so the this should be
```
block.transformExprValues {
case SimpleExprValue ...
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r191372561
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/javaCode.scala ---
@@ -149,6 +152,34 @@ trait Block extends JavaCode {
// Concatenates this block with other block.
def + (other: Block): Block
+
+ /**
+ * Apply a map function to each java expression codes present in this java code, and return a new
+ * java code based on the mapped java expression codes.
+ */
+ def transformExprValues(f: PartialFunction[ExprValue, ExprValue]): this.type = {
--- End diff --
Added test for it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21405
@cloud-fan Updated. Do you think I should address the comment https://github.com/apache/spark/pull/21405#discussion_r200247248 to let `transformExprValues` also transforms child blocks?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/21405
@viirya sorry do you already have some use-cases, goals in mind for this API? May you please share them? Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r200249108
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/javaCode.scala ---
@@ -118,10 +119,10 @@ object JavaCode {
/**
* A trait representing a block of java code.
*/
-trait Block extends JavaCode {
+trait Block extends TreeNode[Block] with JavaCode {
import Block._
- // The expressions to be evaluated inside this block.
+ // All expressions to be evaluated inside this block and underlying blocks.
def exprValues: Set[ExprValue]
--- End diff --
We can remove it, as discussed in other PR previously.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r200227776
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/javaCode.scala ---
@@ -22,14 +22,17 @@ import java.lang.{Boolean => JBool}
import scala.collection.mutable.ArrayBuffer
import scala.language.{existentials, implicitConversions}
+import org.apache.spark.sql.catalyst.trees.TreeNode
import org.apache.spark.sql.types.{BooleanType, DataType}
/**
* Trait representing an opaque fragments of java code.
*/
-trait JavaCode {
+trait JavaCode extends TreeNode[JavaCode] {
--- End diff --
shall we only make `Block` extends `TreeNode`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91014/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21405
thanks, merging to master!
Looking forward to seeing how we can solve the method splitting problem with this infra :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r200249000
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeBlockSuite.scala ---
@@ -133,4 +133,55 @@ class CodeBlockSuite extends SparkFunSuite {
|}""".stripMargin
assert(aliasedCode.toString == expected.toString)
}
+
+ test ("transform expr in nested blocks") {
+ val expr = JavaCode.expression("1 + 1", IntegerType)
+ val isNull = JavaCode.isNullVariable("expr1_isNull")
+ val exprInFunc = JavaCode.variable("expr1", IntegerType)
+
+ val funcs = Seq("callFunc1", "callFunc2", "callFunc3")
+ val subBlocks = funcs.map { funcName =>
+ code"""
+ |$funcName(int $expr) {
+ | boolean $isNull = false;
+ | int $exprInFunc = $expr + 1;
+ |}""".stripMargin
+ }
+
+ val aliasedParam = JavaCode.variable("aliased", expr.javaType)
+
+ val block = code"${subBlocks(0)}\n${subBlocks(1)}\n${subBlocks(2)}"
+ val transformedBlock = block.transform {
+ case b: Block => b.transformExprValues {
--- End diff --
Then I may call it `transformAllExprValues`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r190244637
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeBlockSuite.scala ---
@@ -120,11 +120,11 @@ class CodeBlockSuite extends SparkFunSuite {
|}""".stripMargin
val aliasedParam = JavaCode.variable("aliased", expr.javaType)
- val aliasedInputs = code.asInstanceOf[CodeBlock].blockInputs.map {
- case _: SimpleExprValue => aliasedParam
- case other => other
+
+ // We want to replace all occurrences of `expr` with the variable `aliasedParam`.
+ val aliasedCode = code.transformExprValues {
+ case SimpleExprValue("1 + 1", _) => aliasedParam
--- End diff --
Ok.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r200247248
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeBlockSuite.scala ---
@@ -133,4 +133,55 @@ class CodeBlockSuite extends SparkFunSuite {
|}""".stripMargin
assert(aliasedCode.toString == expected.toString)
}
+
+ test ("transform expr in nested blocks") {
+ val expr = JavaCode.expression("1 + 1", IntegerType)
+ val isNull = JavaCode.isNullVariable("expr1_isNull")
+ val exprInFunc = JavaCode.variable("expr1", IntegerType)
+
+ val funcs = Seq("callFunc1", "callFunc2", "callFunc3")
+ val subBlocks = funcs.map { funcName =>
+ code"""
+ |$funcName(int $expr) {
+ | boolean $isNull = false;
+ | int $exprInFunc = $expr + 1;
+ |}""".stripMargin
+ }
+
+ val aliasedParam = JavaCode.variable("aliased", expr.javaType)
+
+ val block = code"${subBlocks(0)}\n${subBlocks(1)}\n${subBlocks(2)}"
+ val transformedBlock = block.transform {
+ case b: Block => b.transformExprValues {
--- End diff --
hmmm, I'd image `transformExprValues` will also transform child blocks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21405
@cloud-fan @hvanhovell Do you have any comment/suggestion on this change? Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21405
LGTM, can you update the demo in your PR description? thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21405
retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21405
**[Test build #92642 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92642/testReport)** for PR 21405 at commit [`c9dee44`](https://github.com/apache/spark/commit/c9dee44c6b4e9fdd41b2460fbd331f8f0e9235f4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class LiteralValue(val value: String, val javaType: Class[_]) extends ExprValue with Serializable `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92637/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92640/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91031/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21405
cc @cloud-fan @hvanhovell @kiszk @mgaido91 @maropu @rednaxelafx @ueshin
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/701/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r190501136
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/javaCode.scala ---
@@ -149,6 +152,34 @@ trait Block extends JavaCode {
// Concatenates this block with other block.
def + (other: Block): Block
+
+ /**
+ * Apply a map function to each java expression codes present in this java code, and return a new
+ * java code based on the mapped java expression codes.
+ */
+ def transformExprValues(f: PartialFunction[ExprValue, ExprValue]): this.type = {
--- End diff --
`transformExprValues` will create a new instance `Block` because a block is immutable. So once there are change, new block should take new `exprValues`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3503/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21405
**[Test build #92640 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92640/testReport)** for PR 21405 at commit [`707dd6f`](https://github.com/apache/spark/commit/707dd6f0f5385afaa4721c91272a11ca5d12b775).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21405
**[Test build #92637 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92637/testReport)** for PR 21405 at commit [`ce8d9ed`](https://github.com/apache/spark/commit/ce8d9ed2165097e82ad2b3d03da6c841d623f572).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait JavaCode `
* `trait Block extends TreeNode[Block] with JavaCode `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92642/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/703/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21405
**[Test build #91251 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91251/testReport)** for PR 21405 at commit [`e30be7a`](https://github.com/apache/spark/commit/e30be7a3da9061a71e8ac222006e20b4e173cb74).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3664/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21405
**[Test build #91021 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91021/testReport)** for PR 21405 at commit [`2d77332`](https://github.com/apache/spark/commit/2d77332d77d133be1ed1c2c155c9020462628c55).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21405
**[Test build #92640 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92640/testReport)** for PR 21405 at commit [`707dd6f`](https://github.com/apache/spark/commit/707dd6f0f5385afaa4721c91272a11ca5d12b775).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21405: [SPARK-24361][SQL] Polish code block manipulation...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21405#discussion_r190502747
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/javaCode.scala ---
@@ -149,6 +152,34 @@ trait Block extends JavaCode {
// Concatenates this block with other block.
def + (other: Block): Block
+
+ /**
+ * Apply a map function to each java expression codes present in this java code, and return a new
+ * java code based on the mapped java expression codes.
+ */
+ def transformExprValues(f: PartialFunction[ExprValue, ExprValue]): this.type = {
--- End diff --
Ok. Will add them in next commit. Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21405
@mgaido91 Yes, as @rednaxelafx said in previous RP, #19813 will be the first use case. I'd like to use this API to transform statement-based expressions in code blocks.
For example, `code` contains Java code of an expression. We find it is too long and we need to split it out as function. Because statement-based expressions can't be directly passed into. We need to transform them as variables first. I think the usage would looks like:
```scala
val collectedStatements = HashMap[ExprValue, ExprValue]
val transformed = code.transformExprValues {
case s @ SimpleExprValue(_, javaType) =>
if (collectedStatements.contains(s)) {
collectedStatements(s)
} else {
val aliasedVariable = JavaCode.variable(ctx.freshName("var"), javaType)
collectedStatements += s -> aliasedVariable
aliasedVariable
}
}
val createVariables = collectedStatements.foldLeft(EmptyBlock) { (block, (statement, variable)) =>
block + code"${statement.javaType.getName} $variable = $statement;"
}
ev.copy(code = createVariables + transformed)
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21405
**[Test build #92637 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92637/testReport)** for PR 21405 at commit [`ce8d9ed`](https://github.com/apache/spark/commit/ce8d9ed2165097e82ad2b3d03da6c841d623f572).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21405
**[Test build #91031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91031/testReport)** for PR 21405 at commit [`2d77332`](https://github.com/apache/spark/commit/2d77332d77d133be1ed1c2c155c9020462628c55).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/697/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3874/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21405: [SPARK-24361][SQL] Polish code block manipulation API
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21405
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org