You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2017/11/16 17:42:52 UTC
[GitHub] spark pull request #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile err...
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/19767
[WIP][SPARK-22543][SQL] fix java 64kb compile error for deeply nested expressions
## What changes were proposed in this pull request?
A frequently reported issue of Spark is the Java 64kb compile error. This is because Spark generates a very big method and it's usually caused by 2 reasons:
1. a deep expression tree, e.g. a very complex filter condition
2. many individual expressions, e.g. expressions can have many children, operators can have many expressions.
This PR focuses on 1. There are already several patches(#15620 #18972 #18641) trying to fix this issue and some of them are already merged. However this is an endless job as every non-leaf expression has this issue.
This PR proposes to fix this issue in `Expression.genCode`, to make sure the code for a single expression won't grow too big.
TODO: check TPCDS to make sure no performance regression.
## How was this patch tested?
existing test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark codegen
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19767.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19767
----
commit aaa5b6f452034660880d8175f450ba8c8cfad23b
Author: Wenchen Fan <we...@databricks.com>
Date: 2017-11-16T16:55:43Z
fix java 64kb compile error for deeply nested expressions
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #83943 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83943/testReport)** for PR 19767 at commit [`aaa5b6f`](https://github.com/apache/spark/commit/aaa5b6f452034660880d8175f450ba8c8cfad23b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152414088
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -64,52 +64,22 @@ case class If(predicate: Expression, trueValue: Expression, falseValue: Expressi
val trueEval = trueValue.genCode(ctx)
val falseEval = falseValue.genCode(ctx)
- // place generated code of condition, true value and false value in separate methods if
- // their code combined is large
- val combinedLength = condEval.code.length + trueEval.code.length + falseEval.code.length
--- End diff --
I already explained it in https://github.com/apache/spark/pull/19767#issuecomment-345176286
Mostly it's ok because the threshold is just an estimation, not a big deal to make it 2 times larger. CASE WHEN may be a problem and we can evaluate it in #18641 after this PR gets merged.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152417537
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,36 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
+ val localIsNull = ve.isNull
+ ve.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val javaType = ctx.javaType(dataType)
+ val newValue = ctx.freshName("value")
+
+ val funcName = ctx.freshName(nodeName)
+ val funcFullName = ctx.addNewFunction(funcName,
+ s"""
+ |private $javaType $funcName(InternalRow ${ctx.INPUT_ROW}) {
--- End diff --
cc @rednaxelafx @mgaido91 too
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/19767
Looks good direction if we do not see performance degradation.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152449180
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,41 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
+ val localIsNull = ve.isNull
+ ve.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val setValue = {
+ val globalValue = ctx.freshName("globalValue")
+ ctx.addMutableState(
+ ctx.javaType(dataType), globalValue, s"$globalValue = ${ctx.defaultValue(dataType)};")
+ val localValue = ve.value
+ ve.value = globalValue
+ s"$globalValue = $localValue;"
+ }
+
+ val funcName = ctx.freshName(nodeName)
+ val funcFullName = ctx.addNewFunction(funcName,
+ s"""
+ |private void $funcName(InternalRow ${ctx.INPUT_ROW}) {
+ | ${ve.code.trim}
+ | $setValue
--- End diff --
I originally thought we could avoid the overhead by using thread-local singleton? But, it's a bit weird, so the current code looks good.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19767
LGTM except the above comments.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/19767
I like this approach. Does this pr solve the same issue in #18641? or, orthogonal? Anyway, I checked the TPCDS perf with this current pr: https://docs.google.com/spreadsheets/d/1K3_7lX05-ZgxDXi9X_GleNnDjcnJIfoSlSCDZcL4gdg/edit?usp=sharing
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/19767
Probably, I feel we better track the changes of actual bytecode size statistics (e.g, `maxCodeSize`) in tpcds, so I'll check later.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile err...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r151637511
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,41 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
+ val localIsNull = ve.isNull
+ ve.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val setValue = {
+ val globalValue = ctx.freshName("globalValue")
+ ctx.addMutableState(
+ ctx.javaType(dataType), globalValue, s"$globalValue = ${ctx.defaultValue(dataType)};")
+ val localValue = ve.value
+ ve.value = globalValue
+ s"$globalValue = $localValue;"
+ }
+
+ val funcName = ctx.freshName(nodeName)
+ val funcFullName = ctx.addNewFunction(funcName,
+ s"""
+ |private void $funcName(InternalRow ${ctx.INPUT_ROW}) {
+ | ${ve.code.trim}
+ | $setValue
+ | $setIsNull
+ |}
+ """.stripMargin)
+
+ ve.code = s"$funcFullName(${ctx.INPUT_ROW});"
+ }
+
if (ve.code.nonEmpty) {
// Add `this` in the comment.
ve.copy(code = s"${ctx.registerComment(this.toString)}\n" + ve.code.trim)
--- End diff --
I don't have a strong preference, it's ok to have comment at function caller side.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19767
@maropu it partially covers #18641 . One problem is that, for an expression, if its child generates code less than 1024, and it has many children, then we still have an issue. `CaseWhen` is a little different because it at most can have 20 children(depends on `spark.sql.codegen.maxCaseBranches`). So we can still prevent failures, but may not be able to JIT.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19767
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #84110 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84110/testReport)** for PR 19767 at commit [`29188fe`](https://github.com/apache/spark/commit/29188fe5a5211c1e6f2762098f68962c59f251b1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #84099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84099/testReport)** for PR 19767 at commit [`c875329`](https://github.com/apache/spark/commit/c8753298df417d597c1beb49362e0ee1220c79fb).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r151830994
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,41 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
+ val localIsNull = ve.isNull
+ ve.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val setValue = {
+ val globalValue = ctx.freshName("globalValue")
+ ctx.addMutableState(
+ ctx.javaType(dataType), globalValue, s"$globalValue = ${ctx.defaultValue(dataType)};")
+ val localValue = ve.value
+ ve.value = globalValue
+ s"$globalValue = $localValue;"
+ }
+
+ val funcName = ctx.freshName(nodeName)
+ val funcFullName = ctx.addNewFunction(funcName,
+ s"""
+ |private void $funcName(InternalRow ${ctx.INPUT_ROW}) {
+ | ${ve.code.trim}
+ | $setValue
--- End diff --
Is that a bad idea to prepare some utility classes to store a pair (value, isNull) for this splitting cases? I feel class fields are valuable resources.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152470168
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,36 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
--- End diff --
-> `ctx.JAVA_BOOLEAN`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/19767
Would it be better to replace `"SPARK-...."` in test case for (#15620 #18972 #18641) with this JIRA number?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83963/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152572967
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -104,16 +104,48 @@ abstract class Expression extends TreeNode[Expression] {
}.getOrElse {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
- val ve = doGenCode(ctx, ExprCode("", isNull, value))
- if (ve.code.nonEmpty) {
+ val eval = doGenCode(ctx, ExprCode("", isNull, value))
+ reduceCodeSize(ctx, eval)
+ if (eval.code.nonEmpty) {
// Add `this` in the comment.
- ve.copy(code = s"${ctx.registerComment(this.toString)}\n" + ve.code.trim)
+ eval.copy(code = s"${ctx.registerComment(this.toString)}\n" + eval.code.trim)
} else {
- ve
+ eval
}
}
}
+ private def reduceCodeSize(ctx: CodegenContext, eval: ExprCode): Unit = {
+ // TODO: support whole stage codegen too
+ if (eval.code.trim.length > 500 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (eval.isNull != "false" && eval.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState(ctx.JAVA_BOOLEAN, globalIsNull)
+ val localIsNull = eval.isNull
+ eval.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val javaType = ctx.javaType(dataType)
+ val newValue = ctx.freshName("value")
--- End diff --
I see, sorry, you're right. Then I think your previous solution is better: in this way if `eval.value` is used multiple times we are recomputing the function every time, thus your original implementation was fine, sorry for the bad comment.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile err...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r151593662
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,41 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
+ val localIsNull = ve.isNull
+ ve.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val setValue = {
+ val globalValue = ctx.freshName("globalValue")
+ ctx.addMutableState(
+ ctx.javaType(dataType), globalValue, s"$globalValue = ${ctx.defaultValue(dataType)};")
+ val localValue = ve.value
+ ve.value = globalValue
+ s"$globalValue = $localValue;"
+ }
+
+ val funcName = ctx.freshName(nodeName)
+ val funcFullName = ctx.addNewFunction(funcName,
+ s"""
+ |private void $funcName(InternalRow ${ctx.INPUT_ROW}) {
+ | ${ve.code.trim}
+ | $setValue
+ | $setIsNull
+ |}
+ """.stripMargin)
+
+ ve.code = s"$funcFullName(${ctx.INPUT_ROW});"
+ }
+
if (ve.code.nonEmpty) {
// Add `this` in the comment.
ve.copy(code = s"${ctx.registerComment(this.toString)}\n" + ve.code.trim)
--- End diff --
Should we move the comment into the function?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152489116
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,36 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
--- End diff --
I think it won't work because of hitting other limitations, e.g. JVM constant pool.
I'll try something bigger, like `100`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152554854
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -104,16 +104,48 @@ abstract class Expression extends TreeNode[Expression] {
}.getOrElse {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
- val ve = doGenCode(ctx, ExprCode("", isNull, value))
- if (ve.code.nonEmpty) {
+ val eval = doGenCode(ctx, ExprCode("", isNull, value))
+ reduceCodeSize(ctx, eval)
+ if (eval.code.nonEmpty) {
// Add `this` in the comment.
- ve.copy(code = s"${ctx.registerComment(this.toString)}\n" + ve.code.trim)
+ eval.copy(code = s"${ctx.registerComment(this.toString)}\n" + eval.code.trim)
} else {
- ve
+ eval
}
}
}
+ private def reduceCodeSize(ctx: CodegenContext, eval: ExprCode): Unit = {
+ // TODO: support whole stage codegen too
+ if (eval.code.trim.length > 500 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (eval.isNull != "false" && eval.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState(ctx.JAVA_BOOLEAN, globalIsNull)
+ val localIsNull = eval.isNull
+ eval.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val javaType = ctx.javaType(dataType)
+ val newValue = ctx.freshName("value")
--- End diff --
why is this needed? I think we can use `eval.value` instead of it
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84099/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152569647
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -104,16 +104,48 @@ abstract class Expression extends TreeNode[Expression] {
}.getOrElse {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
- val ve = doGenCode(ctx, ExprCode("", isNull, value))
- if (ve.code.nonEmpty) {
+ val eval = doGenCode(ctx, ExprCode("", isNull, value))
+ reduceCodeSize(ctx, eval)
+ if (eval.code.nonEmpty) {
// Add `this` in the comment.
- ve.copy(code = s"${ctx.registerComment(this.toString)}\n" + ve.code.trim)
+ eval.copy(code = s"${ctx.registerComment(this.toString)}\n" + eval.code.trim)
} else {
- ve
+ eval
}
}
}
+ private def reduceCodeSize(ctx: CodegenContext, eval: ExprCode): Unit = {
+ // TODO: support whole stage codegen too
+ if (eval.code.trim.length > 500 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (eval.isNull != "false" && eval.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState(ctx.JAVA_BOOLEAN, globalIsNull)
+ val localIsNull = eval.isNull
+ eval.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val javaType = ctx.javaType(dataType)
+ val newValue = ctx.freshName("value")
--- End diff --
I think it would simply become:
```
eval.code = s"${eval.value} = $funcFullName(${ctx.INPUT_ROW});"
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #84112 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84112/testReport)** for PR 19767 at commit [`c015d33`](https://github.com/apache/spark/commit/c015d33e7002da8a2c7473fcd99f6b9d51b0d7a2).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r151852231
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -64,52 +64,22 @@ case class If(predicate: Expression, trueValue: Expression, falseValue: Expressi
val trueEval = trueValue.genCode(ctx)
val falseEval = falseValue.genCode(ctx)
- // place generated code of condition, true value and false value in separate methods if
- // their code combined is large
- val combinedLength = condEval.code.length + trueEval.code.length + falseEval.code.length
--- End diff --
Actually I think this removed part is orthogonal to what this PR did. Even condition, true, and false expressions are not more than threshold individually, their combination is still more than the threshold.
This PR deals the oversize gen'd codes in deeply nested expressions, not oversize combination of codes from the children.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84112/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152569022
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -104,16 +104,48 @@ abstract class Expression extends TreeNode[Expression] {
}.getOrElse {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
- val ve = doGenCode(ctx, ExprCode("", isNull, value))
- if (ve.code.nonEmpty) {
+ val eval = doGenCode(ctx, ExprCode("", isNull, value))
+ reduceCodeSize(ctx, eval)
+ if (eval.code.nonEmpty) {
// Add `this` in the comment.
- ve.copy(code = s"${ctx.registerComment(this.toString)}\n" + ve.code.trim)
+ eval.copy(code = s"${ctx.registerComment(this.toString)}\n" + eval.code.trim)
} else {
- ve
+ eval
}
}
}
+ private def reduceCodeSize(ctx: CodegenContext, eval: ExprCode): Unit = {
+ // TODO: support whole stage codegen too
+ if (eval.code.trim.length > 500 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (eval.isNull != "false" && eval.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState(ctx.JAVA_BOOLEAN, globalIsNull)
+ val localIsNull = eval.isNull
+ eval.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val javaType = ctx.javaType(dataType)
+ val newValue = ctx.freshName("value")
--- End diff --
ah, do you mean just do `eval.value = s"$funcFullName(${ctx.INPUT_ROW})"`? Let me try
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile err...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r151595494
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,41 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
+ val localIsNull = ve.isNull
+ ve.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val setValue = {
+ val globalValue = ctx.freshName("globalValue")
+ ctx.addMutableState(
+ ctx.javaType(dataType), globalValue, s"$globalValue = ${ctx.defaultValue(dataType)};")
+ val localValue = ve.value
+ ve.value = globalValue
+ s"$globalValue = $localValue;"
+ }
+
+ val funcName = ctx.freshName(nodeName)
+ val funcFullName = ctx.addNewFunction(funcName,
+ s"""
+ |private void $funcName(InternalRow ${ctx.INPUT_ROW}) {
+ | ${ve.code.trim}
+ | $setValue
+ | $setIsNull
--- End diff --
nit: when `isNull` is evaluated to true at runtime, we don't need to set value.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152436838
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -64,52 +64,22 @@ case class If(predicate: Expression, trueValue: Expression, falseValue: Expressi
val trueEval = trueValue.genCode(ctx)
val falseEval = falseValue.genCode(ctx)
- // place generated code of condition, true value and false value in separate methods if
- // their code combined is large
- val combinedLength = condEval.code.length + trueEval.code.length + falseEval.code.length
--- End diff --
Two problems I think for this. One is even the two childs' code don't exceed the threshold individually, a method not over 64k but over 8k is still big and bad for JIT. One is we estimate it with code length, I'm not sure if two 1000 length childs won't definitely generate 64k method in the end.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #83943 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83943/testReport)** for PR 19767 at commit [`aaa5b6f`](https://github.com/apache/spark/commit/aaa5b6f452034660880d8175f450ba8c8cfad23b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #84104 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84104/testReport)** for PR 19767 at commit [`86cba3c`](https://github.com/apache/spark/commit/86cba3c1bad9d4f189baabcfed84b13d9ed2c3e5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152417333
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,36 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
+ val localIsNull = ve.isNull
+ ve.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val javaType = ctx.javaType(dataType)
+ val newValue = ctx.freshName("value")
+
+ val funcName = ctx.freshName(nodeName)
+ val funcFullName = ctx.addNewFunction(funcName,
+ s"""
+ |private $javaType $funcName(InternalRow ${ctx.INPUT_ROW}) {
--- End diff --
To continue the discussion in https://github.com/apache/spark/pull/19767#discussion_r151631456
I think there are more global variables can be eliminated by leveraging the method return value. However in some cases, we use global variables to avoid creating an object for each iteration, then we are facing a trade-off between GC overhead and global variable overhead. It would be great if java has something like C struct and can allocate objects on method stack...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #84099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84099/testReport)** for PR 19767 at commit [`c875329`](https://github.com/apache/spark/commit/c8753298df417d597c1beb49362e0ee1220c79fb).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r151842921
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,41 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
+ val localIsNull = ve.isNull
+ ve.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val setValue = {
+ val globalValue = ctx.freshName("globalValue")
+ ctx.addMutableState(
+ ctx.javaType(dataType), globalValue, s"$globalValue = ${ctx.defaultValue(dataType)};")
+ val localValue = ve.value
+ ve.value = globalValue
+ s"$globalValue = $localValue;"
+ }
+
+ val funcName = ctx.freshName(nodeName)
+ val funcFullName = ctx.addNewFunction(funcName,
+ s"""
+ |private void $funcName(InternalRow ${ctx.INPUT_ROW}) {
+ | ${ve.code.trim}
+ | $setValue
--- End diff --
creating objects will be a big overhead. I think having a global boolean variable is better.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #84111 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84111/testReport)** for PR 19767 at commit [`e494844`](https://github.com/apache/spark/commit/e49484466e6009f13bff1a38a5a06d707c207ed3).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152565923
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -104,16 +104,48 @@ abstract class Expression extends TreeNode[Expression] {
}.getOrElse {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
- val ve = doGenCode(ctx, ExprCode("", isNull, value))
- if (ve.code.nonEmpty) {
+ val eval = doGenCode(ctx, ExprCode("", isNull, value))
+ reduceCodeSize(ctx, eval)
+ if (eval.code.nonEmpty) {
// Add `this` in the comment.
- ve.copy(code = s"${ctx.registerComment(this.toString)}\n" + ve.code.trim)
+ eval.copy(code = s"${ctx.registerComment(this.toString)}\n" + eval.code.trim)
} else {
- ve
+ eval
}
}
}
+ private def reduceCodeSize(ctx: CodegenContext, eval: ExprCode): Unit = {
+ // TODO: support whole stage codegen too
+ if (eval.code.trim.length > 500 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (eval.isNull != "false" && eval.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState(ctx.JAVA_BOOLEAN, globalIsNull)
+ val localIsNull = eval.isNull
+ eval.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val javaType = ctx.javaType(dataType)
+ val newValue = ctx.freshName("value")
--- End diff --
`ev.value` may be a global variable and here we need a local variable.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83943/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #83963 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83963/testReport)** for PR 19767 at commit [`3dab5bd`](https://github.com/apache/spark/commit/3dab5bdbc4d2bb1818c46905afb92422bac04d9e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84111/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile err...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r151644789
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,41 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
+ val localIsNull = ve.isNull
+ ve.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val setValue = {
+ val globalValue = ctx.freshName("globalValue")
+ ctx.addMutableState(
+ ctx.javaType(dataType), globalValue, s"$globalValue = ${ctx.defaultValue(dataType)};")
+ val localValue = ve.value
+ ve.value = globalValue
+ s"$globalValue = $localValue;"
+ }
+
+ val funcName = ctx.freshName(nodeName)
+ val funcFullName = ctx.addNewFunction(funcName,
+ s"""
+ |private void $funcName(InternalRow ${ctx.INPUT_ROW}) {
+ | ${ve.code.trim}
+ | $setValue
--- End diff --
Thanks.
IMHO, I am curious whether we will see no performance degradation by using one array to compact many boolean variables.
I am waiting for the updated result in [this discussion](https://github.com/apache/spark/pull/19518#issuecomment-337965330). This is because the current code seems to measure performance of interpreter due to lack of warmup.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/19767
should this go to 2.2?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile err...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r151636953
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,41 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
+ val localIsNull = ve.isNull
+ ve.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val setValue = {
+ val globalValue = ctx.freshName("globalValue")
+ ctx.addMutableState(
+ ctx.javaType(dataType), globalValue, s"$globalValue = ${ctx.defaultValue(dataType)};")
+ val localValue = ve.value
+ ve.value = globalValue
+ s"$globalValue = $localValue;"
+ }
+
+ val funcName = ctx.freshName(nodeName)
+ val funcFullName = ctx.addNewFunction(funcName,
+ s"""
+ |private void $funcName(InternalRow ${ctx.INPUT_ROW}) {
+ | ${ve.code.trim}
+ | $setValue
--- End diff --
good suggestion! Actually, this is a general strategy which can be applied to more places. If there are only boolean global variables, it's very easy to fold them into one array.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152539689
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,36 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
+ val localIsNull = ve.isNull
+ ve.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val javaType = ctx.javaType(dataType)
+ val newValue = ctx.freshName("value")
+
+ val funcName = ctx.freshName(nodeName)
+ val funcFullName = ctx.addNewFunction(funcName,
+ s"""
+ |private $javaType $funcName(InternalRow ${ctx.INPUT_ROW}) {
--- End diff --
thanks for this fix. I like your approach here.
Actually what happens depends on the type of the variable and anyway I think that most of the time we are reinitializing anyway these objects, thus the only thing we are saving using global variables is the pointer and I am not sure if this is a big deal.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19767
cc @kiszk @rednaxelafx @maropu @gatorsmile
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152476972
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,36 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
--- End diff --
Could you change `1024` to `1`? Just to ensure whether all the tests can pass?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19767
will review it tonight.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19767
Thanks! Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #84085 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84085/testReport)** for PR 19767 at commit [`d126977`](https://github.com/apache/spark/commit/d126977bbdd221287b0825fa78c04b1065d97ab1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19767
@felixcheung I'd like to keep it in master only, it has larger impaction than other related PRs.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84104/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19767
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #83963 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83963/testReport)** for PR 19767 at commit [`3dab5bd`](https://github.com/apache/spark/commit/3dab5bdbc4d2bb1818c46905afb92422bac04d9e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #84085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84085/testReport)** for PR 19767 at commit [`d126977`](https://github.com/apache/spark/commit/d126977bbdd221287b0825fa78c04b1065d97ab1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84110/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19767
Seems a good approach that saves us much effort to add similar codes for many expressions.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152518292
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,41 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
+ val localIsNull = ve.isNull
+ ve.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val setValue = {
+ val globalValue = ctx.freshName("globalValue")
+ ctx.addMutableState(
+ ctx.javaType(dataType), globalValue, s"$globalValue = ${ctx.defaultValue(dataType)};")
+ val localValue = ve.value
+ ve.value = globalValue
+ s"$globalValue = $localValue;"
+ }
+
+ val funcName = ctx.freshName(nodeName)
+ val funcFullName = ctx.addNewFunction(funcName,
+ s"""
+ |private void $funcName(InternalRow ${ctx.INPUT_ROW}) {
+ | ${ve.code.trim}
+ | $setValue
--- End diff --
The current code can pass the `value` as a local variable if this method is inlined, or can pass the `value` on register for return value if this method is not inlined.
On the other hand, to use a object will always introduce memory accesses to access fields in caller and callee.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152570718
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -104,16 +104,48 @@ abstract class Expression extends TreeNode[Expression] {
}.getOrElse {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
- val ve = doGenCode(ctx, ExprCode("", isNull, value))
- if (ve.code.nonEmpty) {
+ val eval = doGenCode(ctx, ExprCode("", isNull, value))
+ reduceCodeSize(ctx, eval)
+ if (eval.code.nonEmpty) {
// Add `this` in the comment.
- ve.copy(code = s"${ctx.registerComment(this.toString)}\n" + ve.code.trim)
+ eval.copy(code = s"${ctx.registerComment(this.toString)}\n" + eval.code.trim)
} else {
- ve
+ eval
}
}
}
+ private def reduceCodeSize(ctx: CodegenContext, eval: ExprCode): Unit = {
+ // TODO: support whole stage codegen too
+ if (eval.code.trim.length > 500 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (eval.isNull != "false" && eval.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState(ctx.JAVA_BOOLEAN, globalIsNull)
+ val localIsNull = eval.isNull
+ eval.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val javaType = ctx.javaType(dataType)
+ val newValue = ctx.freshName("value")
--- End diff --
this won't work because `${eval.value}` is not declared if it's not a global variable. I went with
```
eval.code = ""
eval.value = s"$funcFullName(${ctx.INPUT_ROW})"
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile err...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r151631456
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,41 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
+ val localIsNull = ve.isNull
+ ve.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val setValue = {
+ val globalValue = ctx.freshName("globalValue")
+ ctx.addMutableState(
+ ctx.javaType(dataType), globalValue, s"$globalValue = ${ctx.defaultValue(dataType)};")
+ val localValue = ve.value
+ ve.value = globalValue
+ s"$globalValue = $localValue;"
+ }
+
+ val funcName = ctx.freshName(nodeName)
+ val funcFullName = ctx.addNewFunction(funcName,
+ s"""
+ |private void $funcName(InternalRow ${ctx.INPUT_ROW}) {
+ | ${ve.code.trim}
+ | $setValue
--- End diff --
Can we always pass `value` as a return value? It can reduce # of global variables.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19767
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile err...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r151624776
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,41 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
+ val localIsNull = ve.isNull
+ ve.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val setValue = {
+ val globalValue = ctx.freshName("globalValue")
+ ctx.addMutableState(
+ ctx.javaType(dataType), globalValue, s"$globalValue = ${ctx.defaultValue(dataType)};")
+ val localValue = ve.value
+ ve.value = globalValue
+ s"$globalValue = $localValue;"
+ }
+
+ val funcName = ctx.freshName(nodeName)
+ val funcFullName = ctx.addNewFunction(funcName,
+ s"""
+ |private void $funcName(InternalRow ${ctx.INPUT_ROW}) {
+ | ${ve.code.trim}
+ | $setValue
+ | $setIsNull
--- End diff --
yea it's already done when define `setIsNull`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152441210
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -64,52 +64,22 @@ case class If(predicate: Expression, trueValue: Expression, falseValue: Expressi
val trueEval = trueValue.genCode(ctx)
val falseEval = falseValue.genCode(ctx)
- // place generated code of condition, true value and false value in separate methods if
- // their code combined is large
- val combinedLength = condEval.code.length + trueEval.code.length + falseEval.code.length
--- End diff --
There is no way to guarantee it with the current string based codegen framework, even without this PR. 1000 length code may also generate 64kb byte code in the end.
`1024` is not a good estimation at all, kind of random to me. So multiplying it with 2 doesn't seem a big issue. CASE WHEN may have issue.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152567152
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -104,16 +104,48 @@ abstract class Expression extends TreeNode[Expression] {
}.getOrElse {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
- val ve = doGenCode(ctx, ExprCode("", isNull, value))
- if (ve.code.nonEmpty) {
+ val eval = doGenCode(ctx, ExprCode("", isNull, value))
+ reduceCodeSize(ctx, eval)
+ if (eval.code.nonEmpty) {
// Add `this` in the comment.
- ve.copy(code = s"${ctx.registerComment(this.toString)}\n" + ve.code.trim)
+ eval.copy(code = s"${ctx.registerComment(this.toString)}\n" + eval.code.trim)
} else {
- ve
+ eval
}
}
}
+ private def reduceCodeSize(ctx: CodegenContext, eval: ExprCode): Unit = {
+ // TODO: support whole stage codegen too
+ if (eval.code.trim.length > 500 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (eval.isNull != "false" && eval.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState(ctx.JAVA_BOOLEAN, globalIsNull)
+ val localIsNull = eval.isNull
+ eval.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val javaType = ctx.javaType(dataType)
+ val newValue = ctx.freshName("value")
--- End diff --
why do we strictly need a local variable here? Can't we simply assign `ev.value` to the generated function return value?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19767
Also from the numbers provided by @maropu, looks no significant regression.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #84104 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84104/testReport)** for PR 19767 at commit [`86cba3c`](https://github.com/apache/spark/commit/86cba3c1bad9d4f189baabcfed84b13d9ed2c3e5).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84085/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152568599
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -104,16 +104,48 @@ abstract class Expression extends TreeNode[Expression] {
}.getOrElse {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
- val ve = doGenCode(ctx, ExprCode("", isNull, value))
- if (ve.code.nonEmpty) {
+ val eval = doGenCode(ctx, ExprCode("", isNull, value))
+ reduceCodeSize(ctx, eval)
+ if (eval.code.nonEmpty) {
// Add `this` in the comment.
- ve.copy(code = s"${ctx.registerComment(this.toString)}\n" + ve.code.trim)
+ eval.copy(code = s"${ctx.registerComment(this.toString)}\n" + eval.code.trim)
} else {
- ve
+ eval
}
}
}
+ private def reduceCodeSize(ctx: CodegenContext, eval: ExprCode): Unit = {
+ // TODO: support whole stage codegen too
+ if (eval.code.trim.length > 500 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (eval.isNull != "false" && eval.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState(ctx.JAVA_BOOLEAN, globalIsNull)
+ val localIsNull = eval.isNull
+ eval.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val javaType = ctx.javaType(dataType)
+ val newValue = ctx.freshName("value")
--- End diff --
then how are we going to change this?
`eval.code = s"$javaType $newValue = $funcFullName(${ctx.INPUT_ROW});"`
Saving a local variable is nothing and I think we shouldn't complicate the code(check if a variable is global) because of this.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #84111 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84111/testReport)** for PR 19767 at commit [`e494844`](https://github.com/apache/spark/commit/e49484466e6009f13bff1a38a5a06d707c207ed3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152477535
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala ---
@@ -105,6 +105,36 @@ abstract class Expression extends TreeNode[Expression] {
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val ve = doGenCode(ctx, ExprCode("", isNull, value))
+
+ // TODO: support whole stage codegen too
+ if (ve.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
+ val setIsNull = if (ve.isNull != "false" && ve.isNull != "true") {
+ val globalIsNull = ctx.freshName("globalIsNull")
+ ctx.addMutableState("boolean", globalIsNull, s"$globalIsNull = false;")
+ val localIsNull = ve.isNull
+ ve.isNull = globalIsNull
+ s"$globalIsNull = $localIsNull;"
+ } else {
+ ""
+ }
+
+ val javaType = ctx.javaType(dataType)
+ val newValue = ctx.freshName("value")
+
+ val funcName = ctx.freshName(nodeName)
+ val funcFullName = ctx.addNewFunction(funcName,
+ s"""
+ |private $javaType $funcName(InternalRow ${ctx.INPUT_ROW}) {
+ | ${ve.code.trim}
+ | $setIsNull
+ | return ${ve.value};
+ |}
+ """.stripMargin)
+
+ ve.value = newValue
+ ve.code = s"$javaType $newValue = $funcFullName(${ctx.INPUT_ROW});"
--- End diff --
Create a separate function for this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19767: [SPARK-22543][SQL] fix java 64kb compile error fo...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19767#discussion_r152442103
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -64,52 +64,22 @@ case class If(predicate: Expression, trueValue: Expression, falseValue: Expressi
val trueEval = trueValue.genCode(ctx)
val falseEval = falseValue.genCode(ctx)
- // place generated code of condition, true value and false value in separate methods if
- // their code combined is large
- val combinedLength = condEval.code.length + trueEval.code.length + falseEval.code.length
--- End diff --
BTW if it's really an issue, we can add splitting logic in non-leaf/non-unary nodes. This is much less work than before because: 1. no need to care about unary nodes 2. the splitting logic can be simpler because all children are guaranteed to generate less than 1000 LOC.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #84110 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84110/testReport)** for PR 19767 at commit [`29188fe`](https://github.com/apache/spark/commit/29188fe5a5211c1e6f2762098f68962c59f251b1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19767
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19767: [SPARK-22543][SQL] fix java 64kb compile error for deepl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19767
**[Test build #84112 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84112/testReport)** for PR 19767 at commit [`c015d33`](https://github.com/apache/spark/commit/c015d33e7002da8a2c7473fcd99f6b9d51b0d7a2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org