You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by kiszk <gi...@git.apache.org> on 2017/11/13 18:56:07 UTC
[GitHub] spark pull request #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit pr...
GitHub user kiszk opened a pull request:
https://github.com/apache/spark/pull/19737
[SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem with GenerateUnsafeRowJoiner.create()
## What changes were proposed in this pull request?
This PR changes `GenerateUnsafeRowJoiner.create()` code generation to place generated code for statements to operate bitmap and offset into separated methods if these size could be large.
## How was this patch tested?
Added a new test case into `GenerateUnsafeRowJoinerSuite`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kiszk/spark SPARK-22508
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19737.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19737
----
commit aef727bb9602ae656cceef74e245c836b7420b7d
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2017-11-13T18:52:32Z
initial commit
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit pr...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19737#discussion_r152185474
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala ---
@@ -154,7 +164,10 @@ object GenerateUnsafeRowJoiner extends CodeGenerator[(StructType, StructType), U
|$putLong(buf, $cursor, $getLong(buf, $cursor) + ($shift << 32));
""".stripMargin
}
- }.mkString("\n")
+ }
+
+ val updateOffsets = ctx.splitExpressions(updateOffset, "copyBitsetFunc",
+ ("long", "numBytesVariableRow1") :: Nil)
--- End diff --
The same here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19737
**[Test build #83801 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83801/testReport)** for PR 19737 at commit [`aef727b`](https://github.com/apache/spark/commit/aef727bb9602ae656cceef74e245c836b7420b7d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19737
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84054/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19737
LGTM pending jenkins
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19737
**[Test build #84061 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84061/testReport)** for PR 19737 at commit [`0704a82`](https://github.com/apache/spark/commit/0704a821eb7913768ba57a91ec4fc234d8941ca8).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19737
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84061/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/19737
Jenkins, retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19737
**[Test build #84054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84054/testReport)** for PR 19737 at commit [`0704a82`](https://github.com/apache/spark/commit/0704a821eb7913768ba57a91ec4fc234d8941ca8).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19737
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19737
**[Test build #84061 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84061/testReport)** for PR 19737 at commit [`0704a82`](https://github.com/apache/spark/commit/0704a821eb7913768ba57a91ec4fc234d8941ca8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit pr...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19737#discussion_r152099002
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala ---
@@ -154,7 +164,10 @@ object GenerateUnsafeRowJoiner extends CodeGenerator[(StructType, StructType), U
|$putLong(buf, $cursor, $getLong(buf, $cursor) + ($shift << 32));
""".stripMargin
--- End diff --
nit: `s"$putLong(buf, $cursor, $getLong(buf, $cursor) + ($shift << 32));\n"`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19737
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84037/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit pr...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19737#discussion_r152008389
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala ---
@@ -88,7 +112,20 @@ object GenerateUnsafeRowJoiner extends CodeGenerator[(StructType, StructType), U
s"$getLong(obj2, offset2 + ${(i - bitset1Words) * 8})"
}
}
- s"$putLong(buf, ${offset + i * 8}, $bits);"
+ s"$putLong(buf, ${offset + i * 8}, $bits);\n"
+ }
+
+ val functions = mutable.ArrayBuffer.empty[String]
+ val args = "java.lang.Object obj1, long offset1, java.lang.Object obj2, long offset2"
+ val copyBitsets = splitPlatformCode(copyBitset).zipWithIndex.map { case(body, index) =>
--- End diff --
why can't we use `splitExpressions` here?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19737
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84031/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit pr...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19737#discussion_r152011501
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala ---
@@ -166,6 +214,8 @@ object GenerateUnsafeRowJoiner extends CodeGenerator[(StructType, StructType), U
| private byte[] buf = new byte[64];
| private UnsafeRow out = new UnsafeRow(${schema1.size + schema2.size});
|
+ | ${functions.mkString("\n")}
--- End diff --
this can be `${ctx.declareAddedFunctions()}` if we create an codegen context.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit pr...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19737#discussion_r152098735
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala ---
@@ -88,8 +92,14 @@ object GenerateUnsafeRowJoiner extends CodeGenerator[(StructType, StructType), U
s"$getLong(obj2, offset2 + ${(i - bitset1Words) * 8})"
}
}
- s"$putLong(buf, ${offset + i * 8}, $bits);"
- }.mkString("\n")
+ s"$putLong(buf, ${offset + i * 8}, $bits);\n"
+ }
+
+ val functions = mutable.ArrayBuffer.empty[String]
--- End diff --
not needed?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19737
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit pr...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/19737#discussion_r152009717
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala ---
@@ -88,7 +112,20 @@ object GenerateUnsafeRowJoiner extends CodeGenerator[(StructType, StructType), U
s"$getLong(obj2, offset2 + ${(i - bitset1Words) * 8})"
}
}
- s"$putLong(buf, ${offset + i * 8}, $bits);"
+ s"$putLong(buf, ${offset + i * 8}, $bits);\n"
+ }
+
+ val functions = mutable.ArrayBuffer.empty[String]
+ val args = "java.lang.Object obj1, long offset1, java.lang.Object obj2, long offset2"
+ val copyBitsets = splitPlatformCode(copyBitset).zipWithIndex.map { case(body, index) =>
--- End diff --
We do not have `ctx` here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19737
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit pr...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19737
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19737
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19737
**[Test build #84031 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84031/testReport)** for PR 19737 at commit [`65885bf`](https://github.com/apache/spark/commit/65885bf385b31981a0da044b018a83b2021f6d25).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit pr...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19737#discussion_r152011125
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala ---
@@ -88,7 +112,20 @@ object GenerateUnsafeRowJoiner extends CodeGenerator[(StructType, StructType), U
s"$getLong(obj2, offset2 + ${(i - bitset1Words) * 8})"
}
}
- s"$putLong(buf, ${offset + i * 8}, $bits);"
+ s"$putLong(buf, ${offset + i * 8}, $bits);\n"
+ }
+
+ val functions = mutable.ArrayBuffer.empty[String]
+ val args = "java.lang.Object obj1, long offset1, java.lang.Object obj2, long offset2"
+ val copyBitsets = splitPlatformCode(copyBitset).zipWithIndex.map { case(body, index) =>
--- End diff --
then create one. The benefit is code reuse and the nested class optimization
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19737
**[Test build #83801 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83801/testReport)** for PR 19737 at commit [`aef727b`](https://github.com/apache/spark/commit/aef727bb9602ae656cceef74e245c836b7420b7d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19737
**[Test build #84031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84031/testReport)** for PR 19737 at commit [`65885bf`](https://github.com/apache/spark/commit/65885bf385b31981a0da044b018a83b2021f6d25).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19737
thanks, merging to master/2.2!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19737
**[Test build #84037 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84037/testReport)** for PR 19737 at commit [`65885bf`](https://github.com/apache/spark/commit/65885bf385b31981a0da044b018a83b2021f6d25).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19737
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19737
**[Test build #84037 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84037/testReport)** for PR 19737 at commit [`65885bf`](https://github.com/apache/spark/commit/65885bf385b31981a0da044b018a83b2021f6d25).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit pr...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19737#discussion_r152098859
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala ---
@@ -88,8 +92,14 @@ object GenerateUnsafeRowJoiner extends CodeGenerator[(StructType, StructType), U
s"$getLong(obj2, offset2 + ${(i - bitset1Words) * 8})"
}
}
- s"$putLong(buf, ${offset + i * 8}, $bits);"
- }.mkString("\n")
+ s"$putLong(buf, ${offset + i * 8}, $bits);\n"
+ }
+
+ val functions = mutable.ArrayBuffer.empty[String]
+ val args = "java.lang.Object obj1, long offset1, java.lang.Object obj2, long offset2"
--- End diff --
ditto
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit pr...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/19737#discussion_r152013827
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala ---
@@ -88,7 +112,20 @@ object GenerateUnsafeRowJoiner extends CodeGenerator[(StructType, StructType), U
s"$getLong(obj2, offset2 + ${(i - bitset1Words) * 8})"
}
}
- s"$putLong(buf, ${offset + i * 8}, $bits);"
+ s"$putLong(buf, ${offset + i * 8}, $bits);\n"
+ }
+
+ val functions = mutable.ArrayBuffer.empty[String]
+ val args = "java.lang.Object obj1, long offset1, java.lang.Object obj2, long offset2"
+ val copyBitsets = splitPlatformCode(copyBitset).zipWithIndex.map { case(body, index) =>
--- End diff --
Good idea, I forgot that we can create it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19737
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit pr...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19737#discussion_r152185361
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala ---
@@ -88,8 +92,14 @@ object GenerateUnsafeRowJoiner extends CodeGenerator[(StructType, StructType), U
s"$getLong(obj2, offset2 + ${(i - bitset1Words) * 8})"
}
}
- s"$putLong(buf, ${offset + i * 8}, $bits);"
- }.mkString("\n")
+ s"$putLong(buf, ${offset + i * 8}, $bits);\n"
+ }
+
+ val functions = mutable.ArrayBuffer.empty[String]
+ val args = "java.lang.Object obj1, long offset1, java.lang.Object obj2, long offset2"
+ val copyBitsets = ctx.splitExpressions(copyBitset, "copyBitsetFunc",
+ ("java.lang.Object", "obj1") :: ("long", "offset1") ::
+ ("java.lang.Object", "obj2") :: ("long", "offset2") :: Nil)
--- End diff --
Could you fix use the named arguments when calling `splitExpressions`, like what I did in https://github.com/apache/spark/pull/19790?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19737
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83801/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19737: [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19737
**[Test build #84054 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84054/testReport)** for PR 19737 at commit [`0704a82`](https://github.com/apache/spark/commit/0704a821eb7913768ba57a91ec4fc234d8941ca8).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org