You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by lw-lin <gi...@git.apache.org> on 2016/07/26 14:23:43 UTC
[GitHub] spark pull request #14370: [SPARK-16713][SQL] Check codegen method size ≤ ...
GitHub user lw-lin opened a pull request:
https://github.com/apache/spark/pull/14370
[SPARK-16713][SQL] Check codegen method size \u2264 8K on compile
## What changes were proposed in this pull request?
Ideally, we would wish codegen methods to be less than 8KB for bytecode size. Beyond 8K JIT won't compile and can cause performance degradation.
Instead of understanding the generated source code and automatically breaking large methods into smaller ones (which is also a little hard to do), it'd be better we discover large methods asap and then manually improve the source code.
This patch adds support for checking codegen method size on compile. We can specify for each method what is the expected behavior when it exceeds 8KB for bytecode size:
```scala
/** No-op when the method exceeds 8K size. */
case object NO_OP extends FunctionSizeHint
/** Log a warning when the method exceeds 8K size. */
case object WARN_IF_EXCEEDS_JIT_LIMIT extends FunctionSizeHint
/**
* Throw a compilation exception when the method exceeds 8K size.
* Fail fast so that we can catch it asap; this is useful in testing corner/edge cases.
*/
case object ERROR_IF_EXCEEDS_JIT_LIMIT extends FunctionSizeHint
```
This way we can test against some extreme case such as a 10000-columns-wide table, to see if the generated code is small enough to get a chance to be JITed at runtime.
## Sample Usage
sample usage:
```scala
val codeBody =
s"""
public static void inc() {
int i = 0;
i++;
i++;
... // enough i++ s for this inc() methods to exceed 8K size
}
"""
// == prior to this patch ===================
ctx = new CodegenContext()
ctx.addNewFunction("inc", genCode(15000))
CodeGenerator.compile(
new CodeAndComment(ctx.declareAddedFunctions(), emptyComments))
// == after this patch ======================
// Exception: failed to compile. Method org.apache.spark.sql.catalyst.expressions.GeneratedClass.inc
// should not exceed 8K size limit -- observed size is 45003
ctx = new CodegenContext()
ctx.addNewFunction("inc", genCode(15000), CodegenContext.ERROR_IF_EXCEEDS_JIT_LIMIT)
CodeGenerator.compile(
new CodeAndComment(ctx.declareAddedFunctions(), emptyComments, ctx.getFuncToSizeHintMap))
```
## How was this patch tested?
- new unit test
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lw-lin/spark codegen-method-size-8k
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14370.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14370
----
commit 19b6d1d056f2bd1b1bb4d502c31b418ffbfe8d65
Author: Liwei Lin <lw...@gmail.com>
Date: 2016-07-26T08:39:36Z
Check codegen method size \u2264 8K on compile
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14370: [SPARK-16713][SQL] Check codegen method size ≤ 8K on c...
Posted by lw-lin <gi...@git.apache.org>.
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/14370
@rxin @cloud-fan sorry for the ambiguity, but we're not enforcing this. There are 3 levels:
- when `NO_OP` is on (which is the fault), we do nothing at all to a huge method;
- when `WARN_...` is on, we'll just log a warning;
- when `ERROR...` is on, we fail fast.
So by default, this does not affect anything on end-user's side. We should turn on `ERROR_IF_EXCEEDS_JIT_LIMIT` **_only in test suits_** and only against critical methods, so that we can find performance issues asap because it fails fast.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14370: [SPARK-16713][SQL] Check codegen method size ≤ 8K on c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14370
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14370: [SPARK-16713][SQL] Check codegen method size ≤ 8K on c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14370
**[Test build #62891 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62891/consoleFull)** for PR 14370 at commit [`946797d`](https://github.com/apache/spark/commit/946797df98cb7f439bf50c1cd994aea79da1742a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14370: [SPARK-16713][SQL] Check codegen method size ≤ 8K on c...
Posted by lw-lin <gi...@git.apache.org>.
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/14370
@davies would you also take a look? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14370: [SPARK-16713][SQL] Check codegen method size ≤ ...
Posted by lw-lin <gi...@git.apache.org>.
Github user lw-lin closed the pull request at:
https://github.com/apache/spark/pull/14370
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14370: [SPARK-16713][SQL] Check codegen method size ≤ 8K on c...
Posted by ooq <gi...@git.apache.org>.
Github user ooq commented on the issue:
https://github.com/apache/spark/pull/14370
Thanks for the PR @lw-lin . I think this check would be useful and ideally `WARN_IF_EXCEEDS_JIT_LIMIT` should be on by default if the overhead is minimal.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14370: [SPARK-16713][SQL] Check codegen method size ≤ 8K on c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14370
**[Test build #62890 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62890/consoleFull)** for PR 14370 at commit [`19b6d1d`](https://github.com/apache/spark/commit/19b6d1d056f2bd1b1bb4d502c31b418ffbfe8d65).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class CodeAndComment(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14370: [SPARK-16713][SQL] Check codegen method size ≤ 8K on c...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/14370
Why are we enforcing this? Users can turn on compile huge methods at the VM level.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14370: [SPARK-16713][SQL] Check codegen method size ≤ 8K on c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14370
**[Test build #62890 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62890/consoleFull)** for PR 14370 at commit [`19b6d1d`](https://github.com/apache/spark/commit/19b6d1d056f2bd1b1bb4d502c31b418ffbfe8d65).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14370: [SPARK-16713][SQL] Check codegen method size ≤ 8K on c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14370
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62890/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14370: [SPARK-16713][SQL] Check codegen method size ≤ 8K on c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14370
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62891/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14370: [SPARK-16713][SQL] Check codegen method size ≤ 8K on c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14370
**[Test build #62891 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62891/consoleFull)** for PR 14370 at commit [`946797d`](https://github.com/apache/spark/commit/946797df98cb7f439bf50c1cd994aea79da1742a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14370: [SPARK-16713][SQL] Check codegen method size ≤ 8K on c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14370
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14370: [SPARK-16713][SQL] Check codegen method size ≤ 8K on c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14370
+1 on @rxin 's idea, I think these checks should belong to VM, the 8k JIT limit is not the case for all VMs right? and maybe configurable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org