You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by li...@apache.org on 2017/08/23 19:02:27 UTC
spark git commit: [SPARK-21603][SQL][FOLLOW-UP] Change the default
value of maxLinesPerFunction into 4000
Repository: spark
Updated Branches:
refs/heads/master 1662e9311 -> 6942aeeb0
[SPARK-21603][SQL][FOLLOW-UP] Change the default value of maxLinesPerFunction into 4000
## What changes were proposed in this pull request?
This pr changed the default value of `maxLinesPerFunction` into `4000`. In #18810, we had this new option to disable code generation for too long functions and I found this option only affected `Q17` and `Q66` in TPC-DS. But, `Q66` had some performance regression:
```
Q17 w/o #18810, 3224ms --> q17 w/#18810, 2627ms (improvement)
Q66 w/o #18810, 1712ms --> q66 w/#18810, 3032ms (regression)
```
To keep the previous performance in TPC-DS, we better set higher value at `maxLinesPerFunction` by default.
## How was this patch tested?
Existing tests.
Author: Takeshi Yamamuro <ya...@apache.org>
Closes #19021 from maropu/SPARK-21603-FOLLOWUP-1.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6942aeeb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6942aeeb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6942aeeb
Branch: refs/heads/master
Commit: 6942aeeb0a0095a1ba85a817eb9e0edc410e5624
Parents: 1662e93
Author: Takeshi Yamamuro <ya...@apache.org>
Authored: Wed Aug 23 12:02:24 2017 -0700
Committer: gatorsmile <ga...@gmail.com>
Committed: Wed Aug 23 12:02:24 2017 -0700
----------------------------------------------------------------------
.../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/6942aeeb/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
----------------------------------------------------------------------
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 2c7397c..a685099 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -577,10 +577,10 @@ object SQLConf {
.doc("The maximum lines of a single Java function generated by whole-stage codegen. " +
"When the generated function exceeds this threshold, " +
"the whole-stage codegen is deactivated for this subtree of the current query plan. " +
- "The default value 2667 is the max length of byte code JIT supported " +
- "for a single function(8000) divided by 3.")
+ "The default value 4000 is the max length of byte code JIT supported " +
+ "for a single function(8000) divided by 2.")
.intConf
- .createWithDefault(2667)
+ .createWithDefault(4000)
val FILES_MAX_PARTITION_BYTES = buildConf("spark.sql.files.maxPartitionBytes")
.doc("The maximum number of bytes to pack into a single partition when reading files.")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org