[carbondata] branch master updated: [DOC] add performance-tuning with codegen parameters support

[ja...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

jackylk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git


The following commit(s) were added to refs/heads/master by this push:
     new 6455dbc  [DOC] add performance-tuning with codegen parameters support
6455dbc is described below

commit 6455dbc3ca8c799a04cb6e6aa55c960e82eb1d34
Author: litao <li...@126.com>
AuthorDate: Thu Dec 19 11:36:46 2019 +0800

    [DOC] add performance-tuning with codegen parameters support
    
    Add document for performance-tuning with codegen parameters support
    
    This closes #3518
---
 docs/images/codegen.png                          | Bin 0 -> 8302 bytes
 docs/query-with-spark-sql-performance -tuning.md |  58 +++++++++++++++++++++++
 2 files changed, 58 insertions(+)

diff --git a/docs/images/codegen.png b/docs/images/codegen.png
new file mode 100644
index 0000000..4ea7b26
Binary files /dev/null and b/docs/images/codegen.png differ
diff --git a/docs/query-with-spark-sql-performance -tuning.md b/docs/query-with-spark-sql-performance -tuning.md
new file mode 100644
index 0000000..6150d2e
--- /dev/null
+++ b/docs/query-with-spark-sql-performance -tuning.md	
@@ -0,0 +1,58 @@
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one or more 
+    contributor license agreements.  See the NOTICE file distributed with
+    this work for additional information regarding copyright ownership. 
+    The ASF licenses this file to you under the Apache License, Version 2.0
+    (the "License"); you may not use this file except in compliance with 
+    the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+    
+    Unless required by applicable law or agreed to in writing, software 
+    distributed under the License is distributed on an "AS IS" BASIS, 
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and 
+    limitations under the License.
+-->
+
+# Query with spark-sql performacne tuning
+  This tutorial guides you to create CarbonData Tables and optimize performance.
+  The following sections will elaborate on the below topics :
+
+* [The influence of spark.sql.codegen.wholeStage configuration on query](#the-influence-of-codegen-configuration-on-query) 
+
+## The influence of codegen configuration on query 
+
+In practice, we found that when the number of counters(columns applied SUM operator) reaches a certain threshold, the query time increases dramatically. 
+
+As shown in the figure below(spark 2.1):
+
+![File Directory Structure](../docs/images/codegen.png?raw=true)
+
+The horizontal axis is the number of counter, and the vertical axis is the time consumed in seconds.
+
+It can be seen from the figure that when the number of counter exceeds 85, the query time is significantly increased.
+
+After analysis, this problem is related to [`spark.sql.codegen.wholeStage`](https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-whole-stage-codegen.html), which is enabled by default since spark 2.0. and it will do all the *internal optimization possible from the spark catalyst side*. 
+
+**Whole-Stage Java Code Generation** (aka *Whole-Stage CodeGen*) is a physical query optimization in Spark SQL that fuses multiple physical operators (as a subtree of plans that [support code generation](https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-CodegenSupport.html)) together into a single Java function.
+
+Whole-Stage Java Code Generation improves the execution performance of a query by collapsing a query tree into a single optimized function that eliminates virtual function calls and leverages CPU registers for intermediate data.
+
+As the number of counter grows, generated codes for data process become larger. There are nearly 3000 lines of code when 34 counters. Java does not recommend huge methods, which can not make use of JIT and the code will run in interpretation mode. So the performance drops sharply when the counter in query grows.
+
+But unfortunately, spark 2.1 only provides switching capability. User can only choose to turn the function on or off. This leads to a sharp drop in performance when the aggregation operator is too large. If the query scenario has many counters, user can configure "spark.sql.codegen.wholeStage" = false. This depends on the query scenario.
+
+Fortunately, spark 2.3 provide more configuration. Users can better configure this parameter. 
+
+User can check more configuration [here](https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-properties.html#spark.sql.codegen.wholeStage).
+
+Recommended configuration for spark2.3：
+
+| Parameter name                    | value | describe                                                     |
+| --------------------------------- | ----- | ------------------------------------------------------------ |
+| spark.sql.codegen.wholeStage      | true  | (internal) Whether the whole stage (of multiple physical operators) will be compiled into a single Java method (true) or not (false).   <br/> Default: true<br/>Use SQLConf.wholeStageEnabled method to access the current value. |
+| spark.sql.codegen.hugeMethodLimit | 8000  | (internal) The maximum bytecode size of a single compiled Java function generated by whole-stage codegen.<br/>Default: 65535<br/>The default value 65535 is the largest bytecode size possible for a valid Java method. When running on HotSpot, it may be preferable to set the value to 8000 (which is the value of HugeMethodLimit in the OpenJDK JVM settings)<br/>Use SQLConf.hugeMethodLimit method to access the current value. |
+
+
+