You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by li...@apache.org on 2018/07/28 14:09:14 UTC
svn commit: r1836910 - in /kylin/site: cn/docs/tutorial/cube_spark.html
docs/tutorial/cube_spark.html feed.xml
Author: lidong
Date: Sat Jul 28 14:09:14 2018
New Revision: 1836910
URL: http://svn.apache.org/viewvc?rev=1836910&view=rev
Log:
Update configurations for spark engine
Modified:
kylin/site/cn/docs/tutorial/cube_spark.html
kylin/site/docs/tutorial/cube_spark.html
kylin/site/feed.xml
Modified: kylin/site/cn/docs/tutorial/cube_spark.html
URL: http://svn.apache.org/viewvc/kylin/site/cn/docs/tutorial/cube_spark.html?rev=1836910&r1=1836909&r2=1836910&view=diff
==============================================================================
--- kylin/site/cn/docs/tutorial/cube_spark.html (original)
+++ kylin/site/cn/docs/tutorial/cube_spark.html Sat Jul 28 14:09:14 2018
@@ -193,21 +193,30 @@ export KYLIN_HOME=/usr/local/apache-kyli
<p>Kylin å¨ $KYLIN_HOME/spark ä¸åµå
¥ä¸ä¸ª Spark binary (v2.1.2)ï¼ææä½¿ç¨ <em>âkylin.engine.spark-conf.â</em> ä½ä¸ºåç¼ç Spark é
ç½®å±æ§é½è½å¨ $KYLIN_HOME/conf/kylin.properties ä¸è¿è¡ç®¡çãè¿äºå±æ§å½è¿è¡æ交 Spark job æ¶ä¼è¢«æå并åºç¨ï¼ä¾å¦ï¼å¦ææ¨é
ç½® âkylin.engine.spark-conf.spark.executor.memory=4Gâï¼Kylin å°ä¼å¨æ§è¡ âspark-submitâ æä½æ¶ä½¿ç¨ ââconf spark.executor.memory=4Gâ ä½ä¸ºåæ°ã</p>
-<p>è¿è¡ Spark cubing åï¼å»ºè®®æ¥çä¸ä¸è¿äºé
ç½®å¹¶æ ¹æ®æ¨é群çæ
åµè¿è¡èªå®ä¹ãä¸é¢æ¯é»è®¤é
ç½®ï¼ä¹æ¯ sandbox æä½è¦æ±çé
ç½® (1 个 1GB memory ç executor)ï¼é常ä¸ä¸ªé群ï¼éè¦æ´å¤ç executors ä¸æ¯ä¸ä¸ªè³å°æ 4GB memory å 2 cores:</p>
+<p>è¿è¡ Spark cubing åï¼å»ºè®®æ¥çä¸ä¸è¿äºé
ç½®å¹¶æ ¹æ®æ¨é群çæ
åµè¿è¡èªå®ä¹ãä¸é¢æ¯å»ºè®®é
ç½®ï¼å¼å¯äº Spark å¨æèµæºåé
:</p>
<div class="highlight"><pre><code class="language-groff" data-lang="groff">kylin.engine.spark-conf.spark.master=yarn
kylin.engine.spark-conf.spark.submit.deployMode=cluster
+kylin.engine.spark-conf.spark.dynamicAllocation.enabled=true
+kylin.engine.spark-conf.spark.dynamicAllocation.minExecutors=1
+kylin.engine.spark-conf.spark.dynamicAllocation.maxExecutors=1000
+kylin.engine.spark-conf.spark.dynamicAllocation.executorIdleTimeout=300
kylin.engine.spark-conf.spark.yarn.queue=default
+kylin.engine.spark-conf.spark.driver.memory=2G
kylin.engine.spark-conf.spark.executor.memory=4G
kylin.engine.spark-conf.spark.yarn.executor.memoryOverhead=1024
-kylin.engine.spark-conf.spark.executor.cores=2
-kylin.engine.spark-conf.spark.executor.instances=40
+kylin.engine.spark-conf.spark.executor.cores=1
+kylin.engine.spark-conf.spark.network.timeout=600
kylin.engine.spark-conf.spark.shuffle.service.enabled=true
+#kylin.engine.spark-conf.spark.executor.instances=1
kylin.engine.spark-conf.spark.eventLog.enabled=true
+kylin.engine.spark-conf.spark.hadoop.dfs.replication=2
+kylin.engine.spark-conf.spark.hadoop.mapreduce.output.fileoutputformat.compress=true
+kylin.engine.spark-conf.spark.hadoop.mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec
+kylin.engine.spark-conf.spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec
kylin.engine.spark-conf.spark.eventLog.dir=hdfs\:///kylin/spark-history
kylin.engine.spark-conf.spark.history.fs.logDirectory=hdfs\:///kylin/spark-history
-#kylin.engine.spark-conf.spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec
## uncomment for HDP
#kylin.engine.spark-conf.spark.driver.extraJavaOptions=-Dhdp.version=current
Modified: kylin/site/docs/tutorial/cube_spark.html
URL: http://svn.apache.org/viewvc/kylin/site/docs/tutorial/cube_spark.html?rev=1836910&r1=1836909&r2=1836910&view=diff
==============================================================================
--- kylin/site/docs/tutorial/cube_spark.html (original)
+++ kylin/site/docs/tutorial/cube_spark.html Sat Jul 28 14:09:14 2018
@@ -5624,20 +5624,30 @@ export KYLIN_HOME=/usr/local/apache-kyli
<p>Kylin embedes a Spark binary (v2.1.0) in $KYLIN_HOME/spark, all the Spark configurations can be managed in $KYLIN_HOME/conf/kylin.properties with prefix <em>âkylin.engine.spark-conf.â</em>. These properties will be extracted and applied when runs submit Spark job; E.g, if you configure âkylin.engine.spark-conf.spark.executor.memory=4Gâ, Kylin will use ââconf spark.executor.memory=4Gâ as parameter when execute âspark-submitâ.</p>
-<p>Before you run Spark cubing, suggest take a look on these configurations and do customization according to your cluster. Below is the default configurations, which is also the minimal config for a sandbox (1 executor with 1GB memory); usually in a normal cluster, need much more executors and each has at least 4GB memory and 2 cores:</p>
+<p>Before you run Spark cubing, suggest take a look on these configurations and do customization according to your cluster. Below is the recommended configurations:</p>
<div class="highlight"><pre><code class="language-groff" data-lang="groff">kylin.engine.spark-conf.spark.master=yarn
kylin.engine.spark-conf.spark.submit.deployMode=cluster
+kylin.engine.spark-conf.spark.dynamicAllocation.enabled=true
+kylin.engine.spark-conf.spark.dynamicAllocation.minExecutors=1
+kylin.engine.spark-conf.spark.dynamicAllocation.maxExecutors=1000
+kylin.engine.spark-conf.spark.dynamicAllocation.executorIdleTimeout=300
kylin.engine.spark-conf.spark.yarn.queue=default
-kylin.engine.spark-conf.spark.executor.memory=1G
-kylin.engine.spark-conf.spark.executor.cores=2
-kylin.engine.spark-conf.spark.executor.instances=1
+kylin.engine.spark-conf.spark.driver.memory=2G
+kylin.engine.spark-conf.spark.executor.memory=4G
+kylin.engine.spark-conf.spark.yarn.executor.memoryOverhead=1024
+kylin.engine.spark-conf.spark.executor.cores=1
+kylin.engine.spark-conf.spark.network.timeout=600
+kylin.engine.spark-conf.spark.shuffle.service.enabled=true
+#kylin.engine.spark-conf.spark.executor.instances=1
kylin.engine.spark-conf.spark.eventLog.enabled=true
+kylin.engine.spark-conf.spark.hadoop.dfs.replication=2
+kylin.engine.spark-conf.spark.hadoop.mapreduce.output.fileoutputformat.compress=true
+kylin.engine.spark-conf.spark.hadoop.mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec
+kylin.engine.spark-conf.spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec
kylin.engine.spark-conf.spark.eventLog.dir=hdfs\:///kylin/spark-history
kylin.engine.spark-conf.spark.history.fs.logDirectory=hdfs\:///kylin/spark-history
-#kylin.engine.spark-conf.spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec
-
## uncomment for HDP
#kylin.engine.spark-conf.spark.driver.extraJavaOptions=-Dhdp.version=current
#kylin.engine.spark-conf.spark.yarn.am.extraJavaOptions=-Dhdp.version=current
Modified: kylin/site/feed.xml
URL: http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1836910&r1=1836909&r2=1836910&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Sat Jul 28 14:09:14 2018
@@ -19,8 +19,8 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Wed, 25 Jul 2018 06:59:25 -0700</pubDate>
- <lastBuildDate>Wed, 25 Jul 2018 06:59:25 -0700</lastBuildDate>
+ <pubDate>Sat, 28 Jul 2018 06:59:23 -0700</pubDate>
+ <lastBuildDate>Sat, 28 Jul 2018 06:59:23 -0700</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>