You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by li...@apache.org on 2018/06/26 14:08:51 UTC
svn commit: r1834432 - in /kylin/site: docs/release_notes.html
docs/tutorial/cube_spark.html docs/tutorial/use_cube_planner.html
docs23/tutorial/use_cube_planner.html feed.xml
Author: lidong
Date: Tue Jun 26 14:08:50 2018
New Revision: 1834432
URL: http://svn.apache.org/viewvc?rev=1834432&view=rev
Log:
Refine document
Modified:
kylin/site/docs/release_notes.html
kylin/site/docs/tutorial/cube_spark.html
kylin/site/docs/tutorial/use_cube_planner.html
kylin/site/docs23/tutorial/use_cube_planner.html
kylin/site/feed.xml
Modified: kylin/site/docs/release_notes.html
URL: http://svn.apache.org/viewvc/kylin/site/docs/release_notes.html?rev=1834432&r1=1834431&r2=1834432&view=diff
==============================================================================
--- kylin/site/docs/release_notes.html (original)
+++ kylin/site/docs/release_notes.html Tue Jun 26 14:08:50 2018
@@ -4909,7 +4909,7 @@ there are source code package, binary pa
</ul>
<h2 id="v240---2018-06-23">v2.4.0 - 2018-06-23</h2>
-<p><em>Tag:</em> <a href="https://github.com/apache/kylin/tree/kylin-2.3.1">kylin-2.3.1</a><br />
+<p><em>Tag:</em> <a href="https://github.com/apache/kylin/tree/kylin-2.4.0">kylin-2.4.0</a><br />
This is a major release after 2.3.x, with 8 new features and more than 30 bug fixes bug fixes and enhancement. Check <a href="/docs/howto/howto_upgrade.html">How to upgrade</a>.</p>
<p><strong>New Feature</strong><br />
Modified: kylin/site/docs/tutorial/cube_spark.html
URL: http://svn.apache.org/viewvc/kylin/site/docs/tutorial/cube_spark.html?rev=1834432&r1=1834431&r2=1834432&view=diff
==============================================================================
--- kylin/site/docs/tutorial/cube_spark.html (original)
+++ kylin/site/docs/tutorial/cube_spark.html Tue Jun 26 14:08:50 2018
@@ -4901,33 +4901,19 @@ var _hmt = _hmt || [];
<h2 id="preparation">Preparation</h2>
<p>To finish this tutorial, you need a Hadoop environment which has Kylin v2.1.0 or above installed. Here we will use Hortonworks HDP 2.4 Sandbox VM, the Hadoop components as well as Hive/HBase has already been started.</p>
-<h2 id="install-kylin-v210-or-above">Install Kylin v2.1.0 or above</h2>
+<h2 id="install-kylin-v240-or-above">Install Kylin v2.4.0 or above</h2>
-<p>Download the Kylin v2.1.0 for HBase 1.x from Kylinâs download page, and then uncompress the tar ball into <em>/usr/local/</em> folder:</p>
+<p>Download the Kylin binary for HBase 1.x from Kylinâs download page, and then uncompress the tar ball into <em>/usr/local/</em> folder:</p>
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">wget http://www-us.apache.org/dist/kylin/apache-kylin-2.1.0/apache-kylin-2.1.0-bin-hbase1x.tar.gz -P /tmp
+<div class="highlight"><pre><code class="language-groff" data-lang="groff">wget http://www-us.apache.org/dist/kylin/apache-kylin-2.4.0/apache-kylin-2.4.0-bin-hbase1x.tar.gz -P /tmp
-tar -zxvf /tmp/apache-kylin-2.1.0-bin-hbase1x.tar.gz -C /usr/local/
+tar -zxvf /tmp/apache-kylin-2.4.0-bin-hbase1x.tar.gz -C /usr/local/
-export KYLIN_HOME=/usr/local/apache-kylin-2.1.0-bin-hbase1x</code></pre></div>
+export KYLIN_HOME=/usr/local/apache-kylin-2.4.0-bin-hbase1x</code></pre></div>
<h2 id="prepare-kylinenvhadoop-conf-dir">Prepare âkylin.env.hadoop-conf-dirâ</h2>
-<p>To run Spark on Yarn, need specify <strong>HADOOP_CONF_DIR</strong> environment variable, which is the directory that contains the (client side) configuration files for Hadoop. In many Hadoop distributions the directory is â/etc/hadoop/confâ; But Kylin not only need access HDFS, Yarn and Hive, but also HBase, so the default directory might not have all necessary files. In this case, you need create a new directory and then copying or linking those client files (core-site.xml, hdfs-site.xml, yarn-site.xml, hive-site.xml and hbase-site.xml) there. In HDP 2.4, there is a conflict between hive-tez and Spark, so need change the default engine from âtezâ to âmrâ when copy for Kylin.</p>
-
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">mkdir $KYLIN_HOME/hadoop-conf
-ln -s /etc/hadoop/conf/core-site.xml $KYLIN_HOME/hadoop-conf/core-site.xml
-ln -s /etc/hadoop/conf/hdfs-site.xml $KYLIN_HOME/hadoop-conf/hdfs-site.xml
-ln -s /etc/hadoop/conf/yarn-site.xml $KYLIN_HOME/hadoop-conf/yarn-site.xml
-ln -s /etc/hbase/2.4.0.0-169/0/hbase-site.xml $KYLIN_HOME/hadoop-conf/hbase-site.xml
-cp /etc/hive/2.4.0.0-169/0/hive-site.xml $KYLIN_HOME/hadoop-conf/hive-site.xml
-vi $KYLIN_HOME/hadoop-conf/hive-site.xml (change "hive.execution.engine" value from "tez" to "mr")</code></pre></div>
-
-<p>Now, let Kylin know this directory with property âkylin.env.hadoop-conf-dirâ in kylin.properties:</p>
-
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">kylin.env.hadoop-conf-dir=/usr/local/apache-kylin-2.1.0-bin-hbase1x/hadoop-conf</code></pre></div>
-
-<p>If this property isnât set, Kylin will use the directory that âhive-site.xmlâ locates in; while that folder may have no âhbase-site.xmlâ, will get HBase/ZK connection error in Spark.</p>
+<p>To run Spark on Yarn, need specify <strong>HADOOP_CONF_DIR</strong> environment variable, which is the directory that contains the (client side) configuration files for Hadoop. In many Hadoop distributions the directory is â/etc/hadoop/confâ; Kylin can automatically detect this folder from Hadoop configuration, so by default you donât need to set this property. If your configuration files are not in default folder, please set this property explicitly.</p>
<h2 id="check-spark-configuration">Check Spark configuration</h2>
@@ -5006,7 +4992,7 @@ $KYLIN_HOME/bin/kylin.sh start</code></p
<p>When getting error, you should check âlogs/kylin.logâ firstly. There has the full Spark command that Kylin executes, e.g:</p>
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">2017-03-06 14:44:38,574 INFO [Job 2d5c1178-c6f6-4b50-8937-8e5e3b39227e-306] spark.SparkExecutable:121 : cmd:export HADOOP_CONF_DIR=/usr/local/apache-kylin-2.1.0-bin-hbase1x/hadoop-conf && /usr/local/apache-kylin-2.1.0-bin-hbase1x/spark/bin/spark-submit --class org.apache.kylin.common.util.SparkEntry --conf spark.executor.instances=1 --conf spark.yarn.queue=default --conf spark.yarn.am.extraJavaOptions=-Dhdp.version=current --conf spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf spark.driver.extraJavaOptions=-Dhdp.version=current --conf spark.master=yarn --conf spark.executor.extraJavaOptions=-Dhdp.version=current --conf spark.executor.memory=1G --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs:///kylin/spark-history --conf spark.executor.cores=2 --conf spark.submit.deployMode=cluster --files /etc/hbase/2.4.0.0-169/0/hbase-site.xml --jars /usr/hdp/2.4.0.0-16
9/hbase/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/2.4.0.0-169/hbase/lib/hbase-client-1.1.2.2.4.0.0-169.jar,/usr/hdp/2.4.0.0-169/hbase/lib/hbase-common-1.1.2.2.4.0.0-169.jar,/usr/hdp/2.4.0.0-169/hbase/lib/hbase-protocol-1.1.2.2.4.0.0-169.jar,/usr/hdp/2.4.0.0-169/hbase/lib/metrics-core-2.2.0.jar,/usr/hdp/2.4.0.0-169/hbase/lib/guava-12.0.1.jar, /usr/local/apache-kylin-2.1.0-bin-hbase1x/lib/kylin-job-2.1.0.jar -className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e -segmentId 555c4d32-40bb-457d-909a-1bb017bf2d9e -confPath /usr/local/apache-kylin-2.1.0-bin-hbase1x/conf -output hdfs:///kylin/kylin_metadata/kylin-2d5c1178-c6f6-4b50-8937-8e5e3b39227e/kylin_sales_cube/cuboid/ -cubename kylin_sales_cube</code></pre></div>
+<div class="highlight"><pre><code class="language-groff" data-lang="groff">2017-03-06 14:44:38,574 INFO [Job 2d5c1178-c6f6-4b50-8937-8e5e3b39227e-306] spark.SparkExecutable:121 : cmd:export HADOOP_CONF_DIR=/etc/hadoop/conf && /usr/local/apache-kylin-2.4.0-bin-hbase1x/spark/bin/spark-submit --class org.apache.kylin.common.util.SparkEntry --conf spark.executor.instances=1 --conf spark.yarn.queue=default --conf spark.yarn.am.extraJavaOptions=-Dhdp.version=current --conf spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf spark.driver.extraJavaOptions=-Dhdp.version=current --conf spark.master=yarn --conf spark.executor.extraJavaOptions=-Dhdp.version=current --conf spark.executor.memory=1G --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs:///kylin/spark-history --conf spark.executor.cores=2 --conf spark.submit.deployMode=cluster --files /etc/hbase/2.4.0.0-169/0/hbase-site.xml /usr/local/apache-kylin-2.4.0-bin-hbase1x/lib/kylin-job-2.4.0.j
ar -className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e -segmentId 555c4d32-40bb-457d-909a-1bb017bf2d9e -confPath /usr/local/apache-kylin-2.4.0-bin-hbase1x/conf -output hdfs:///kylin/kylin_metadata/kylin-2d5c1178-c6f6-4b50-8937-8e5e3b39227e/kylin_sales_cube/cuboid/ -cubename kylin_sales_cube</code></pre></div>
<p>You can copy the cmd to execute manually in shell and then tunning the parameters quickly; During the execution, you can access Yarn resource manager to check more. If the job has already finished, you can check the history info in Spark history server.</p>
Modified: kylin/site/docs/tutorial/use_cube_planner.html
URL: http://svn.apache.org/viewvc/kylin/site/docs/tutorial/use_cube_planner.html?rev=1834432&r1=1834431&r2=1834432&view=diff
==============================================================================
--- kylin/site/docs/tutorial/use_cube_planner.html (original)
+++ kylin/site/docs/tutorial/use_cube_planner.html Tue Jun 26 14:08:50 2018
@@ -4910,9 +4910,11 @@ var _hmt = _hmt || [];
<p><img src="/images/CubePlanner/CubePlanner.png" alt="CubePlanner" /></p>
+<p>Read more at <a href="https://www.ebayinc.com/stories/blogs/tech/cube-planner-build-an-apache-kylin-olap-cube-efficiently-and-intelligently/">eBay tech blog</a></p>
+
<h2 id="prerequisites">Prerequisites</h2>
-<p>To enable Dashboard on WebUI, you need to set <strong>kylin.cube.cubeplanner.enabled=true</strong> and other properties in <strong>kylin.properties</strong>.</p>
+<p>To enable Dashboard on WebUI, you need to set <code class="highlighter-rouge">kylin.cube.cubeplanner.enabled=true</code> and other properties in<code class="highlighter-rouge">kylin.properties</code></p>
<div class="highlight"><pre><code class="language-groff" data-lang="groff">kylin.cube.cubeplanner.enabled=true
kylin.server.query-metrics2-enabled=true
@@ -4936,9 +4938,7 @@ kylin.metrics.monitor-enabled=true</code
<p>You should make sure the status of the Cube is â<strong>READY</strong>â</p>
-<p>If the status of the Cube is â<strong>DISABLED</strong>â<img src="/images/CubePlanner/Status_Disabled.png" alt="Status_Disabled" />, you will not be able to use the Cube planner.</p>
-
-<p>You should change the status of the Cube from â<strong>DISABLED</strong>â to â<strong>READY</strong>â by building it or enabling it if it has been built before.</p>
+<p>If the status of the Cube is â<strong>DISABLED</strong>â, you will not be able to use the Cube planner. You should change the status of the Cube from â<strong>DISABLED</strong>â to â<strong>READY</strong>â by building it or enabling it if it has been built before.</p>
<h4 id="step-3">Step 3:</h4>
@@ -4946,7 +4946,7 @@ kylin.metrics.monitor-enabled=true</code
<ul>
<li>
- <p>The data will be displayed in <em><a href="https://en.wikipedia.org/wiki/Pie_chart#Ring_chart_.2F_Sunburst_chart_.2F_Multilevel_pie_chart">Sunburst Chart</a></em>.</p>
+ <p>The data will be displayed in Sunburst Chart.</p>
</li>
<li>
<p>Each part refers to a cuboid, is shown in different colors determined by the query <strong>frequency</strong> against this cuboid.</p>
@@ -5036,6 +5036,8 @@ kylin.metrics.monitor-enabled=true</code
</li>
</ul>
+<p>Please note: if you donât see the last optimized time, upgrade to Kylin v2.3.2 or above, check KYLIN-3404.</p>
+
<p><img src="/images/CubePlanner/column_name+optimize_time.png" alt="column name+optimize time" /></p>
<ul>
Modified: kylin/site/docs23/tutorial/use_cube_planner.html
URL: http://svn.apache.org/viewvc/kylin/site/docs23/tutorial/use_cube_planner.html?rev=1834432&r1=1834431&r2=1834432&view=diff
==============================================================================
--- kylin/site/docs23/tutorial/use_cube_planner.html (original)
+++ kylin/site/docs23/tutorial/use_cube_planner.html Tue Jun 26 14:08:50 2018
@@ -4911,9 +4911,11 @@ var _hmt = _hmt || [];
<p><img src="/images/CubePlanner/CubePlanner.png" alt="CubePlanner" /></p>
+<p>Read more at <a href="https://www.ebayinc.com/stories/blogs/tech/cube-planner-build-an-apache-kylin-olap-cube-efficiently-and-intelligently/">eBay tech blog</a></p>
+
<h2 id="prerequisites">Prerequisites</h2>
-<p>To enable Dashboard on WebUI, you need to set <strong>kylin.cube.cubeplanner.enabled=true</strong> and other properties in <strong>kylin.properties</strong>.</p>
+<p>To enable Dashboard on WebUI, you need to set <code class="highlighter-rouge">kylin.cube.cubeplanner.enabled=true</code> and other properties in <code class="highlighter-rouge">kylin.properties</code></p>
<div class="highlight"><pre><code class="language-groff" data-lang="groff">kylin.cube.cubeplanner.enabled=true
kylin.server.query-metrics2-enabled=true
@@ -4937,7 +4939,7 @@ kylin.metrics.monitor-enabled=true</code
<p>You should make sure the status of the Cube is â<strong>READY</strong>â</p>
-<p>If the status of the Cube is â<strong>DISABLED</strong>â<img src="/images/CubePlanner/Status_Disabled.png" alt="Status_Disabled" />, you will not be able to use the Cube planner.</p>
+<p>If the status of the Cube is â<strong>DISABLED</strong>â, you will not be able to use the Cube planner.</p>
<p>You should change the status of the Cube from â<strong>DISABLED</strong>â to â<strong>READY</strong>â by building it or enabling it if it has been built before.</p>
@@ -4947,7 +4949,7 @@ kylin.metrics.monitor-enabled=true</code
<ul>
<li>
- <p>The data will be displayed in <em><a href="https://en.wikipedia.org/wiki/Pie_chart#Ring_chart_.2F_Sunburst_chart_.2F_Multilevel_pie_chart">Sunburst Chart</a></em>.</p>
+ <p>The data will be displayed in <strong>Sunburst Chart</strong>.</p>
</li>
<li>
<p>Each part refers to a cuboid, is shown in different colors determined by the query <strong>frequency</strong> against this cuboid.</p>
@@ -5037,6 +5039,8 @@ kylin.metrics.monitor-enabled=true</code
</li>
</ul>
+<p>Please note: if you donât see the last optimized time, upgrade to Kylin v2.3.2 or above, check KYLIN-3404.</p>
+
<p><img src="/images/CubePlanner/column_name+optimize_time.png" alt="column name+optimize time" /></p>
<ul>
Modified: kylin/site/feed.xml
URL: http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1834432&r1=1834431&r2=1834432&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Tue Jun 26 14:08:50 2018
@@ -19,8 +19,8 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Mon, 25 Jun 2018 06:59:04 -0700</pubDate>
- <lastBuildDate>Mon, 25 Jun 2018 06:59:04 -0700</lastBuildDate>
+ <pubDate>Tue, 26 Jun 2018 06:59:25 -0700</pubDate>
+ <lastBuildDate>Tue, 26 Jun 2018 06:59:25 -0700</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>