You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by li...@apache.org on 2016/10/28 02:28:19 UTC
svn commit: r1766917 - in /kylin/site: docs15/gettingstarted/faq.html
feed.xml
Author: lidong
Date: Fri Oct 28 02:28:18 2016
New Revision: 1766917
URL: http://svn.apache.org/viewvc?rev=1766917&view=rev
Log:
update faq for EMR
Modified:
kylin/site/docs15/gettingstarted/faq.html
kylin/site/feed.xml
Modified: kylin/site/docs15/gettingstarted/faq.html
URL: http://svn.apache.org/viewvc/kylin/site/docs15/gettingstarted/faq.html?rev=1766917&r1=1766916&r2=1766917&view=diff
==============================================================================
--- kylin/site/docs15/gettingstarted/faq.html (original)
+++ kylin/site/docs15/gettingstarted/faq.html Fri Oct 28 02:28:18 2016
@@ -2104,26 +2104,52 @@
<h4 id="binfind-hive-dependencysh-can-locate-hivehcat-jars-in-local-but-kylin-reports-error-like-javalangnoclassdeffounderror-orgapachehivehcatalogmapreducehcatinputformat">1. “bin/find-hive-dependency.sh” can locate hive/hcat jars in local, but Kylin reports error like “java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat”</h4>
<ul>
- <li>
- <p>Kylin need many dependent jars (hadoop/hive/hcat/hbase/kafka) on classpath to work, but Kylin doesn’t ship them. It will seek these jars from your local machine by running commands like <code class="highlighter-rouge">hbase classpath</code>, <code class="highlighter-rouge">hive -e set</code> etc. The founded jars’ path will be appended to the environment variable <em>HBASE_CLASSPATH</em> (Kylin uses <code class="highlighter-rouge">hbase</code> shell command to start up, which will read this). But in some Hadoop distribution (like EMR 5.0), the <code class="highlighter-rouge">hbase</code> shell doesn’t keep the origin <code class="highlighter-rouge">HBASE_CLASSPATH</code> value, that causes the “NoClassDefFoundError”.</p>
- </li>
- <li>
- <p>To fix this, find the hbase shell script (in hbase/bin folder), and search <em>HBASE_CLASSPATH</em>, check whether it overwrite the value like :</p>
- </li>
+ <li>Kylin need many dependent jars (hadoop/hive/hcat/hbase/kafka) on classpath to work, but Kylin doesn’t ship them. It will seek these jars from your local machine by running commands like <code class="highlighter-rouge">hbase classpath</code>, <code class="highlighter-rouge">hive -e set</code> etc. The founded jars’ path will be appended to the environment variable <em>HBASE_CLASSPATH</em> (Kylin uses <code class="highlighter-rouge">hbase</code> shell command to start up, which will read this). But in some Hadoop distribution (like EMR 5.0), the <code class="highlighter-rouge">hbase</code> shell doesn’t keep the origin <code class="highlighter-rouge">HBASE_CLASSPATH</code> value, that causes the “NoClassDefFoundError”.</li>
</ul>
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">export HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*</code></pre></div>
+<p>To fix this, find the hbase shell script (in hbase/bin or hbase/conf folder), and search <em>HBASE_CLASSPATH</em>, check whether it overwrite the value like :</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code> export HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*
+</code></pre>
+</div>
+
+<p>If true, change it to keep the origin value like:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code> export HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*:$HBASE_CLASSPATH
+</code></pre>
+</div>
<ul>
- <li>If true, change it to keep the origin value like:</li>
+ <li>For EMR5.0, need change the hbase-env.sh</li>
</ul>
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">export HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*:$HBASE_CLASSPATH</code></pre></div>
+<div class="highlighter-rouge"><pre class="highlight"><code> sudo vi /usr/lib/hbase/conf/hbase-env.sh
+</code></pre>
+</div>
+
+<p>in around line 30 it was :</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>export HBASE_CLASSPATH=/etc/hadoop/conf
+</code></pre>
+</div>
+
+<p>change to :</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>export HBASE_CLASSPATH=/etc/hadoop/conf:$HBASE_CLASSPATH
+</code></pre>
+</div>
+
+<p>then restart Kylin.</p>
<h4 id="get-javalangillegalargumentexception-too-high-cardinality-is-not-suitable-for-dictionary----cardinality-5220674-in-build-dimension-dictionary-step">2. Get “java.lang.IllegalArgumentException: Too high cardinality is not suitable for dictionary – cardinality: 5220674” in “Build Dimension Dictionary” step</h4>
<ul>
- <li>Kylin uses “Dictionary” encoding to encode/decode the dimension values (check <a href="/blog/2015/08/13/kylin-dictionary/">this blog</a>); Usually a dimension’s cardinality is less than millions, so the “Dict” encoding is good to use. As dictionary need be persisted and loaded into memory, if a dimension’s cardinality is very high, the memory footprint will be tremendous, so Kylin add a check on this. If you see this error, suggest to identify the UHC dimension first and then re-evaluate the design (whether need to make that as dimension?). If must keep it, you can by-pass this error with couple ways: 1) change to use other encoding (like <code class="highlighter-rouge">fixed_length</code>, <code class="highlighter-rouge">integer</code>) 2) or set a bigger value for <code class="highlighter-rouge">kylin.dictionary.max.cardinality</code> in <code class="highlighter-rouge">conf/kylin.properties</code>.</li>
+ <li>Kylin uses “Dictionary” encoding to encode/decode the dimension values (check <a href="/blog/2015/08/13/kylin-dictionary/">this blog</a>); Usually a dimension’s cardinality is less than millions, so the “Dict” encoding is good to use. As dictionary need be persisted and loaded into memory, if a dimension’s cardinality is very high, the memory footprint will be tremendous, so Kylin add a check on this. If you see this error, suggest to identify the UHC dimension first and then re-evaluate the design (whether need to make that as dimension?). If must keep it, you can by-pass this error with couple ways:
+ <ul>
+ <li>change to use other encoding (like <code class="highlighter-rouge">fixed_length</code>, <code class="highlighter-rouge">integer</code>)</li>
+ <li>or set a bigger value for <code class="highlighter-rouge">kylin.dictionary.max.cardinality</code> in <code class="highlighter-rouge">conf/kylin.properties</code>.</li>
+ </ul>
+ </li>
</ul>
<h4 id="build-cube-failed-due-to-error-check-status">3. Build cube failed due to “error check status”</h4>
Modified: kylin/site/feed.xml
URL: http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1766917&r1=1766916&r2=1766917&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Fri Oct 28 02:28:18 2016
@@ -19,8 +19,8 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Wed, 26 Oct 2016 08:06:28 -0700</pubDate>
- <lastBuildDate>Wed, 26 Oct 2016 08:06:28 -0700</lastBuildDate>
+ <pubDate>Fri, 28 Oct 2016 03:24:28 -0700</pubDate>
+ <lastBuildDate>Fri, 28 Oct 2016 03:24:28 -0700</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>