You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by nj...@apache.org on 2019/01/13 05:04:15 UTC
svn commit: r1851193 - in /kylin/site: cn/download/index.html
download/index.html feed.xml
Author: nju_yaho
Date: Sun Jan 13 05:04:15 2019
New Revision: 1851193
URL: http://svn.apache.org/viewvc?rev=1851193&view=rev
Log:
update download documents
Modified:
kylin/site/cn/download/index.html
kylin/site/download/index.html
kylin/site/feed.xml
Modified: kylin/site/cn/download/index.html
URL: http://svn.apache.org/viewvc/kylin/site/cn/download/index.html?rev=1851193&r1=1851192&r2=1851193&view=diff
==============================================================================
--- kylin/site/cn/download/index.html (original)
+++ kylin/site/cn/download/index.html Sun Jan 13 05:04:15 2019
@@ -174,6 +174,19 @@ var _hmt = _hmt || [];
</header>
<p>æ¨å¯ä»¥æç
§è¿äº<a href="https://www.apache.org/info/verification.html">æ¥éª¤</a> 并使ç¨è¿äº<a href="https://www.apache.org/dist/kylin/KEYS">KEYS</a>æ¥éªè¯ä¸è½½æ件çæææ§.</p>
+<h4 id="v260">v2.6.0</h4>
+<ul>
+ <li>è¿æ¯2.5 çæ¬åçä¸ä¸ªä¸»è¦åå¸çæ¬ï¼å
å«äº94个é®é¢çä¿®å¤ä»¥ååç§æ¹è¿ãå
³äºå
·ä½å
容请æ¥çåå¸è¯´æ.</li>
+ <li><a href="/docs/release_notes.html">åå¸è¯´æ</a> and <a href="/docs/howto/howto_upgrade.html">å级æå</a></li>
+ <li>æºç ä¸è½½: <a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-source-release.zip">apache-kylin-2.6.0-source-release.zip</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-source-release.zip.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-source-release.zip.sha256">sha256</a>]</li>
+ <li>äºè¿å¶å
ä¸è½½:
+ <ul>
+ <li>for HBase 1.x (includes HDP 2.3+, AWS EMR 5.0+, Azure HDInsight 3.4 - 3.6) - <a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-bin-hbase1x.tar.gz">apache-kylin-2.6.0-bin-hbase1x.tar.gz</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-bin-hbase1x.tar.gz.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-bin-hbase1x.tar.gz.sha256">sha256</a>]</li>
+ <li>for CDH 5.7+ - <a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-bin-cdh57.tar.gz">apache-kylin-2.6.0-bin-cdh57.tar.gz</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-bin-cdh57.tar.gz.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-bin-cdh57.tar.gz.sha256">sha256</a>]</li>
+ </ul>
+ </li>
+</ul>
+
<h4 id="v252">v2.5.2</h4>
<ul>
<li>è¿æ¯2.5.1 çæ¬åçä¸ä¸ªbugfixåå¸çæ¬ï¼å
å«äº12 个以ååç§æ¹è¿ãå
³äºå
·ä½å
容请æ¥çåå¸è¯´æ.</li>
@@ -193,18 +206,18 @@ var _hmt = _hmt || [];
</li>
</ul>
-<h4 id="jdbc-">JDBC 驱å¨ç¨åº</h4>
+<h4 id="jdbc-驱å¨ç¨åº">JDBC 驱å¨ç¨åº</h4>
<p>Kylin JDBC 驱å¨ç¨åº, <code class="highlighter-rouge">kylin-jdbc-<version>.jar</code>, å¨ Kylin äºè¿å¶å
ç âlibâ ç®å½ä¸.</p>
-<h4 id="odbc-">ODBC 驱å¨ç¨åº</h4>
+<h4 id="odbc-驱å¨ç¨åº">ODBC 驱å¨ç¨åº</h4>
<ul>
<li><a href="http://kylin.apache.org/download/KylinODBCDriver-2.1.0.zip">Kylin ODBC é©±å¨ v2.1.0</a> ï¼ä¸ææ Kylin çæ¬å
¼å®¹ï¼</li>
</ul>
<p>注: Kylin ODBC 驱å¨ä¾èµ <a href="http://www.microsoft.com/en-us/download/details.aspx?id=30679">Microsoft Visual C++ 2012 Redistributable</a>ã</p>
-<h4 id="section">以åççæ¬</h4>
+<h4 id="以åççæ¬">以åççæ¬</h4>
<p>Apache Kylinçæ§çæ¬å¯ä»¥ä» <a href="https://archive.apache.org/dist/kylin/">https://archive.apache.org/dist/kylin/</a> ä¸è½½ã</p>
</div>
Modified: kylin/site/download/index.html
URL: http://svn.apache.org/viewvc/kylin/site/download/index.html?rev=1851193&r1=1851192&r2=1851193&view=diff
==============================================================================
--- kylin/site/download/index.html (original)
+++ kylin/site/download/index.html Sun Jan 13 05:04:15 2019
@@ -6033,6 +6033,19 @@ var _hmt = _hmt || [];
</header>
<p>You can verify the download by following these <a href="https://www.apache.org/info/verification.html">procedures</a> and using these <a href="https://www.apache.org/dist/kylin/KEYS">KEYS</a>.</p>
+<h4 id="v260">v2.6.0</h4>
+<ul>
+ <li>This is a major release after 2.5, with 94 bug fixes and enhancement. Check the release notes.</li>
+ <li><a href="/docs/release_notes.html">Release notes</a> and <a href="/docs/howto/howto_upgrade.html">upgrade guide</a></li>
+ <li>Source download: <a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-source-release.zip">apache-kylin-2.6.0-source-release.zip</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-source-release.zip.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-source-release.zip.sha256">sha256</a>]</li>
+ <li>Binary download:
+ <ul>
+ <li>for HBase 1.x (includes HDP 2.3+, AWS EMR 5.0+, Azure HDInsight 3.4 - 3.6) - <a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-bin-hbase1x.tar.gz">apache-kylin-2.6.0-bin-hbase1x.tar.gz</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-bin-hbase1x.tar.gz.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-bin-hbase1x.tar.gz.sha256">sha256</a>]</li>
+ <li>for CDH 5.7+ - <a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-bin-cdh57.tar.gz">apache-kylin-2.6.0-bin-cdh57.tar.gz</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-bin-cdh57.tar.gz.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-2.6.0/apache-kylin-2.6.0-bin-cdh57.tar.gz.sha256">sha256</a>]</li>
+ </ul>
+ </li>
+</ul>
+
<h4 id="v252">v2.5.2</h4>
<ul>
<li>This is a bugfix release after 2.5.1, with 12 bug fixes and enhancement. Check the release notes.</li>
Modified: kylin/site/feed.xml
URL: http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1851193&r1=1851192&r2=1851193&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Sun Jan 13 05:04:15 2019
@@ -19,8 +19,8 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Thu, 10 Jan 2019 05:59:22 -0800</pubDate>
- <lastBuildDate>Thu, 10 Jan 2019 05:59:22 -0800</lastBuildDate>
+ <pubDate>Sat, 12 Jan 2019 20:42:57 -0800</pubDate>
+ <lastBuildDate>Sat, 12 Jan 2019 20:42:57 -0800</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>
@@ -31,12 +31,12 @@
<p>During the Apache Kylin Meetup in August 2018, the Meituan team shared their Kylin on Druid (KoD) solution. Why did they develop this hybrid system? Whatâs the rationale behind it? This article will answer these questions and help you to understand the differences and the pros and cons of each OLAP engine.</p>
-<h2 id="introduction-to-apache-kylin">01 Introduction to Apache Kylin</h2>
+<h2 id="01-introduction-to-apache-kylin">01 Introduction to Apache Kylin</h2>
<p>Apache Kylin is an open source distributed big data analytics engine. It constructs data models on top of huge datasets, builds pre-calculated Cubes to support multi-dimensional analysis, and provides a SQL query interface and multi-dimensional analysis on top of Hadoop, with general ODBC, JDBC, and RESTful API interfaces. Apache Kylinâs unique pre-calculation ability enables it to handle extremely large datasets with sub-second query response times.<br />
<img src="/images/blog/Kylin-On-Durid/1 kylin_architecture.png" alt="" /><br />
Graphic 1 Kylin Architecture</p>
-<h2 id="apache-kylins-advantage">02 Apache Kylinâs Advantage</h2>
+<h2 id="02-apache-kylins-advantage">02 Apache Kylinâs Advantage</h2>
<ol>
<li>The mature, Hadoop-based computing engines (MapReduce and Spark) that provide strong capability of pre-calculation on super large datasets, which can be deployed out-of-the-box on any mainstream Hadoop platform.</li>
<li>Support of ANSI SQL that allows users to do data analysis with SQL directly.</li>
@@ -48,34 +48,34 @@ Graphic 1 Kylin Architecture</p>
<li>Support of both batch loading of super large historical datasets and micro-batches of data streams.</li>
</ol>
-<h2 id="introduction-to-apache-druid-incubating">03 Introduction to Apache Druid (incubating)</h2>
+<h2 id="03-introduction-to-apache-druid-incubating">03 Introduction to Apache Druid (incubating)</h2>
<p>Druid was created in 2012. Itâs an open source distributed data store. Its core design combines the concept of analytical databases, time-series databases, and search systems, and it can support data collection and analytics on fairly large datasets. Druid uses an Apache V2 license and is an Apache incubator project.</p>
<p>Druid Architecture<br />
From the perspective of deployment architectures, Druidâs processes mostly fall into 3 categories based on their roles.</p>
-<h3 id="data-node-slave-node-for-data-ingestion-and-calculation">⢠Data Node (Slave node for data ingestion and calculation)</h3>
+<h3 id="-data-node-slave-node-for-data-ingestion-and-calculation">⢠Data Node (Slave node for data ingestion and calculation)</h3>
<p>The Historical node is in charge of loading segments (committed immutable data) and receiving queries on historical data.<br />
Middle Manager is in charge of data ingestion and commit segments. Each task is done by a separate JVM. <br />
Peon is in charge of completing a single task, which is managed and monitored by the Middle Manager.</p>
-<h3 id="query-node">⢠Query Node</h3>
+<h3 id="-query-node">⢠Query Node</h3>
<p>Broker receives query requests, determines on which segment the data resides, and distributes sub-queries and merges query results.</p>
-<h3 id="master-node-task-coordinator-and-cluster-manager">⢠Master Node (Task Coordinator and Cluster Managerï¼</h3>
+<h3 id="-master-node-task-coordinator-and-cluster-manager">⢠Master Node (Task Coordinator and Cluster Managerï¼</h3>
<p>Coordinator monitors Historical nodes, dispatches segments and monitor workload.<br />
Overlord monitors Middle Manager, dispatches tasks to Middle Manager, and assists releasing of segments.</p>
<h3 id="external-dependency">External Dependency</h3>
<p>At the same time, Druid has 3 replaceable external dependencies.</p>
-<h3 id="deep-storage-distributed-storage">⢠Deep Storage (distributed storage)</h3>
+<h3 id="-deep-storage-distributed-storage">⢠Deep Storage (distributed storage)</h3>
<p>Druid uses Deep storage to transfer data files between nodes.</p>
-<h3 id="metadata-storage">⢠Metadata Storage</h3>
+<h3 id="-metadata-storage">⢠Metadata Storage</h3>
<p>Metadata Storage stores the metadata about segment positions and task output.</p>
-<h3 id="zookeeper-cluster-management-and-task-coordination">⢠Zookeeper (cluster management and task coordination)</h3>
+<h3 id="-zookeeper-cluster-management-and-task-coordination">⢠Zookeeper (cluster management and task coordination)</h3>
<p>Druid uses Zookeeper (ZK) to ensure consistency of the cluster status.<br />
<img src="/images/blog/Kylin-On-Durid/2 druid_architecture.png" alt="" /><br />
Graphic 2 Druid Architecture</p>
@@ -98,7 +98,7 @@ Graphic 4 Druid Schema</p>
<li>Separation of cold/hot data.</li>
</ol>
-<h2 id="why-did-meituan-develop-kylin-on-druid">04 Why did Meituan develop Kylin on Druid?</h2>
+<h2 id="04-why-did-meituan-develop-kylin-on-druid">04 Why did Meituan develop Kylin on Druid?</h2>
<p>Meituan deployed into production an offline OLAP platform with Apache Kylin as its core component in 2015. Since then the platform has served almost all business lines with fast growing data volume and query executions, and the stress on the cluster has increased accordingly. Throughout the time, the tech team in Meituan keeps exploring better solutions for some of Kylinâs challenges. The major one is Apache HBase, the storage that Kylin relies on.</p>
<p>Kylin stores its data in HBase by converting the Dimensions and Measures into HBase Keys and Values, respectively. As HBase doesnât support secondary index and only has one RowKey index, Kylinâs Dimension values will be combined into a fixed sequence to store as RowKey. In this way, filtering on a Dimension in the front of the sequence will perform better than those at the back. Hereâs an example:</p>
@@ -112,14 +112,13 @@ Graphic 6 Cube2 RowKey Sequence</p>
<p><strong>Now letâs query each Cube with the same SQL and compare the response time.</strong></p>
-<div class="highlighter-rouge"><pre class="highlight"><code>select S_SUPPKEY, C_CUSTKEY, sum(LO_EXTENDEDPRICE) as m1
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>select S_SUPPKEY, C_CUSTKEY, sum(LO_EXTENDEDPRICE) as m1
from P_LINEORDER
left join SUPPLIER on P_LINEORDER.LO_SUPPKEY = SUPPLIER.S_SUPPKEY
left join CUSTOMER on P_LINEORDER.LO_CUSTKEY = CUSTOMER.C_CUSTKEY
WHERE (LO_ORDERKEY &gt; 1799905 and LO_ORDERKEY &lt; 1799915) or (LO_ORDERKEY &gt; 1999905 and LO_ORDERKEY &lt; 1999935)
GROUP BY S_SUPPKEY, C_CUSTKEY;
-</code></pre>
-</div>
+</code></pre></div></div>
<p><strong>Below shows the time consumed and data scanned:</strong><br />
<img src="/images/blog/Kylin-On-Durid/7 cube1_query_log.png" alt="" /><br />
@@ -136,11 +135,12 @@ Graphic 8 Cube1 Query Log</p>
<p>Kylinâs query performance and user experience can be greatly improved with pure columnar storage and multiple indexes on Dimensions. As analyzed above, Druid happens to meet the requirements of columnar + multi-index. So the Meituan Kylin team decided to try replacing HBase with Druid.</p>
-<p>Why not just use Druid then? Meituanâs engineers shared their thoughts:<br />
-1. Druidâs native query language is in its own specific JSON format, which is not as easy to pick up as SQL. Although the Druid community added SQL support later on, the support is not complete and does not meet the data analystsâ requirement of complex SQL queries. On the contrary, Kylin natively supports ANSI SQL, uses Apache Calcite for semantic parsing, and supports SQL features such as join, sub query, window functions, etc. In addition, it provides standard interfaces including ODBC/JDBC, and can directly connect with BI tools such as Tableau, Power BI, Superset, and Redash.</p>
-
+<p>Why not just use Druid then? Meituanâs engineers shared their thoughts:</p>
<ol>
<li>
+ <p>Druidâs native query language is in its own specific JSON format, which is not as easy to pick up as SQL. Although the Druid community added SQL support later on, the support is not complete and does not meet the data analystsâ requirement of complex SQL queries. On the contrary, Kylin natively supports ANSI SQL, uses Apache Calcite for semantic parsing, and supports SQL features such as join, sub query, window functions, etc. In addition, it provides standard interfaces including ODBC/JDBC, and can directly connect with BI tools such as Tableau, Power BI, Superset, and Redash.</p>
+ </li>
+ <li>
<p>Druid can support only single-table query. Multi-table joins are very common in practice, but they cannot be supported by Druid. Kylin, however, supports Star Schema and Snowflake Schema, satisfying multi-table join requirements.</p>
</li>
<li>
@@ -167,9 +167,9 @@ Graphic 8 Cube1 Query Log</p>
<p>Therefore, it appears to be a promising OLAP solution to combine Druidâs excellent columnar storage with Kylinâs usability, compatibility, and completeness. Druid has columnar storage, inverted index, better filtering performance than HBase, native OLAP features, and good secondary aggregation capabilities. Meituanâs tech team decided to try replacing HBase with Druid as the storage for Kylin.</p>
-<h2 id="section">05</h2>
-<p>### Kylin on Druid Design<br />
-At v1.5, Apache Kylin introduced plugable architecture and de-coupled computing and storage components, which makes the replacement of HBase possible. Here is a brief introduction to the main design concept of Kylin on Druid based on Meituan engineer Kaisen Kangâs design doc. (Graphics 9 and 10 are from reference[1], and text are from reference[1] and [3])</p>
+<h2 id="05">05</h2>
+<h3 id="kylin-on-druid-design">Kylin on Druid Design</h3>
+<p>At v1.5, Apache Kylin introduced plugable architecture and de-coupled computing and storage components, which makes the replacement of HBase possible. Here is a brief introduction to the main design concept of Kylin on Druid based on Meituan engineer Kaisen Kangâs design doc. (Graphics 9 and 10 are from reference[1], and text are from reference[1] and [3])</p>
<h3 id="process-of-building-cube">Process of Building Cube</h3>
<ol>
@@ -202,12 +202,12 @@ Graphic 10 Process of Querying Cube</
<li>Kylin measure columns map to Druid measure columns.</li>
</ol>
-<h2 id="summary">06 Summary</h2>
+<h2 id="06-summary">06 Summary</h2>
<p>In this article, we first analyzed features and pros/cons of both Kylin and Druid, and the reasons for poor performance of Hbase in Kylin in some cases. Then we searched solutions and found the feasible option of using Druid as the Kylin storage engine. At last, we illustrated the Kylin-on-Druid architecture and the processes developed by Meituan.</p>
<p>Stay tuned for our next article about how to use Kylin on Druid, how it performs, and how it can be improved.</p>
-<h2 id="reference">07 Reference</h2>
+<h2 id="07-reference">07 Reference</h2>
<ol>
<li>
@@ -225,7 +225,7 @@ Graphic 10 Process of Querying Cube</
</ol>
</description>
- <pubDate>Wed, 12 Dec 2018 09:30:00 -0800</pubDate>
+ <pubDate>Wed, 12 Dec 2018 17:30:00 +0000</pubDate>
<link>http://kylin.apache.org/blog/2018/12/12/why-did-meituan-develop-kylin-on-druid-part1-of-2/</link>
<guid isPermaLink="true">http://kylin.apache.org/blog/2018/12/12/why-did-meituan-develop-kylin-on-druid-part1-of-2/</guid>
@@ -242,37 +242,37 @@ Graphic 10 Process of Querying Cube</
<p>è¿æ¯ç»§2.4.0 åçä¸ä¸ªæ°åè½çæ¬ã该çæ¬å¼å
¥äºå¾å¤æä»·å¼çæ¹è¿ï¼å®æ´çæ¹å¨å表请åè§<a href="https://kylin.apache.org/docs/release_notes.html">release notes</a>ï¼è¿éæä¸äºä¸»è¦æ¹è¿å说æï¼</p>
-<h3 id="all-in-spark--cubing-">All-in-Spark ç Cubing å¼æ</h3>
+<h3 id="all-in-spark-ç-cubing-å¼æ">All-in-Spark ç Cubing å¼æ</h3>
<p>Kylin ç Spark å¼æå°ä½¿ç¨ Spark è¿è¡ cube 计ç®ä¸çææåå¸å¼ä½ä¸ï¼å
æ¬è·åå个维度çä¸åå¼ï¼å° cuboid æ件转æ¢ä¸º HBase HFileï¼å并 segmentï¼å并è¯å
¸çãé»è®¤ç Spark é
ç½®ä¹ç»è¿ä¼åï¼ä½¿å¾ç¨æ·å¯ä»¥è·å¾å¼ç®±å³ç¨çä½éªãç¸å
³å¼åä»»å¡æ¯ KYLIN-3427, KYLIN-3441, KYLIN-3442.</p>
<p>Spark ä»»å¡ç®¡çä¹æææ¹è¿ï¼ä¸æ¦ Spark ä»»å¡å¼å§è¿è¡ï¼æ¨å°±å¯ä»¥å¨Webæ§å¶å°ä¸è·å¾ä½ä¸é¾æ¥ï¼å¦ææ¨ä¸¢å¼è¯¥ä½ä¸ï¼Kylin å°ç«å»ç»æ¢ Spark ä½ä¸ä»¥åæ¶éæ¾èµæºï¼å¦æéæ°å¯å¨ Kylinï¼å®å¯ä»¥ä»ä¸ä¸ä¸ªä½ä¸æ¢å¤ï¼èä¸æ¯éæ°æ交æ°ä½ä¸.</p>
-<h3 id="mysql--kylin-">MySQL å Kylin å
æ°æ®çåå¨</h3>
+<h3 id="mysql-å-kylin-å
æ°æ®çåå¨">MySQL å Kylin å
æ°æ®çåå¨</h3>
<p>å¨è¿å»ï¼HBase æ¯ Kylin å
æ°æ®åå¨çå¯ä¸éæ©ã å¨æäºæ
åµä¸ HBaseä¸éç¨ï¼ä¾å¦ä½¿ç¨å¤ä¸ª HBase é群æ¥ä¸º Kylin æä¾è·¨åºåçé«å¯ç¨ï¼è¿éå¤å¶ç HBase é群æ¯åªè¯»çï¼æ以ä¸è½åå
æ°æ®åå¨ãç°å¨æ们å¼å
¥äº MySQL Metastore 以满足è¿ç§éæ±ãæ¤åè½ç°å¨å¤äºæµè¯é¶æ®µãæ´å¤å
容åè§ KYLIN-3488ã</p>
-<h3 id="hybrid-model-">Hybrid model å¾å½¢çé¢</h3>
+<h3 id="hybrid-model-å¾å½¢çé¢">Hybrid model å¾å½¢çé¢</h3>
<p>Hybrid æ¯ä¸ç§ç¨äºç»è£
å¤ä¸ª cube çé«çº§æ¨¡åã å®å¯ç¨äºæ»¡è¶³ cube ç schema è¦åçæ¹åçæ
åµãè¿ä¸ªåè½è¿å»æ²¡æå¾å½¢çé¢ï¼å æ¤åªæä¸å°é¨åç¨æ·ç¥éå®ãç°å¨æä»¬å¨ Web çé¢ä¸å¼å¯äºå®ï¼ä»¥ä¾¿æ´å¤ç¨æ·å¯ä»¥å°è¯ã</p>
-<h3 id="cube-planner">é»è®¤å¼å¯ Cube planner</h3>
+<h3 id="é»è®¤å¼å¯-cube-planner">é»è®¤å¼å¯ Cube planner</h3>
<p>Cube planner å¯ä»¥æ大å°ä¼å cube ç»æï¼åå°æ建ç cuboid æ°éï¼ä»èèç计ç®/åå¨èµæºå¹¶æé«æ¥è¯¢æ§è½ãå®æ¯å¨v2.3ä¸å¼å
¥çï¼ä½é»è®¤æ
åµä¸æ²¡æå¼å¯ã为äºè®©æ´å¤ç¨æ·çå°å¹¶å°è¯å®ï¼æ们é»è®¤å¨v2.5ä¸å¯ç¨å®ã ç®æ³å°å¨ç¬¬ä¸æ¬¡æ建 segment çæ¶åï¼æ ¹æ®æ°æ®ç»è®¡èªå¨ä¼å cuboid éå.</p>
-<h3 id="segment-">æ¹è¿ç Segment åªæ</h3>
+<h3 id="æ¹è¿ç-segment-åªæ">æ¹è¿ç Segment åªæ</h3>
<p>Segmentï¼ååºï¼ä¿®åªå¯ä»¥ææå°åå°ç£çåç½ç»I / Oï¼å æ¤å¤§å¤§æé«äºæ¥è¯¢æ§è½ã è¿å»ï¼Kylin åªæååºå (partition date column) çå¼è¿è¡ segment çä¿®åªã å¦ææ¥è¯¢ä¸æ²¡æå°ååºåä½ä¸ºè¿æ»¤æ¡ä»¶ï¼é£ä¹ä¿®åªå°ä¸èµ·ä½ç¨ï¼ä¼æ«æææsegmentã.<br />
ç°å¨ä»v2.5å¼å§ï¼Kylin å°å¨ segment 级å«è®°å½æ¯ä¸ªç»´åº¦çæå°/æ大å¼ã å¨æ«æ segment ä¹åï¼ä¼å°æ¥è¯¢çæ¡ä»¶ä¸æå°/æ大索å¼è¿è¡æ¯è¾ã å¦æä¸å¹é
ï¼å°è·³è¿è¯¥ segmentã æ£æ¥KYLIN-3370äºè§£æ´å¤ä¿¡æ¯ã</p>
-<h3 id="yarn-">å¨ YARN ä¸å并åå
¸</h3>
+<h3 id="å¨-yarn-ä¸å并åå
¸">å¨ YARN ä¸å并åå
¸</h3>
<p>å½ segment å并æ¶ï¼å®ä»¬çè¯å
¸ä¹éè¦å并ãå¨è¿å»ï¼åå
¸å并åçå¨ Kylin ç JVM ä¸ï¼è¿éè¦ä½¿ç¨å¤§éçæ¬å°å
åå CPU èµæºã å¨æ端æ
åµä¸ï¼å¦ææå 个并åä½ä¸ï¼ï¼å¯è½ä¼å¯¼è´ Kylin è¿ç¨å´©æºã å æ¤ï¼ä¸äºç¨æ·ä¸å¾ä¸ä¸º Kylin ä»»å¡èç¹åé
æ´å¤å
åï¼æè¿è¡å¤ä¸ªä»»å¡èç¹ä»¥å¹³è¡¡å·¥ä½è´è½½ã<br />
ç°å¨ä»v2.5å¼å§ï¼Kylin å°æè¿é¡¹ä»»å¡æäº¤ç» Hadoop MapReduce å Sparkï¼è¿æ ·å°±å¯ä»¥è§£å³è¿ä¸ªç¶é¢é®é¢ã æ¥çKYLIN-3471äºè§£æ´å¤ä¿¡æ¯.</p>
-<h3 id="cube-">æ¹è¿ä½¿ç¨å
¨å±åå
¸ç cube æ建æ§è½</h3>
+<h3 id="æ¹è¿ä½¿ç¨å
¨å±åå
¸ç-cube-æ建æ§è½">æ¹è¿ä½¿ç¨å
¨å±åå
¸ç cube æ建æ§è½</h3>
<p>å
¨å±åå
¸ (Global Dictionary) æ¯ bitmap 精确å»é计æ°çå¿
è¦æ¡ä»¶ãå¦æå»éåå
·æé常é«çåºæ°ï¼å GD å¯è½é常大ãå¨ cube æ建é¶æ®µï¼Kylin éè¦éè¿ GD å°éæ´æ°å¼è½¬æ¢ä¸ºæ´æ°ã尽管 GD 已被åæå¤ä¸ªåçï¼å¯ä»¥åå¼å è½½å°å
åï¼ä½æ¯ç±äºå»éåçå¼æ¯ä¹±åºçãKylin éè¦åå¤è½½å
¥åè½½åº(swap in/out)åçï¼è¿ä¼å¯¼è´æ建任å¡é常ç¼æ
¢ã<br />
该å¢å¼ºåè½å¼å
¥äºä¸ä¸ªæ°æ¥éª¤ï¼ä¸ºæ¯ä¸ªæ°æ®åä»å
¨å±åå
¸ä¸æ建ä¸ä¸ªç¼©å°çåå
¸ã éåæ¯ä¸ªä»»å¡åªéè¦å 载缩å°çåå
¸ï¼ä»èé¿å
é¢ç¹çè½½å
¥åè½½åºãæ§è½å¯ä»¥æ¯ä»¥åå¿«3åãæ¥ç KYLIN-3491 äºè§£æ´å¤ä¿¡æ¯.</p>
-<h3 id="topn-count-distinct--cube-">æ¹è¿å« TOPN, COUNT DISTINCT ç cube 大å°ç估计</h3>
+<h3 id="æ¹è¿å«-topn-count-distinct-ç-cube-大å°ç估计">æ¹è¿å« TOPN, COUNT DISTINCT ç cube 大å°ç估计</h3>
<p>Cube ç大å°å¨æ建æ¶æ¯é¢å
估计çï¼å¹¶è¢«åç»å 个æ¥éª¤ä½¿ç¨ï¼ä¾å¦å³å® MR / Spark ä½ä¸çååºæ°ï¼è®¡ç® HBase region åå²çãå®çåç¡®ä¸å¦ä¼å¯¹æ建æ§è½äº§çå¾å¤§å½±åã å½åå¨ COUNT DISTINCTï¼TOPN ç度éæ¶åï¼å 为å®ä»¬ç大å°æ¯çµæ´»çï¼å æ¤ä¼°è®¡å¼å¯è½è·çå®å¼æå¾å¤§åå·®ã å¨è¿å»ï¼ç¨æ·éè¦è°æ´è¥å¹²ä¸ªåæ°ä»¥ä½¿å°ºå¯¸ä¼°è®¡æ´æ¥è¿å®é
尺寸ï¼è¿å¯¹æ®éç¨æ·æç¹å°é¾ã<br />
ç°å¨ï¼Kylin å°æ ¹æ®æ¶éçç»è®¡ä¿¡æ¯èªå¨è°æ´å¤§å°ä¼°è®¡ãè¿å¯ä»¥ä½¿ä¼°è®¡å¼ä¸å®é
大å°æ´æ¥è¿ãæ¥ç KYLIN-3453 äºè§£æ´å¤ä¿¡æ¯ã</p>
-<h3 id="hadoop-30hbase-20">æ¯æHadoop 3.0/HBase 2.0</h3>
+<h3 id="æ¯æhadoop-30hbase-20">æ¯æHadoop 3.0/HBase 2.0</h3>
<p>Hadoop 3å HBase 2å¼å§è¢«è®¸å¤ç¨æ·éç¨ãç°å¨ Kylin æä¾ä½¿ç¨æ°ç Hadoop å HBase API ç¼è¯çæ°äºè¿å¶å
ãæ们已ç»å¨ Hortonworks HDP 3.0 å Cloudera CDH 6.0 ä¸è¿è¡äºæµè¯</p>
<p><strong>ä¸è½½</strong></p>
@@ -289,7 +289,7 @@ Graphic 10 Process of Querying Cube</
<p><em>é常æè°¢ææè´¡ç®Apache Kylinçæå!</em></p>
</description>
- <pubDate>Thu, 20 Sep 2018 13:00:00 -0700</pubDate>
+ <pubDate>Thu, 20 Sep 2018 20:00:00 +0000</pubDate>
<link>http://kylin.apache.org/cn/blog/2018/09/20/release-v2.5.0/</link>
<guid isPermaLink="true">http://kylin.apache.org/cn/blog/2018/09/20/release-v2.5.0/</guid>
@@ -361,7 +361,7 @@ Graphic 10 Process of Querying Cube</
<p><em>Great thanks to everyone who contributed!</em></p>
</description>
- <pubDate>Thu, 20 Sep 2018 13:00:00 -0700</pubDate>
+ <pubDate>Thu, 20 Sep 2018 20:00:00 +0000</pubDate>
<link>http://kylin.apache.org/blog/2018/09/20/release-v2.5.0/</link>
<guid isPermaLink="true">http://kylin.apache.org/blog/2018/09/20/release-v2.5.0/</guid>
@@ -407,22 +407,20 @@ GRANT ROLE ssb_write_role TO GROUP ssb_w
# Then add kylin_manager_user to kylin_manager_group in OpenLDAP, so kylin_manager_user has access to the ssb database.
</pre>
<p>2 Assign HDFS directory /user/kylin_manager_user read and write permissions to kylin_manager_user user.<br />
-3 Configure the HADOOP_STREAMING_JAR environment variable under the kylin_manager_user user home directory.<br />
-<code class="highlighter-rouge">
-Export HADOOP_STREAMING_JAR=/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-streaming.jar
-</code></p>
+3 Configure the HADOOP_STREAMING_JAR environment variable under the kylin_manager_user user home directory.</p>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Export HADOOP_STREAMING_JAR=/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-streaming.jar
+</code></pre></div></div>
<h2 id="download-the-ssb-tool-and-compile">Download the SSB tool and compile</h2>
<p>You can quickly download and compile the ssb test tool by entering the following command in the linux terminal.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>git clone https://github.com/jiangshouzhuang/ssb-kylin.git
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/jiangshouzhuang/ssb-kylin.git
cd ssb-kylin
cd ssb-benchmark
make clean
make
-</code></pre>
-</div>
+</code></pre></div></div>
<h2 id="adjust-the-ssb-parameters">Adjust the SSB parameters</h2>
@@ -430,7 +428,7 @@ make
<p>Part of the ssb.conf file is:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code> # customer base, default value is 30,000
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> # customer base, default value is 30,000
customer_base = 30000
# part base, default value is 200,000
part_base = 200000
@@ -440,30 +438,27 @@ make
date_base = 2556
# lineorder base (purchase record), default value is 6,000,000
lineorder_base = 6000000
-</code></pre>
-</div>
+</code></pre></div></div>
<p>Of course, the above base parameters can be adjusted according to their actual needs, I use the default parameters.<br />
In the ssb.conf file, there are some parameters as follows.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code># manufacturer max. The value range is (1 .. manu_max)
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># manufacturer max. The value range is (1 .. manu_max)
manu_max = 5
# category max. The value range is (1 .. cat_max)
cat_max = 5
# brand max. The value range is (1 .. brand_max)
brand_max = 40
-</code></pre>
-</div>
+</code></pre></div></div>
<p><strong>The explanation is as follows:</strong> <br />
manu_max, cat_max and brand_max are used to define hierarchical scale. For example, manu_max=10, cat_max=10, and brand_max=10 refer to a total of 10 manufactures, and each manufactures has a maximum of 10 category parts, and each category has up to 10 brands. Therefore, the cardinality of manufacture is 10, the cardinality of category is 100, and the cardinality of brand is 1000.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code># customer: num of cities per country, default value is 100
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># customer: num of cities per country, default value is 100
cust_city_max = 9
# supplier: num of cities per country, default value is 100
supp_city_max = 9
-</code></pre>
-</div>
+</code></pre></div></div>
<p><strong>The explanation is as follows:</strong> <br />
cust_city_max and supp_city_max are used to define the number of city for each country in customer and supplier tables. If the total number of country is 30, and cust_city_max=100, supp_city_max=10, then the customer table will have 3000 different city, and the supplier table will have 300 different city.</p>
@@ -494,19 +489,17 @@ ${KYLIN_INSTALL_USER_PASSWD} -d org.apac
<p>If your CDH or other big data platform is not using beeline, but hive cli, please modify it yourself.<br />
Once everything is ready, we start running the program and generate test data:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>cd ssb-kylin
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd ssb-kylin
bin/run.sh --scale 20
-</code></pre>
-</div>
+</code></pre></div></div>
<p>We set the scale to 20, the program will run for a while, the maximum lineorder table data has more than 100 million. After the program is executed, we look at the tables in the hive database and the amount of data:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>use ssb;
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>use ssb;
show tables;
select count(1) from lineorder;
select count(1) from p_lineorder;
-</code></pre>
-</div>
+</code></pre></div></div>
<p><img src="/images/blog/2.1 generated tables.png" alt="" /></p>
@@ -519,10 +512,9 @@ select count(1) from p_lineorder;
<p>The ssb-kylin project has helped us build the project, model, and cube in advance. Just import the Kylin directly like the learn_kylin example. Cube Metadataâs directory is cubemeta, because our kylin integrates OpenLDAP, there is no ADMIN user, so the owner parameter in cubemeta/cube/ssb.json is set to null.<br />
Execute the following command to import cubemeta:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>cd ssb-kylin
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd ssb-kylin
$KYLIN_HOME/bin/metastore.sh restore cubemeta
-</code></pre>
-</div>
+</code></pre></div></div>
<p>Then log in to Kylin and execute Reload Metadata operation. This creates new project, model and cube in Kylin. Before building cube, first Disable, then Purge, delete old temporary files.</p>
@@ -532,19 +524,17 @@ $KYLIN_HOME/bin/metastore.sh restore cub
<p>Here I test the performance of Spark to build Cube again, disable the previously created Cube, and then Purge. Since the Cube is used by Purge, the useless HBase tables and HDFS files need to be deleted. Here, manually clean up the junk files. First execute the following command:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete false
-</code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete false
+</code></pre></div></div>
<p>Then check whether the listed HBase table and the HDFS file are useless. After confirming the error, perform the delete operation:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete true
-</code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete true
+</code></pre></div></div>
<p>When using Spark to build a cube, it consumes a lot of memory. After all, using memory resources improves the speed of cube building. Here I will list some of the parameters of Spark in the kylin.properties configuration file:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>kylin.engine.spark-conf.spark.master=yarn
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kylin.engine.spark-conf.spark.master=yarn
kylin.engine.spark-conf.spark.submit.deployMode=cluster
kylin.engine.spark-conf.spark.yarn.queue=root.kylin_manager_group
# config Dynamic resource allocation
@@ -560,8 +550,7 @@ kylin.engine.spark-conf.spark.driver.mem
kylin.engine.spark-conf.spark.executor.memory=4G
kylin.engine.spark-conf.spark.executor.cores=1
kylin.engine.spark-conf.spark.network.timeout=600
-</code></pre>
-</div>
+</code></pre></div></div>
<p>The above parameters can meet most of the requirements, so users basically do not need to configure when designing the Cube. Of course, if the situation is special, you can still set Spark-related tuning parameters at the Cube level.</p>
@@ -599,7 +588,7 @@ The query result of Scale=10 is as follo
</ol>
</description>
- <pubDate>Mon, 16 Jul 2018 05:28:00 -0700</pubDate>
+ <pubDate>Mon, 16 Jul 2018 12:28:00 +0000</pubDate>
<link>http://kylin.apache.org/blog/2018/07/16/Star-Schema-Benchmark-on-Apache-Kylin/</link>
<guid isPermaLink="true">http://kylin.apache.org/blog/2018/07/16/Star-Schema-Benchmark-on-Apache-Kylin/</guid>
@@ -649,7 +638,7 @@ The query result of Scale=10 is as follo
<p>Wish you have a good time with Redash-Kylin!</p>
</description>
- <pubDate>Tue, 08 May 2018 13:00:00 -0700</pubDate>
+ <pubDate>Tue, 08 May 2018 20:00:00 +0000</pubDate>
<link>http://kylin.apache.org/blog/2018/05/08/redash-kylin-plugin-strikingly/</link>
<guid isPermaLink="true">http://kylin.apache.org/blog/2018/05/08/redash-kylin-plugin-strikingly/</guid>
@@ -691,11 +680,11 @@ The query result of Scale=10 is as follo
<p>Any issue or question,<br />
open JIRA to Apache Kylin project: <a href="https://issues.apache.org/jira/browse/KYLIN/">https://issues.apache.org/jira/browse/KYLIN/</a><br />
or<br />
-send mail to Apache Kylin dev mailing list: <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a></p>
+send mail to Apache Kylin dev mailing list: <a href="mailto:dev@kylin.apache.org">dev@kylin.apache.org</a></p>
<p><em>Great thanks to everyone who contributed!</em></p>
</description>
- <pubDate>Sun, 04 Mar 2018 12:00:00 -0800</pubDate>
+ <pubDate>Sun, 04 Mar 2018 20:00:00 +0000</pubDate>
<link>http://kylin.apache.org/blog/2018/03/04/release-v2.3.0/</link>
<guid isPermaLink="true">http://kylin.apache.org/blog/2018/03/04/release-v2.3.0/</guid>
@@ -766,15 +755,14 @@ Figure 4: Build Cube in Apache Kylin<
<li>Execute SQL in the âInsightâ tab, for example:</li>
</ol>
-<div class="highlighter-rouge"><pre class="highlight"><code> select part_dtï¼
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> select part_dtï¼
sum(price) as total_selledï¼
count(distinct seller_id) as sellers
from kylin_sales
group by part_dt
order by part_dt
-- #This query will hit on the newly built Cube âKylin_sales_cubeâ.
-</code></pre>
-</div>
+</code></pre></div></div>
<ol>
<li>Next, we will install Apache Superset and initialize it.<br />
@@ -782,15 +770,14 @@ Figure 4: Build Cube in Apache Kylin<
<li>Install kylinpy</li>
</ol>
-<div class="highlighter-rouge"><pre class="highlight"><code> $ pip install kylinpy
-</code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ pip install kylinpy
+</code></pre></div></div>
<ol>
<li>Verify your installation, if everything goes well, Apache Superset daemon should be up and running.</li>
</ol>
-<div class="highlighter-rouge"><pre class="highlight"><code>$ superset runserver -d
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ superset runserver -d
Starting server with command:
gunicorn -w 2 --timeout 60 -b 0.0.0.0:8088 --limit-request-line 0 --limit-request-field_size 0 superset:app
@@ -799,18 +786,19 @@ gunicorn -w 2 --timeout 60 -b 0.0.0.0:8
[2018-01-03 15:54:03 +0800] [73673] [INFO] Using worker: sync
[2018-01-03 15:54:03 +0800] [73676] [INFO] Booting worker with pid: 73676
[2018-01-03 15:54:03 +0800] [73679] [INFO] Booting worker with pid: 73679
-</code></pre>
-</div>
+</code></pre></div></div>
<h2 id="connect-apache-kylin-from-apachesuperset">Connect Apache Kylin from ApacheSuperset</h2>
-<p>Now everything you need is installed and ready to go. Letâs try to create an Apache Kylin data source in Apache Superset.<br />
-1. Open up http://localhost:8088 in your web browser with the credential you set during Apache Superset installation.<br />
+<p>Now everything you need is installed and ready to go. Letâs try to create an Apache Kylin data source in Apache Superset.</p>
+<ol>
+ <li>
+ <p>Open up http://localhost:8088 in your web browser with the credential you set during Apache Superset installation.<br />
<img src="/images/Kylin-and-Superset/png/5. superset_1.png" alt="" /><br />
Figure 5: Apache Superset Login Page</p>
-
-<ol>
- <li>Go to Source -&gt; Datasource to configure a new data source.
+ </li>
+ <li>
+ <p>Go to Source -&gt; Datasource to configure a new data source.</p>
<ul>
<li>SQLAlchemy URI pattern is : kylin://<username>:<password>@<hostname>:<port>/<project name=""></project></port></hostname></password></username></li>
<li>Check âExpose in SQL Labâ if you want to expose this data source in SQL Lab.</li>
@@ -856,9 +844,8 @@ Figure 11 Query multiple tables from Apa
<img src="/images/Kylin-and-Superset/png/12. SQL_Lab_2.png" alt="" /><br />
Figure 12 Define your query and visualize it immediately</p>
-<p>You may copy the entire SQL below to experience how you can query Kylin Cube in SQL Lab. <br />
-<code class="highlighter-rouge">
-select
+<p>You may copy the entire SQL below to experience how you can query Kylin Cube in SQL Lab.</p>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>select
YEAR_BEG_DT,
MONTH_BEG_DTï¼
WEEK_BEG_DTï¼
@@ -876,8 +863,8 @@ join KYLIN_CATEGORY_GROUPINGS on SITE_ID
join KYLIN_ACCOUNT on ACCOUNT_ID=BUYER_ID
join KYLIN_COUNTRY on ACCOUNT_COUNTRY=COUNTRY
group by YEAR_BEG_DT, MONTH_BEG_DTï¼WEEK_BEG_DTï¼META_CATEG_NAMEï¼CATEG_LVL2_NAME, CATEG_LVL3_NAME, OPS_REGION, NAME
-</code><br />
-## Experience All Features in Apache Superset with Apache Kylin</p>
+</code></pre></div></div>
+<h2 id="experience-all-features-in-apache-superset-with-apache-kylin">Experience All Features in Apache Superset with Apache Kylin</h2>
<p>Most of the common reporting features are available in Apache Superset. Now letâs see how we can use those features to analyze data from Apache Kylin.</p>
@@ -890,13 +877,14 @@ group by YEAR_BEG_DT, MONTH_BEG_DTï¼
Figure 13 Sort by</p>
<h3 id="filtering">Filtering</h3>
-<p>There are multiple ways you may filter data from Apache Kylin.<br />
-1. Date Filter<br />
+<p>There are multiple ways you may filter data from Apache Kylin.</p>
+<ol>
+ <li>
+ <p>Date Filter<br />
You may filter date and time dimension with the calendar filter. <br />
<img src="/images/Kylin-and-Superset/png/14. time_filter.png" alt="" /><br />
Figure 14 Filtering time</p>
-
-<ol>
+ </li>
<li>
<p>Dimension Filter<br />
For other dimensions, you may filter it with SQL conditions like âin, not in, equal to, not equal to, greater than and equal to, smaller than and equal to, greater than, smaller than, likeâ.<br />
@@ -964,7 +952,7 @@ Figure 13 Sort by</p>
</ol>
</description>
- <pubDate>Mon, 01 Jan 2018 04:28:00 -0800</pubDate>
+ <pubDate>Mon, 01 Jan 2018 12:28:00 +0000</pubDate>
<link>http://kylin.apache.org/blog/2018/01/01/kylin-and-superset/</link>
<guid isPermaLink="true">http://kylin.apache.org/blog/2018/01/01/kylin-and-superset/</guid>
@@ -993,12 +981,11 @@ Figure 13 Sort by</p>
<h3 id="make-spark-connect-hbase-with-kerberos-enabled">Make Spark connect HBase with Kerberos enabled</h3>
<p>If just want to run Spark Cubing in Yarn client mode, we only need to add three line code before new SparkConf() in SparkCubingByLayer:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code> Configuration configuration = HBaseConnection.getCurrentHBaseConfiguration();
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> Configuration configuration = HBaseConnection.getCurrentHBaseConfiguration();
HConnection connection = HConnectionManager.createConnection(configuration);
//Obtain an authentication token for the given user and add it to the user's credentials.
TokenUtil.obtainAndCacheToken(connection, UserProvider.instantiate(configuration).create(UserGroupInformation.getCurrentUser()));
-</code></pre>
-</div>
+</code></pre></div></div>
<p>As for How to make Spark connect HBase using Kerberos in Yarn cluster mode, please refer to SPARK-6918, SPARK-12279, and HBASE-17040. The solution may work, but not elegant. So I tried the sencond solution.</p>
@@ -1039,7 +1026,7 @@ This following picture shows the content
<p>Following is the Spark configuration I used in our environment. It enables Spark dynamic resource allocation; the goal is to let our user set less Spark configurations.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>//running in yarn-cluster mode
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>//running in yarn-cluster mode
kylin.engine.spark-conf.spark.master=yarn
kylin.engine.spark-conf.spark.submit.deployMode=cluster
@@ -1064,8 +1051,7 @@ kylin.engine.spark-conf.spark.network.ti
kylin.engine.spark-conf.spark.yarn.queue=root.hadoop.test
kylin.engine.spark.rdd-partition-cut-mb=100
-</code></pre>
-</div>
+</code></pre></div></div>
<h3 id="performance-test-of-spark-cubing">Performance test of Spark Cubing</h3>
@@ -1121,7 +1107,7 @@ kylin.engine.spark.rdd-partition-cut-mb=
<p>Spark Cubing is a great feature for Kylin 2.0, Thanks Kylin community. We will apply Spark Cubing in real scenarios in our company. I believe Spark Cubing will be more robust and efficient in the future releases.</p>
</description>
- <pubDate>Fri, 21 Jul 2017 15:22:22 -0700</pubDate>
+ <pubDate>Fri, 21 Jul 2017 22:22:22 +0000</pubDate>
<link>http://kylin.apache.org/blog/2017/07/21/Improving-Spark-Cubing/</link>
<guid isPermaLink="true">http://kylin.apache.org/blog/2017/07/21/Improving-Spark-Cubing/</guid>
@@ -1141,11 +1127,10 @@ kylin.engine.spark.rdd-partition-cut-mb=
<p>In Apache Kylin, we support the similar SQL sytanx like Apache Hive, with a aggregation function called <strong>percentile(&lt;Number Column&gt;, &lt;Double&gt;)</strong>:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">seller_id</span><span class="p">,</span> <span class="n">percentile</span><span class="p">(</span><span class="n">price</span><span class="p">,</span> <span class="mi">0</span><span class="p">.</span><span class="mi">5</span><span class="p">)</span>
+<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">seller_id</span><span class="p">,</span> <span class="n">percentile</span><span class="p">(</span><span class="n">price</span><span class="p">,</span> <span class="mi">0</span><span class="p">.</span><span class="mi">5</span><span class="p">)</span>
<span class="k">FROM</span> <span class="n">test_kylin_fact</span>
<span class="k">GROUP</span> <span class="k">BY</span> <span class="n">seller_id</span>
-</code></pre>
-</div>
+</code></pre></div></div>
<h3 id="how-to-use">How to use</h3>
<p>If you know little about <em>Cubes</em>, please go to <a href="http://kylin.apache.org/docs20/tutorial/kylin_sample.html">QuickStart</a> first to learn basic knowledge.</p>
@@ -1162,7 +1147,7 @@ kylin.engine.spark.rdd-partition-cut-mb=
<p><img src="/images/blog/percentile_3.png" alt="" /></p>
</description>
- <pubDate>Sat, 01 Apr 2017 15:22:22 -0700</pubDate>
+ <pubDate>Sat, 01 Apr 2017 22:22:22 +0000</pubDate>
<link>http://kylin.apache.org/blog/2017/04/01/percentile-measure/</link>
<guid isPermaLink="true">http://kylin.apache.org/blog/2017/04/01/percentile-measure/</guid>
@@ -1191,23 +1176,23 @@ kylin.engine.spark.rdd-partition-cut-mb=
<li>å¨ Cloud ä¸éè¿æµè¯ (<a href="https://issues.apache.org/jira/browse/KYLIN-2351">KYLIN-2351</a>)</li>
</ul>
-<p>é常欢è¿å¤§å®¶ä¸è½½å¹¶æµè¯ v2.0.0 betaãæ¨çåé¦å¯¹æ们é常éè¦ï¼è¯·åé®ä»¶å° <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a>ã</p>
+<p>é常欢è¿å¤§å®¶ä¸è½½å¹¶æµè¯ v2.0.0 betaãæ¨çåé¦å¯¹æ们é常éè¦ï¼è¯·åé®ä»¶å° <a href="mailto:dev@kylin.apache.org">dev@kylin.apache.org</a>ã</p>
<hr />
-<h2 id="section">å®è£
</h2>
+<h2 id="å®è£
">å®è£
</h2>
<p>ææ¶ v2.0.0 beta æ æ³ä» v1.6.0 ç´æ¥å级ï¼å¿
éå
¨æ°å®è£
ãè¿æ¯ç±äºæ°çæ¬çå
æ°æ®å¹¶ä¸ååå
¼å®¹ãå¥½å¨ Cube æ°æ®æ¯ååå
¼å®¹çï¼å æ¤åªéè¦å¼åä¸ä¸ªå
æ°æ®è½¬æ¢å·¥å
·ï¼å°±è½å¨ä¸ä¹
çå°æ¥å®ç°å¹³æ»å级ãæ们æ£å¨ä¸ºæ¤åªåã</p>
<hr />
-<h2 id="tpc-h-">è¿è¡ TPC-H åºåæµè¯</h2>
+<h2 id="è¿è¡-tpc-h-åºåæµè¯">è¿è¡ TPC-H åºåæµè¯</h2>
<p>å¨ Apache Kylin ä¸è¿è¡ TPC-H çå
·ä½æ¥éª¤: <a href="https://github.com/Kyligence/kylin-tpch">https://github.com/Kyligence/kylin-tpch</a></p>
<hr />
-<h2 id="spark-">Spark æ建å¼æ</h2>
+<h2 id="spark-æ建å¼æ">Spark æ建å¼æ</h2>
<p>Apache Kylin v2.0.0 å¼å
¥äºä¸ä¸ªå
¨æ°çåºäº Apache Spark çæ建å¼æãå®å¯ç¨äºæ¿æ¢åæç MapReduce æ建å¼æãåæ¥æµè¯æ¾ç¤º Cube çæ建æ¶é´ä¸è¬è½ç¼©çå°åå
ç 50% å·¦å³ã</p>
@@ -1217,7 +1202,7 @@ kylin.engine.spark.rdd-partition-cut-mb=
<p><em>æè°¢æ¯ä¸ä½æåçåä¸åè´¡ç®!</em></p>
</description>
- <pubDate>Sat, 25 Feb 2017 12:00:00 -0800</pubDate>
+ <pubDate>Sat, 25 Feb 2017 20:00:00 +0000</pubDate>
<link>http://kylin.apache.org/cn/blog/2017/02/25/v2.0.0-beta-ready/</link>
<guid isPermaLink="true">http://kylin.apache.org/cn/blog/2017/02/25/v2.0.0-beta-ready/</guid>