You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by sh...@apache.org on 2016/06/08 02:03:30 UTC
svn commit: r1747333 - in /kylin/site: cn/download/index.html
docs15/howto/howto_use_restapi.html docs15/release_notes.html
download/index.html feed.xml
Author: shaofengshi
Date: Wed Jun 8 02:03:30 2016
New Revision: 1747333
URL: http://svn.apache.org/viewvc?rev=1747333&view=rev
Log:
UPDATE download page for v1.5.2.1
Modified:
kylin/site/cn/download/index.html
kylin/site/docs15/howto/howto_use_restapi.html
kylin/site/docs15/release_notes.html
kylin/site/download/index.html
kylin/site/feed.xml
Modified: kylin/site/cn/download/index.html
URL: http://svn.apache.org/viewvc/kylin/site/cn/download/index.html?rev=1747333&r1=1747332&r2=1747333&view=diff
==============================================================================
--- kylin/site/cn/download/index.html (original)
+++ kylin/site/cn/download/index.html Wed Jun 8 02:03:30 2016
@@ -162,38 +162,38 @@
<h4 class="section-title"><span>Apache Kylin™ 相关下载</span></h4>
</div>
</header>
- <p><strong>1.5.2 最新发布(源代码)</strong></p>
+ <p><strong>最新发布(源代码)</strong></p>
<p>最新发布的Apache Kylin™可以从ASF网站下载：:</p>
<ul>
- <li><a href="http://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-1.5.2/">Apache Kylin v1.5.2</a></li>
+ <li><a href="http://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-1.5.2.1/">Apache Kylin v1.5.2.1</a></li>
<li><a href="http://kylin.apache.org/docs15/release_notes.html">发布日志</a></li>
- <li>Git 标签: <a href="https://github.com/apache/kylin/tree/kylin-1.5.2">kylin-1.5.2</a></li>
- <li>Git Commit: <a href="https://github.com/apache/kylin/commit/af2646b72fbb6dc81699ad6661303fd612a2eebf">af2646b72fbb6dc81699ad6661303fd612a2eebf</a></li>
+ <li>Git 标签: <a href="https://github.com/apache/kylin/tree/kylin-1.5.2.1">kylin-1.5.2.1</a></li>
+ <li>Git Commit: <a href="https://github.com/apache/kylin/commit/71a847bbe97ec4e6f23b1bf271132d6ff9955a96">71a847bbe97ec4e6f23b1bf271132d6ff9955a96</a></li>
</ul>
-<p><strong>1.5.2 二进制包 for HBase 0.98/0.99</strong></p>
+<p><strong>1.5.2.1 二进制包 for HBase 0.98/0.99</strong></p>
<p>为方便使用，我们提供预打包的二进制安装包：</p>
<ul>
- <li><a href="https://dist.apache.org/repos/dist/release/kylin/apache-kylin-1.5.1/apache-kylin-1.5.2-bin.tar.gz">apache-kylin-1.5.2-bin.tar.gz</a></li>
+ <li><a href="https://dist.apache.org/repos/dist/release/kylin/apache-kylin-1.5.1/apache-kylin-1.5.2.1-bin.tar.gz">apache-kylin-1.5.2.1-bin.tar.gz</a></li>
<li><a href="http://kylin.apache.org/docs15/install">安装帮助</a></li>
</ul>
-<p><strong>1.5.2 二进制包 for HBase 1.x</strong></p>
+<p><strong>1.5.2.1 二进制包 for HBase 1.x</strong></p>
<ul>
- <li><a href="https://dist.apache.org/repos/dist/release/kylin/apache-kylin-1.5.1/apache-kylin-1.5.2-HBase1.x-bin.tar.gz">apache-kylin-1.5.2-HBase1.x-bin.tar.gz</a></li>
- <li>Git commit <a href="https://github.com/apache/kylin/commit/e85b49823e3cfea027cca497e898064fbefbd449">e85b49823e3cfea027cca497e898064fbefbd449</a></li>
+ <li><a href="https://dist.apache.org/repos/dist/release/kylin/apache-kylin-1.5.1/apache-kylin-1.5.2.1-HBase1.x-bin.tar.gz">apache-kylin-1.5.2.1-HBase1.x-bin.tar.gz</a></li>
+ <li>Git commit <a href="https://github.com/apache/kylin/commit/cf4d2940b67d622eacd2ac9a913b221091a35c2e">cf4d2940b67d622eacd2ac9a913b221091a35c2e</a></li>
</ul>
-<p><strong>1.5.2 二进制包 for CDH 5.7</strong></p>
+<p><strong>1.5.2.1 二进制包 for CDH 5.7</strong></p>
<ul>
- <li><a href="https://dist.apache.org/repos/dist/release/kylin/apache-kylin-1.5.1/apache-kylin-1.5.2-cdh5.7-bin.tar.gz">apache-kylin-1.5.2-cdh5.7-bin.tar.gz</a></li>
- <li>Git commit <a href="https://github.com/apache/kylin/commit/e0fbb3417063fe33b99e82724e19e107787b2e62">e0fbb3417063fe33b99e82724e19e107787b2e62</a></li>
+ <li><a href="https://dist.apache.org/repos/dist/release/kylin/apache-kylin-1.5.1/apache-kylin-1.5.2.1-cdh5.7-bin.tar.gz">apache-kylin-1.5.2.1-cdh5.7-bin.tar.gz</a></li>
+ <li>Git commit <a href="https://github.com/apache/kylin/commit/e760242daad739b66813dc2c2d8a52bf74c0ec22">e760242daad739b66813dc2c2d8a52bf74c0ec22</a></li>
</ul>
<p><strong>构建二进制包</strong><br />
Modified: kylin/site/docs15/howto/howto_use_restapi.html
URL: http://svn.apache.org/viewvc/kylin/site/docs15/howto/howto_use_restapi.html?rev=1747333&r1=1747332&r2=1747333&view=diff
==============================================================================
--- kylin/site/docs15/howto/howto_use_restapi.html (original)
+++ kylin/site/docs15/howto/howto_use_restapi.html Wed Jun 8 02:03:30 2016
@@ -2116,7 +2116,7 @@ Authorization:Basic {data}</p>
</code></pre>
</div>
-<p>Example with <code class="highlighter-rouge">curl</code>:</p>
+<h4 id="curl-example">Curl Example</h4>
<div class="highlighter-rouge"><pre class="highlight"><code>curl -c /path/to/cookiefile.txt -X POST -H "Authorization: Basic XXXXXXXXX" -H 'Content-Type: application/json' http://<host>:<port>/kylin/api/user/authentication
</code></pre>
@@ -2249,6 +2249,12 @@ Authorization:Basic {data}</p>
</code></pre>
</div>
+<h4 id="curl-example-1">Curl Example</h4>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>curl -X POST -H "Authorization: Basic XXXXXXXXX" -H "Content-Type: application/json" -d '{ "sql":"select count(*) from TEST_KYLIN_FACT", "project":"learn_kylin" }' http://YOUR_HOST:7070/kylin/api/query
+</code></pre>
+</div>
+
<h2 id="list-queryable-tables">List queryable tables</h2>
<p><code class="highlighter-rouge">GET /tables_and_columns</code></p>
Modified: kylin/site/docs15/release_notes.html
URL: http://svn.apache.org/viewvc/kylin/site/docs15/release_notes.html?rev=1747333&r1=1747332&r2=1747333&view=diff
==============================================================================
--- kylin/site/docs15/release_notes.html (original)
+++ kylin/site/docs15/release_notes.html Wed Jun 8 02:03:30 2016
@@ -2048,6 +2048,18 @@ there are source code package, binary pa
<li>Development relative: <a href="mailto:dev@kylin.apache.org">dev@kylin.apache.org</a></li>
</ul>
+<h2 id="v1521---2016-06-07">v1.5.2.1 - 2016-06-07</h2>
+<p><em>Tag:</em> <a href="https://github.com/apache/kylin/tree/kylin-1.5.2.1">kylin-1.5.2.1</a></p>
+
+<p>This is a hot-fix version on v1.5.2, no new feature introduced, please upgrade to this version;</p>
+
+<p><strong>Bug</strong></p>
+
+<ul>
+ <li>[KYLIN-1758] - createLookupHiveViewMaterializationStep will create intermediate table for fact table</li>
+ <li>[KYLIN-1739] - kylin_job_conf_inmem.xml can impact non-inmem MR job</li>
+</ul>
+
<h2 id="v152---2016-05-26">v1.5.2 - 2016-05-26</h2>
<p><em>Tag:</em> <a href="https://github.com/apache/kylin/tree/kylin-1.5.2">kylin-1.5.2</a></p>
Modified: kylin/site/download/index.html
URL: http://svn.apache.org/viewvc/kylin/site/download/index.html?rev=1747333&r1=1747332&r2=1747333&view=diff
==============================================================================
--- kylin/site/download/index.html (original)
+++ kylin/site/download/index.html Wed Jun 8 02:03:30 2016
@@ -169,10 +169,10 @@
The latest release of Apache Kylin™ can be downloaded from the ASF:</p>
<ul>
- <li><a href="http://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-1.5.2/">Apache Kylin v1.5.2</a></li>
+ <li><a href="http://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-1.5.2.1/">Apache Kylin v1.5.2.1</a></li>
<li><a href="http://kylin.apache.org/docs15/release_notes.html">Release Notes</a></li>
- <li>Git Tag: <a href="https://github.com/apache/kylin/tree/kylin-1.5.2">kylin-1.5.2</a></li>
- <li>Git Commit: <a href="https://github.com/apache/kylin/commit/af2646b72fbb6dc81699ad6661303fd612a2eebf">af2646b72fbb6dc81699ad6661303fd612a2eebf</a></li>
+ <li>Git Tag: <a href="https://github.com/apache/kylin/tree/kylin-1.5.2.1">kylin-1.5.2.1</a></li>
+ <li>Git Commit: <a href="https://github.com/apache/kylin/commit/71a847bbe97ec4e6f23b1bf271132d6ff9955a96">71a847bbe97ec4e6f23b1bf271132d6ff9955a96</a></li>
</ul>
<p><strong>Binary Package (for running on HBase 0.98/0.99)</strong></p>
@@ -180,22 +180,22 @@ The latest release of Apache Kylin™
<p>For convenience, there’s binary package also available:</p>
<ul>
- <li><a href="https://dist.apache.org/repos/dist/release/kylin/apache-kylin-1.5.2/apache-kylin-1.5.2-bin.tar.gz">apache-kylin-1.5.2-bin.tar.gz</a></li>
+ <li><a href="https://dist.apache.org/repos/dist/release/kylin/apache-kylin-1.5.2.1/apache-kylin-1.5.2.1-bin.tar.gz">apache-kylin-1.5.2.1-bin.tar.gz</a></li>
<li><a href="http://kylin.apache.org/docs15/install">Installation Guide</a></li>
</ul>
<p><strong>Binary Package (for running on HBase 1.x)</strong></p>
<ul>
- <li><a href="https://dist.apache.org/repos/dist/release/kylin/apache-kylin-1.5.2/apache-kylin-1.5.2-HBase1.x-bin.tar.gz">apache-kylin-1.5.2-HBase1.x-bin.tar.gz</a></li>
- <li>Git commit <a href="https://github.com/apache/kylin/commit/e85b49823e3cfea027cca497e898064fbefbd449">e85b49823e3cfea027cca497e898064fbefbd449</a></li>
+ <li><a href="https://dist.apache.org/repos/dist/release/kylin/apache-kylin-1.5.2.1/apache-kylin-1.5.2.1-HBase1.x-bin.tar.gz">apache-kylin-1.5.2.1-HBase1.x-bin.tar.gz</a></li>
+ <li>Git commit <a href="https://github.com/apache/kylin/commit/cf4d2940b67d622eacd2ac9a913b221091a35c2e">cf4d2940b67d622eacd2ac9a913b221091a35c2e</a></li>
</ul>
<p><strong>Binary Package (for running on CDH 5.7)</strong></p>
<ul>
- <li><a href="https://dist.apache.org/repos/dist/release/kylin/apache-kylin-1.5.2/apache-kylin-1.5.2-cdh5.7-bin.tar.gz">apache-kylin-1.5.2-cdh5.7-bin.tar.gz</a></li>
- <li>Git commit <a href="https://github.com/apache/kylin/commit/e0fbb3417063fe33b99e82724e19e107787b2e62">e0fbb3417063fe33b99e82724e19e107787b2e62</a></li>
+ <li><a href="https://dist.apache.org/repos/dist/release/kylin/apache-kylin-1.5.2.1/apache-kylin-1.5.2.1-cdh5.7-bin.tar.gz">apache-kylin-1.5.2.1-cdh5.7-bin.tar.gz</a></li>
+ <li>Git commit <a href="https://github.com/apache/kylin/commit/e760242daad739b66813dc2c2d8a52bf74c0ec22">e760242daad739b66813dc2c2d8a52bf74c0ec22</a></li>
</ul>
<p><strong>Build Binary Package</strong><br />
Modified: kylin/site/feed.xml
URL: http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1747333&r1=1747332&r2=1747333&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Wed Jun 8 02:03:30 2016
@@ -19,11 +19,188 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Sun, 29 May 2016 06:59:20 -0700</pubDate>
- <lastBuildDate>Sun, 29 May 2016 06:59:20 -0700</lastBuildDate>
+ <pubDate>Tue, 07 Jun 2016 18:56:49 -0700</pubDate>
+ <lastBuildDate>Tue, 07 Jun 2016 18:56:49 -0700</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>
+ <title>RAW measure in Apache Kylin</title>
+ <description><h2 id="introduction">Introduction</h2>
+
+<blockquote>
+ <p><code class="highlighter-rouge">RAW</code> measure function is use to query the detail data on the measure column in Kylin.</p>
+</blockquote>
+
+<p>Example data:</p>
+
+<table>
+ <thead>
+ <tr>
+ <th style="text-align: left">DT</th>
+ <th style="text-align: center">SITE_ID</th>
+ <th style="text-align: center">SELLER_ID</th>
+ <th style="text-align: center">ITEM_COUNT</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">2016-05-01</td>
+ <td style="text-align: center">0</td>
+ <td style="text-align: center">SELLER-001</td>
+ <td style="text-align: center">100</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2016-05-01</td>
+ <td style="text-align: center">0</td>
+ <td style="text-align: center">SELLER-002</td>
+ <td style="text-align: center">200</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2016-05-02</td>
+ <td style="text-align: center">1</td>
+ <td style="text-align: center">SELLER-003</td>
+ <td style="text-align: center">300</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2016-05-02</td>
+ <td style="text-align: center">1</td>
+ <td style="text-align: center">SELLER-004</td>
+ <td style="text-align: center">400</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2016-05-03</td>
+ <td style="text-align: center">2</td>
+ <td style="text-align: center">SELLER-005</td>
+ <td style="text-align: center">500</td>
+ </tr>
+ </tbody>
+</table>
+
+<p>We design the cube desc is the <code class="highlighter-rouge">DT,SITE_ID</code> columns as dimensions, and <code class="highlighter-rouge">SUM(ITEM_COUNT)</code> as measure. So, the base cuboid data will like this:</p>
+
+<table>
+ <thead>
+ <tr>
+ <th style="text-align: left">Rowkey of base cuboid</th>
+ <th style="text-align: center">SUM(ITEM_COUNT)</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">2016-05-01_0</td>
+ <td style="text-align: center">300</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2016-05-02_1</td>
+ <td style="text-align: center">700</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2016-05-03_2</td>
+ <td style="text-align: center">500</td>
+ </tr>
+ </tbody>
+</table>
+
+<p>For the first row in the base cuboid data, Kylin can extract the dimension column values <code class="highlighter-rouge">2016-05-01,0</code> from the HBase Rowkey, and in the measure cell will store the measure function’s aggregated results <code class="highlighter-rouge">300</code>, we can’t get the raw value <code class="highlighter-rouge">100</code> and <code class="highlighter-rouge">200</code> which before the aggregation on the <code class="highlighter-rouge">ITEM_COUNT</code> column.</p>
+
+<p>The RAW function is use to make the SQL:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT DT,SITE_ID,ITEM_COUNT FROM FACT_TABLE
+</code></pre>
+</div>
+
+<p>to return the correct result:</p>
+
+<table>
+ <thead>
+ <tr>
+ <th style="text-align: left">DT</th>
+ <th style="text-align: center">SITE_ID</th>
+ <th style="text-align: center">ITEM_COUNT</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">2016-05-01</td>
+ <td style="text-align: center">0</td>
+ <td style="text-align: center">100</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2016-05-01</td>
+ <td style="text-align: center">0</td>
+ <td style="text-align: center">200</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2016-05-02</td>
+ <td style="text-align: center">1</td>
+ <td style="text-align: center">300</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2016-05-02</td>
+ <td style="text-align: center">1</td>
+ <td style="text-align: center">400</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2016-05-03</td>
+ <td style="text-align: center">2</td>
+ <td style="text-align: center">500</td>
+ </tr>
+ </tbody>
+</table>
+
+<h2 id="how-to-use">How to use</h2>
+
+<ul>
+ <li>Choose the Kylin version <code class="highlighter-rouge">1.5.1+</code>.</li>
+ <li>Like the above case, we can make the <code class="highlighter-rouge">DT,SITE_ID</code> as dimensions, and <code class="highlighter-rouge">RAW(ITEM_COUNT)</code>as measure.</li>
+ <li>After the cube build, you can use the SQL to query the raw data:</li>
+</ul>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT DT,SITE_ID,ITEM_COUNT FROM FACT_TABLE WHERE SITE_ID = 0
+</code></pre>
+</div>
+
+<h2 id="optimize">Optimize</h2>
+
+<p>The column which define <code class="highlighter-rouge">RAW</code> measure will be encoded with dictionary by default. So, you must know you data’s cardinality and distribution characteristics.</p>
+
+<ul>
+ <li>As far as possible to define the value uniform distribution column to dimensions, this will make the measure cell value size more uniform and avoid data skew.</li>
+ <li>If choose the ultra high cardinality column to define <code class="highlighter-rouge">RAW</code> measure, you can try the following to avoid the dictionary build error:
+ <ol>
+ <li>Cut a big segment into several segments, if you were trying to build a large data set at once;</li>
+ <li>Set <code class="highlighter-rouge">kylin.dictionary.max.cardinality</code> in conf/kylin.properties to a bigger value (default is 5000000).</li>
+ </ol>
+ </li>
+</ul>
+
+<h2 id="to-be-improved">To be improved</h2>
+
+<ul>
+ <li>Now, the maximum storage 1M values of <code class="highlighter-rouge">RAW</code> measure in one cuboid. If exceed 1M values, it will throw <code class="highlighter-rouge">BufferOverflowException</code> in the cube build. This will be optimized in the later release.</li>
+ <li>Only dimension column can use in <code class="highlighter-rouge">WHERE</code> condition, <code class="highlighter-rouge">RAW</code> measure column is not support.</li>
+</ul>
+
+<h2 id="implement">Implement</h2>
+
+<ul>
+ <li>Custom one aggregation function RAW implement, the function’s return type depends on the column type.</li>
+ <li>Make the RAW aggregation function to save the column raw data in the base cuboid data.</li>
+ <li>The HBase value cell will store the dictionary id of the raw data to save space.</li>
+ <li>The SQL which contains the RAW measure column will be routed to the base cuboid query.</li>
+ <li>Extract the raw data from base cuboid data with dimension values to assemble into a complete row when query.</li>
+</ul>
+</description>
+ <pubDate>Sun, 29 May 2016 17:30:00 -0700</pubDate>
+ <link>http://kylin.apache.org/blog/2016/05/29/raw-measure-in-kylin/</link>
+ <guid isPermaLink="true">http://kylin.apache.org/blog/2016/05/29/raw-measure-in-kylin/</guid>
+
+
+ <category>blog</category>
+
+ </item>
+
+ <item>
<title>Apache Kylin v1.5.2 正式发布</title>
<description><p>Apache Kylin社区非常高兴宣布Apache Kylin v1.5.2正式发布。</p>
@@ -933,87 +1110,6 @@ send mail to Apache Kylin dev mailing li
</item>
<item>
- <title>Apache Kylin v1.3.0 正式发布</title>
- <description><p>Apache Kylin社区非常高兴宣布Apache Kylin v1.3.0正式发布，这是自顺利毕业成Apache顶级项目后的第二个发布版本。</p>
-
-<p>Apache Kylin是一个开源的分布式分析引擎，提供Hadoop之上的SQL查询接口及多维分析（OLAP）能力以支持超大规模数据，最初由eBay Inc. 开发并贡献至开源社区。</p>
-
-<p>下载Apache Kylin v1.3.0源代码及二进制安装包，<br />
-请访问<a href="http://kylin.apache.org/cn/download/">下载</a>页面.</p>
-
-<p>这是一个主要的版本发布带来了更稳定，健壮及更好管理的版本，Apache Kylin社区解决了57个issue，包括Bug修复，功能增强及一些新特性等。</p>
-
-<h2 id="section">主要变化</h2>
-
-<p><strong>新功能</strong></p>
-
-<ul>
- <li>从Kylin中卸载表 <a href="https://issues.apache.org/jira/browse/KYLIN-579">KYLIN-579</a></li>
- <li>支持自定义的Aggregation Types <a href="https://issues.apache.org/jira/browse/KYLIN-976">KYLIN-976</a></li>
- <li>支持Hive Beeline客户端 <a href="https://issues.apache.org/jira/browse/KYLIN-1054">KYLIN-1054</a></li>
- <li>克隆cube元数据 <a href="https://issues.apache.org/jira/browse/KYLIN-1128">KYLIN-1128</a></li>
- <li>在有限条件下使用bitmap支持精确的distinct count <a href="https://issues.apache.org/jira/browse/KYLIN-1186">KYLIN-1186</a></li>
-</ul>
-
-<p><strong>改进</strong></p>
-
-<ul>
- <li>HiveColumnCardinalityJob使用conf/kylin_job_conf.xml下的配置 <a href="https://issues.apache.org/jira/browse/KYLIN-955">KYLIN-955</a></li>
- <li>从RM获取状态时支持kerberos <a href="https://issues.apache.org/jira/browse/KYLIN-1014">KYLIN-1014</a></li>
- <li>允许通过选择模式来选择hive的表 <a href="https://issues.apache.org/jira/browse/KYLIN-1074">KYLIN-1074</a></li>
- <li>hive的依赖加入到tmpjars <a href="https://issues.apache.org/jira/browse/KYLIN-1082">KYLIN-1082</a></li>
- <li>cube创建cube时简化filter输入 <a href="https://issues.apache.org/jira/browse/KYLIN-1132">KYLIN-1132</a></li>
- <li>增强project级别的ACL <a href="https://issues.apache.org/jira/browse/KYLIN-1201">KYLIN-1201</a></li>
- <li>增加’Enable Cache’ 按钮 <a href="https://issues.apache.org/jira/browse/KYLIN-1211">KYLIN-1211</a></li>
- <li>增强DeployCoprocessorCLI支持filter <a href="https://issues.apache.org/jira/browse/KYLIN-1312">KYLIN-1312</a></li>
- <li>放弃job时杀死后台hadoop进程 <a href="https://issues.apache.org/jira/browse/KYLIN-1317">KYLIN-1317</a></li>
- <li>改进convert hfile的性能 <a href="https://issues.apache.org/jira/browse/KYLIN-1323">KYLIN-1323</a></li>
- <li>升级calcite到1.6 <a href="https://issues.apache.org/jira/browse/KYLIN-1343">KYLIN-1343</a></li>
-</ul>
-
-<p><strong>Bug修复</strong></p>
-
-<ul>
- <li>Calcite报出 “java.lang.Float cannot be cast to java.lang.Double” <a href="https://issues.apache.org/jira/browse/KYLIN-918">KYLIN-918</a></li>
- <li>不支持select [MeasureCol] from [FactTbl] <a href="https://issues.apache.org/jira/browse/KYLIN-1075">KYLIN-1075</a></li>
- <li>长维度值导致ArrayIndexOutOfBoundsException <a href="https://issues.apache.org/jira/browse/KYLIN-1104">KYLIN-1104</a></li>
- <li>前端相关bug <a href="https://issues.apache.org/jira/browse/KYLIN-1159">KYLIN-1159</a></li>
- <li>CubeMigrationCLI ACL 相关问题 <a href="https://issues.apache.org/jira/browse/KYLIN-1230">KYLIN-1230</a></li>
- <li>创建同名cube时model被覆盖 <a href="https://issues.apache.org/jira/browse/KYLIN-1254">KYLIN-1254</a></li>
- <li>从JDBC查询时默认返回partial result <a href="https://issues.apache.org/jira/browse/KYLIN-1274">KYLIN-1274</a></li>
- <li>有多个OR语句的PrepareStatement失败 <a href="https://issues.apache.org/jira/browse/KYLIN-1372">KYLIN-1372</a></li>
- <li>kylin.metadata.url HBase 名字空间问题 <a href="https://issues.apache.org/jira/browse/KYLIN-1400">KYLIN-1400</a></li>
- <li>Hive Column Cardinality 作业时报出 StringIndexOutOfBoundsException <a href="https://issues.apache.org/jira/browse/KYLIN-1402">KYLIN-1402</a></li>
- <li>子查询中有orderby时报出NullPointerException <a href="https://issues.apache.org/jira/browse/KYLIN-1419">KYLIN-1419</a></li>
- <li>HBase 大小精确度问题 <a href="https://issues.apache.org/jira/browse/KYLIN-1423">KYLIN-1423</a></li>
- <li>取消使用 “1970-01-01” 作为默认cube结束时间 <a href="https://issues.apache.org/jira/browse/KYLIN-1456">KYLIN-1456</a></li>
- <li>Hive的依赖在测试中被写死 <a href="https://issues.apache.org/jira/browse/KYLIN-1469">KYLIN-1469</a></li>
-</ul>
-
-<p><strong>升级</strong></p>
-
-<p>我们建议从早前颁布升级到此版本已获得更好的性能，稳定性及Bug修复等。<br />
-并且与社区最新特性及支持保持同步。</p>
-
-<p><strong>支持</strong></p>
-
-<p>升级和使用过程中有任何问题，请：<br />
-提交至Kylin的JIRA: <a href="https://issues.apache.org/jira/browse/KYLIN/">https://issues.apache.org/jira/browse/KYLIN/</a><br />
-或者<br />
-发送邮件到Apache Kylin邮件列表: <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a></p>
-
-<p><em>感谢每一位朋友的参与和贡献!</em></p>
-</description>
- <pubDate>Wed, 16 Mar 2016 15:00:00 -0700</pubDate>
- <link>http://kylin.apache.org/cn/blog/2016/03/16/release-v1.3.0/</link>
- <guid isPermaLink="true">http://kylin.apache.org/cn/blog/2016/03/16/release-v1.3.0/</guid>
-
-
- <category>blog</category>
-
- </item>
-
- <item>
<title>Apache Kylin v1.3.0 Release Announcement</title>
<description><p>The Apache Kylin community is pleased to announce the release of Apache Kylin v1.3.0, the second release after graduation.</p>
@@ -1127,124 +1223,80 @@ send mail to Apache Kylin dev mailing li
</item>
<item>
- <title>New Aggregation Group</title>
- <description><p>Full title of this article: <code class="highlighter-rouge">New Aggregation Group Design to Tackle Curse of Dimension Problem (Especially when high cardinality dimensions exist)</code></p>
-
-<h2 id="abstract">Abstract</h2>
-
-<p>Curse of dimension is an infamous problem for all of the OLAP engines based on pre-calculation. In versions prior to v1.5, Kylin tried to address the problem by some simple techniques, which relieved the problem to some degree. During our open source practices, we found these techniques lack of systematic design thinking, and incapable of addressing lots of common issues. In Kylin v1.5 we redesigned the aggregation group mechanism to make it better server all kinds of cube design scenarios.</p>
-
-<h2 id="introduction">Introduction</h2>
-
-<p>It is a known fact that Kylin speeds up query performance by pre-calculating cubes, which in term contains different combination of all dimensions, a.k.a. cuboids. The problem is that #cuboids grows exponentially with the #dimension. For example, there’re totally 8 possible cuboids for a cube with 3 dimensions, however there are 16 possible cuboids for a cube with 4 dimensions. Even though Kylin is using scalable computation framework (MapReduce) and scalable storage (HBase) to compute and store the cubes, it is still unacceptable if cube size turns up to be times bigger than the original data source.</p>
-
-<p>The solution is to prune unnecessary dimensions. As we previously discussed in http://kylin.apache.org/docs/howto/howto_optimize_cubes.html, it can be approached by two ways:</p>
-
-<p>First, we can remove dimensions those do NOT necessarily have to be dimensions. For example, imagine a date lookup table where keeps cal_dt is the PK column as well as lots of deriving columns like week_begin_dt, month_begin_dt. Even though analysts need week_begin_dt as a dimension, we can prune it as it can always be calculated from dimension cal_dt, this is the “derived” optimization.</p>
-
-<p>Second, some of combinations between dimensions can be pruned. This is the main discuss for this article, and let’s call it “combination pruning”. For example, if a dimension is specified as “mandatory”, then all of the combinations without such dimension can be pruned. If dimension A,B,C forms a “hierarchy” relation, then only combinations with A, AB or ABC shall be remained. Prior to v1.5, Kylin also had an “aggregation group” concept, which also serves for combination pruning. However it is poorly documented and hard to understand (I also found it is difficult to explain). Anyway we’ll skip it as we will re-define what “aggregation group” really is.</p>
-
-<p>During our open source practice we found some significant drawbacks for the original combination pruning techniques. Firstly, these techniques are isolated rather than systematically well designed. Secondly, the original aggregation group is poorly designed and documented that it is hardly used outside eBay. Thirdly, which is the most important one, it’s not expressive enough in terms of describing semantics.</p>
-
-<p>To illustrate the describing semantic issue, let’s imagine a transaction data cube where there is a very high cardinality dimension called buyer_id, as well as other normal dimensions like transaction date cal_dt, buyers’ location city, etc. The analyst might need to get an overview impression by grouping non-buyer_id dimensions, like grouping only cal_dt. The analyst might also need to drill down to check a specific buyer’s behavior by providing a buyer_id filter. Given the fact that buy_id has really high cardinality, once the buyer_id is determined, the related records should be very few (so just use the base cuboid and do some query time aggregation to “aggregate out” the unwanted dimensions is okay). In such cases the expected output of pruning policy should be:</p>
-
-<table>
- <tbody>
- <tr>
- <td>Cuboid</td>
- <td>Compute or Skip</td>
- <td>Reason</td>
- </tr>
- <tr>
- <td>———————-</td>
- <td>—————–</td>
- <td>————————————————————————————————————————</td>
- </tr>
- <tr>
- <td>city</td>
- <td>compute</td>
- <td>Group by location</td>
- </tr>
- <tr>
- <td>cal_dt</td>
- <td>compute</td>
- <td>Group by date</td>
- </tr>
- <tr>
- <td>buyer_id</td>
- <td>skip</td>
- <td>Group by buyer yield too many results to analyze, buyer_id should be used as a filter and used by visiting base cuboid</td>
- </tr>
- <tr>
- <td>city,cal_dt</td>
- <td>compute</td>
- <td>Group by location and date</td>
- </tr>
- <tr>
- <td>city,buyer_id</td>
- <td>skip</td>
- <td>Group by buyer yield too many results to analyze, buyer_id should be used as a filter and used by visiting base cuboid</td>
- </tr>
- <tr>
- <td>cal_dt,buyer_id</td>
- <td>skip</td>
- <td>Group by buyer yield too many results to analyze, buyer_id should be used as a filter and used by visiting base cuboid</td>
- </tr>
- <tr>
- <td>city,cal_dt,buyer_id</td>
- <td>compute</td>
- <td>Base cuboid</td>
- </tr>
- </tbody>
-</table>
-
-<p>Unfortunately there is no way to express such pruning settings with the existing semantic tools prior to Kylin v1.5</p>
-
-<h2 id="new-aggregation-group-design">New Aggregation Group Design</h2>
+ <title>Apache Kylin v1.3.0 正式发布</title>
+ <description><p>Apache Kylin社区非常高兴宣布Apache Kylin v1.3.0正式发布，这是自顺利毕业成Apache顶级项目后的第二个发布版本。</p>
-<p>In Kylin v1.5 we redesigned the aggregation group mechanism in the jira issue https://issues.apache.org/jira/browse/KYLIN-242. The issue was named “Kylin Cuboid Whitelist” because the new design even enables cube designer to specify expected cuboids by keeping a whitelist, imagine how expressive it can be!</p>
+<p>Apache Kylin是一个开源的分布式分析引擎，提供Hadoop之上的SQL查询接口及多维分析（OLAP）能力以支持超大规模数据，最初由eBay Inc. 开发并贡献至开源社区。</p>
-<p>In the new design, aggregation group (abbr. AGG) is defined as a cluster of cuboids that subject to shared rules. Cube designer can define one or more AGG for a cube, and the union of all AGGs’ contributed cuboids consists of the valid combination for a cube. Notice a cuboid is allowed to appear in multiple AGGs, and it will only be computed once during cube building.</p>
+<p>下载Apache Kylin v1.3.0源代码及二进制安装包，<br />
+请访问<a href="http://kylin.apache.org/cn/download/">下载</a>页面.</p>
-<p>If you look into the internal of AGG (https://github.com/apache/kylin/blob/kylin-1.5.0/core-cube/src/main/java/org/apache/kylin/cube/model/AggregationGroup.java) there’re two important properties defined: <code class="highlighter-rouge">@JsonProperty("includes")</code> and <code class="highlighter-rouge">@JsonProperty("select_rule")</code>.</p>
+<p>这是一个主要的版本发布带来了更稳定，健壮及更好管理的版本，Apache Kylin社区解决了57个issue，包括Bug修复，功能增强及一些新特性等。</p>
-<p><code class="highlighter-rouge">@JsonProperty("includes")</code><br />
-This property is for specifying which dimensions are included in the AGG. The value of the property must be a subset of the complete dimensions. Keep the proper minimal by including only necessary dimensions.</p>
+<h2 id="section">主要变化</h2>
-<p><code class="highlighter-rouge">@JsonProperty("select_rule")</code><br />
-Select rules are the rules that all valid cuboids in the AGG will subject to. Here cube designers can define multiple rules to apply on the included dimensions, currently there’re three types of rule:</p>
+<p><strong>新功能</strong></p>
<ul>
- <li>Hierarchy rules, described above</li>
- <li>Mandatory rule, described above</li>
- <li>Joint rules. This is a newly introduced rule. If two or more dimensions are “joint”, then any valid cuboid will either contain none of these dimensions, or contain them all. In other words, these dimensions will always be “together”. This is useful when the cube designer is sure some of the dimensions will always be queried together. It is also a nuclear weapon for combination pruning on less-likely-to-use dimensions. Suppose you have 20 dimensions, the first 10 dimensions are frequently used and the latter 10 are less likely to be used. By joining the latter 10 dimensions as “joint”, you’re effectively reducing cuboid numbers from 2^20 to 2^11. Actually this is pretty much what the old “aggregation group” mechanism was for. If you’re using it prior Kylin v1.5, our metadata upgrade tool will automatically translate it to joint semantics.<br />
-By flexibly using the new aggregation group you can in theory control whatever cuboid to compute/skip. This could significant reduce the computation and storage overhead, especially when the cube is serving for a fixed dashboard, which will reproduce SQL queries that only require some specific cuboids. In extreme cases you can configure each AGG contain only one cuboid, and a handful of AGGs will consists of the cuboid whitelist that you’ll need.</li>
+ <li>从Kylin中卸载表 <a href="https://issues.apache.org/jira/browse/KYLIN-579">KYLIN-579</a></li>
+ <li>支持自定义的Aggregation Types <a href="https://issues.apache.org/jira/browse/KYLIN-976">KYLIN-976</a></li>
+ <li>支持Hive Beeline客户端 <a href="https://issues.apache.org/jira/browse/KYLIN-1054">KYLIN-1054</a></li>
+ <li>克隆cube元数据 <a href="https://issues.apache.org/jira/browse/KYLIN-1128">KYLIN-1128</a></li>
+ <li>在有限条件下使用bitmap支持精确的distinct count <a href="https://issues.apache.org/jira/browse/KYLIN-1186">KYLIN-1186</a></li>
</ul>
-<p>Kylin’s cuboid computation scheduler will arrange all the valid cuboids’ computation order based on AGG definition. You don’t need to care about how it’s implemented, because every cuboid will just got computed and computed only once. The only thing you need to keep in mind is: don’t abuse AGG. Leverage AGG’s select rules as much as possible, and avoid introducing a lot of “single cuboid AGG” unless it’s really necessary. Too many AGG is a burden for cuboid computation scheduler, as well as the query engine.</p>
+<p><strong>改进</strong></p>
-<h2 id="buyerid-issue-revisited">Buyer_id issue revisited</h2>
+<ul>
+ <li>HiveColumnCardinalityJob使用conf/kylin_job_conf.xml下的配置 <a href="https://issues.apache.org/jira/browse/KYLIN-955">KYLIN-955</a></li>
+ <li>从RM获取状态时支持kerberos <a href="https://issues.apache.org/jira/browse/KYLIN-1014">KYLIN-1014</a></li>
+ <li>允许通过选择模式来选择hive的表 <a href="https://issues.apache.org/jira/browse/KYLIN-1074">KYLIN-1074</a></li>
+ <li>hive的依赖加入到tmpjars <a href="https://issues.apache.org/jira/browse/KYLIN-1082">KYLIN-1082</a></li>
+ <li>cube创建cube时简化filter输入 <a href="https://issues.apache.org/jira/browse/KYLIN-1132">KYLIN-1132</a></li>
+ <li>增强project级别的ACL <a href="https://issues.apache.org/jira/browse/KYLIN-1201">KYLIN-1201</a></li>
+ <li>增加’Enable Cache’ 按钮 <a href="https://issues.apache.org/jira/browse/KYLIN-1211">KYLIN-1211</a></li>
+ <li>增强DeployCoprocessorCLI支持filter <a href="https://issues.apache.org/jira/browse/KYLIN-1312">KYLIN-1312</a></li>
+ <li>放弃job时杀死后台hadoop进程 <a href="https://issues.apache.org/jira/browse/KYLIN-1317">KYLIN-1317</a></li>
+ <li>改进convert hfile的性能 <a href="https://issues.apache.org/jira/browse/KYLIN-1323">KYLIN-1323</a></li>
+ <li>升级calcite到1.6 <a href="https://issues.apache.org/jira/browse/KYLIN-1343">KYLIN-1343</a></li>
+</ul>
-<p>Now that we have got the new AGG tool, the buyer_id issue can be revisited. What we need to do is to define two AGGs for the cube:</p>
+<p><strong>Bug修复</strong></p>
<ul>
- <li>AGG1 includes: [cal_dt, city, buyer_id] select_rules:{joint:[cal_dt,city,buyer_id]}</li>
- <li>AGG2 includes: [cal_dt,city] select rules:{}</li>
+ <li>Calcite报出 “java.lang.Float cannot be cast to java.lang.Double” <a href="https://issues.apache.org/jira/browse/KYLIN-918">KYLIN-918</a></li>
+ <li>不支持select [MeasureCol] from [FactTbl] <a href="https://issues.apache.org/jira/browse/KYLIN-1075">KYLIN-1075</a></li>
+ <li>长维度值导致ArrayIndexOutOfBoundsException <a href="https://issues.apache.org/jira/browse/KYLIN-1104">KYLIN-1104</a></li>
+ <li>前端相关bug <a href="https://issues.apache.org/jira/browse/KYLIN-1159">KYLIN-1159</a></li>
+ <li>CubeMigrationCLI ACL 相关问题 <a href="https://issues.apache.org/jira/browse/KYLIN-1230">KYLIN-1230</a></li>
+ <li>创建同名cube时model被覆盖 <a href="https://issues.apache.org/jira/browse/KYLIN-1254">KYLIN-1254</a></li>
+ <li>从JDBC查询时默认返回partial result <a href="https://issues.apache.org/jira/browse/KYLIN-1274">KYLIN-1274</a></li>
+ <li>有多个OR语句的PrepareStatement失败 <a href="https://issues.apache.org/jira/browse/KYLIN-1372">KYLIN-1372</a></li>
+ <li>kylin.metadata.url HBase 名字空间问题 <a href="https://issues.apache.org/jira/browse/KYLIN-1400">KYLIN-1400</a></li>
+ <li>Hive Column Cardinality 作业时报出 StringIndexOutOfBoundsException <a href="https://issues.apache.org/jira/browse/KYLIN-1402">KYLIN-1402</a></li>
+ <li>子查询中有orderby时报出NullPointerException <a href="https://issues.apache.org/jira/browse/KYLIN-1419">KYLIN-1419</a></li>
+ <li>HBase 大小精确度问题 <a href="https://issues.apache.org/jira/browse/KYLIN-1423">KYLIN-1423</a></li>
+ <li>取消使用 “1970-01-01” 作为默认cube结束时间 <a href="https://issues.apache.org/jira/browse/KYLIN-1456">KYLIN-1456</a></li>
+ <li>Hive的依赖在测试中被写死 <a href="https://issues.apache.org/jira/browse/KYLIN-1469">KYLIN-1469</a></li>
</ul>
-<hr />
+<p><strong>升级</strong></p>
-<p>The first AGG will contribute the base cuboid only, and the second AGG will contribute all the cuboids without buyer_id.</p>
+<p>我们建议从早前颁布升级到此版本已获得更好的性能，稳定性及Bug修复等。<br />
+并且与社区最新特性及支持保持同步。</p>
-<h2 id="start-using-it">Start using it</h2>
+<p><strong>支持</strong></p>
-<p>The new aggregation group mechanism should be available in Kylin v1.5. Up to today (2016.2.18) Kylin has not released v1.5 version yet. Use it at your own risk by compiling the latest master code branch.</p>
+<p>升级和使用过程中有任何问题，请：<br />
+提交至Kylin的JIRA: <a href="https://issues.apache.org/jira/browse/KYLIN/">https://issues.apache.org/jira/browse/KYLIN/</a><br />
+或者<br />
+发送邮件到Apache Kylin邮件列表: <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a></p>
-<p>For legacy users you will need to upgrade your metadata store from Kylin v1.2 or v1.3 to Kylin v1.5.</p>
+<p><em>感谢每一位朋友的参与和贡献!</em></p>
</description>
- <pubDate>Thu, 18 Feb 2016 08:30:00 -0800</pubDate>
- <link>http://kylin.apache.org/blog/2016/02/18/new-aggregation-group/</link>
- <guid isPermaLink="true">http://kylin.apache.org/blog/2016/02/18/new-aggregation-group/</guid>
+ <pubDate>Wed, 16 Mar 2016 15:00:00 -0700</pubDate>
+ <link>http://kylin.apache.org/cn/blog/2016/03/16/release-v1.3.0/</link>
+ <guid isPermaLink="true">http://kylin.apache.org/cn/blog/2016/03/16/release-v1.3.0/</guid>
<category>blog</category>