You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by li...@apache.org on 2016/03/12 14:18:11 UTC
svn commit: r1734700 [14/14] - in /kylin/site: ./
blog/2016/02/03/streaming-cubing/ blog/2016/02/18/new-aggregation-group/
development/ docs/ docs/gettingstarted/ docs/howto/ docs15/
docs15/gettingstarted/ docs15/howto/ docs15/install/ docs15/tutorial/
Modified: kylin/site/feed.xml
URL: http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1734700&r1=1734699&r2=1734700&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Sat Mar 12 13:18:10 2016
@@ -19,8 +19,8 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Thu, 10 Mar 2016 12:01:13 -0800</pubDate>
- <lastBuildDate>Thu, 10 Mar 2016 12:01:13 -0800</lastBuildDate>
+ <pubDate>Sat, 12 Mar 2016 13:16:43 -0800</pubDate>
+ <lastBuildDate>Sat, 12 Mar 2016 13:16:43 -0800</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>
@@ -29,7 +29,7 @@
<h2 id="abstract">Abstract</h2>
-<p>Curse of dimension is an infamous problem for all of the OLAP engines based on pre-calculation. In versions prior to 2.1, Kylin tried to address the problem by some simple techniques, which relieved the problem to some degree. During our open source practices, we found these techniques lack of systematic design thinking, and incapable of addressing lots of common issues. In Kylin 2.1 we redesigned the aggregation group mechanism to make it better server all kinds of cube design scenarios.</p>
+<p>Curse of dimension is an infamous problem for all of the OLAP engines based on pre-calculation. In versions prior to v1.5, Kylin tried to address the problem by some simple techniques, which relieved the problem to some degree. During our open source practices, we found these techniques lack of systematic design thinking, and incapable of addressing lots of common issues. In Kylin v1.5 we redesigned the aggregation group mechanism to make it better server all kinds of cube design scenarios.</p>
<h2 id="introduction">Introduction</h2>
@@ -39,7 +39,7 @@
<p>First, we can remove dimensions those do NOT necessarily have to be dimensions. For example, imagine a date lookup table where keeps cal_dt is the PK column as well as lots of deriving columns like week_begin_dt, month_begin_dt. Even though analysts need week_begin_dt as a dimension, we can prune it as it can always be calculated from dimension cal_dt, this is the âderivedâ optimization.</p>
-<p>Second, some of combinations between dimensions can be pruned. This is the main discuss for this article, and letâs call it âcombination pruningâ. For example, if a dimension is specified as âmandatoryâ, then all of the combinations without such dimension can be pruned. If dimension A,B,C forms a âhierarchyâ relation, then only combinations with A, AB or ABC shall be remained. Prior to 2.1, Kylin also had an âaggregation groupâ concept, which also serves for combination pruning. However it is poorly documented and hard to understand (I also found it is difficult to explain). Anyway weâll skip it as we will re-define what âaggregation groupâ really is.</p>
+<p>Second, some of combinations between dimensions can be pruned. This is the main discuss for this article, and letâs call it âcombination pruningâ. For example, if a dimension is specified as âmandatoryâ, then all of the combinations without such dimension can be pruned. If dimension A,B,C forms a âhierarchyâ relation, then only combinations with A, AB or ABC shall be remained. Prior to v1.5, Kylin also had an âaggregation groupâ concept, which also serves for combination pruning. However it is poorly documented and hard to understand (I also found it is difficult to explain). Anyway weâll skip it as we will re-define what âaggregation groupâ really is.</p>
<p>During our open source practice we found some significant drawbacks for the original combination pruning techniques. Firstly, these techniques are isolated rather than systematically well designed. Secondly, the original aggregation group is poorly designed and documented that it is hardly used outside eBay. Thirdly, which is the most important one, itâs not expressive enough in terms of describing semantics.</p>
@@ -95,15 +95,15 @@
</tbody>
</table>
-<p>Unfortunately there is no way to express such pruning settings with the existing semantic tools prior to Kylin 2.1</p>
+<p>Unfortunately there is no way to express such pruning settings with the existing semantic tools prior to Kylin v1.5</p>
<h2 id="new-aggregation-group-design">New Aggregation Group Design</h2>
-<p>In Kylin 2.1 we redesigned the aggregation group mechanism in the jira issue https://issues.apache.org/jira/browse/KYLIN-242. The issue was named âKylin Cuboid Whitelistâ because the new design even enables cube designer to specify expected cuboids by keeping a whitelist, imagine how expressive it can be!</p>
+<p>In Kylin v1.5 we redesigned the aggregation group mechanism in the jira issue https://issues.apache.org/jira/browse/KYLIN-242. The issue was named âKylin Cuboid Whitelistâ because the new design even enables cube designer to specify expected cuboids by keeping a whitelist, imagine how expressive it can be!</p>
<p>In the new design, aggregation group (abbr. AGG) is defined as a cluster of cuboids that subject to shared rules. Cube designer can define one or more AGG for a cube, and the union of all AGGsâ contributed cuboids consists of the valid combination for a cube. Notice a cuboid is allowed to appear in multiple AGGs, and it will only be computed once during cube building.</p>
-<p>If you look into the internal of AGG ( https://github.com/apache/kylin/blob/2.x-staging/core-cube/src/main/java/org/apache/kylin/cube/model/AggregationGroup.java) thereâre two important properties defined: <code class="highlighter-rouge">@JsonProperty("includes")</code> and <code class="highlighter-rouge">@JsonProperty("select_rule")</code>.</p>
+<p>If you look into the internal of AGG (https://github.com/apache/kylin/blob/kylin-1.5.0/core-cube/src/main/java/org/apache/kylin/cube/model/AggregationGroup.java) thereâre two important properties defined: <code class="highlighter-rouge">@JsonProperty("includes")</code> and <code class="highlighter-rouge">@JsonProperty("select_rule")</code>.</p>
<p><code class="highlighter-rouge">@JsonProperty("includes")</code><br />
This property is for specifying which dimensions are included in the AGG. The value of the property must be a subset of the complete dimensions. Keep the proper minimal by including only necessary dimensions.</p>
@@ -114,7 +114,7 @@ Select rules are the rules that all vali
<ul>
<li>Hierarchy rules, described above</li>
<li>Mandatory rule, described above</li>
- <li>Joint rules. This is a newly introduced rule. If two or more dimensions are âjointâ, then any valid cuboid will either contain none of these dimensions, or contain them all. In other words, these dimensions will always be âtogetherâ. This is useful when the cube designer is sure some of the dimensions will always be queried together. It is also a nuclear weapon for combination pruning on less-likely-to-use dimensions. Suppose you have 20 dimensions, the first 10 dimensions are frequently used and the latter 10 are less likely to be used. By joining the latter 10 dimensions as âjointâ, youâre effectively reducing cuboid numbers from 220 to 211. Actually this is pretty much what the old âaggregation groupâ mechanism was for. If youâre using it prior Kylin 2.1, our metadata upgrade tool will automatically translate it to joint semantics.<br />
+ <li>Joint rules. This is a newly introduced rule. If two or more dimensions are âjointâ, then any valid cuboid will either contain none of these dimensions, or contain them all. In other words, these dimensions will always be âtogetherâ. This is useful when the cube designer is sure some of the dimensions will always be queried together. It is also a nuclear weapon for combination pruning on less-likely-to-use dimensions. Suppose you have 20 dimensions, the first 10 dimensions are frequently used and the latter 10 are less likely to be used. By joining the latter 10 dimensions as âjointâ, youâre effectively reducing cuboid numbers from 220 to 211. Actually this is pretty much what the old âaggregation groupâ mechanism was for. If youâre using it prior Kylin v1.5, our metadata upgrade tool will automatically translate it to joint semantics.<br />
By flexibly using the new aggregation group you can in theory control whatever cuboid to compute/skip. This could significant reduce the computation and storage overhead, especially when the cube is serving for a fixed dashboard, which will reproduce SQL queries that only require some specific cuboids. In extreme cases you can configure each AGG contain only one cuboid, and a handful of AGGs will consists of the cuboid whitelist that youâll need.</li>
</ul>
@@ -135,9 +135,9 @@ By flexibly using the new aggregation gr
<h2 id="start-using-it">Start using it</h2>
-<p>The new aggregation group mechanism should be available in Kylin 2.1. Up to today (2016.2.18) Kylin has not released 2.1 version yet. Use it at your own risk by compiling the latest 2.x-staging code branch.</p>
+<p>The new aggregation group mechanism should be available in Kylin v1.5. Up to today (2016.2.18) Kylin has not released v1.5 version yet. Use it at your own risk by compiling the latest master code branch.</p>
-<p>For legacy users you will need to upgrade your metadata store from Kylin 2.0 to Kylin 2.1. Cube rebuild is not required if youâre upgrading from Kylin 2.0.</p>
+<p>For legacy users you will need to upgrade your metadata store from Kylin v1.4 to Kylin v1.5. Cube rebuild is not required if youâre upgrading from Kylin v1.4.</p>
</description>
<pubDate>Thu, 18 Feb 2016 08:30:00 -0800</pubDate>
<link>http://kylin.apache.org/blog/2016/02/18/new-aggregation-group/</link>
@@ -150,7 +150,7 @@ By flexibly using the new aggregation gr
<item>
<title>Streaming cubing (Prototype)</title>
- <description><p>One of the most important features in 2.x branches is streaming cubing which enables OLAP analysis on streaming data. Streaming cubing delivers faster insights on the data to help more promptly business decisions. Even though there are already many real time analysis tools in open source community, Kylin Streaming cubing still differs from them in multiple angles:</p>
+ <description><p>One of the most important features in v1.5 is streaming cubing which enables OLAP analysis on streaming data. Streaming cubing delivers faster insights on the data to help more promptly business decisions. Even though there are already many real time analysis tools in open source community, Kylin Streaming cubing still differs from them in multiple angles:</p>
<p>Firstly, Kylin Streaming Cubing aligns with Kylin traditional cubing to provided unified, ANSI SQL interface. Actually Kylin Streaming shares the storage engine and query engine with traditional Kylin cubes, so in theory all of the optimization techniques to save storage and speed up query performance can also be applied on streaming cubes. Besides, all the supported aggregations/filters/UDFs still work for streaming cubes. By unifying the storage engine and query engine we also get freed from double amount of maintaince work.</p>
@@ -168,7 +168,7 @@ By flexibly using the new aggregation gr
<li>Job Scheduling Module to trigger Streaming Batch Ingestion. Kylin does not put too much efforts in job scheduling, streaming cubing is not a exception. Currently we provided a simple implementation based on Linux Crontab.</li>
</ul>
-<p>Weâll publish more detailed documents on how to use Kylin Streaming soon. In latest 2.x branch we are also working on more complicated load balancing schemes for streaming cubing. Please stay tuned.</p>
+<p>Weâll publish more detailed documents on how to use Kylin Streaming soon. In latest v1.5 we are also working on more complicated load balancing schemes for streaming cubing. Please stay tuned.</p>
</description>
<pubDate>Wed, 03 Feb 2016 08:30:00 -0800</pubDate>