You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by ma...@apache.org on 2016/02/03 09:35:20 UTC
svn commit: r1728270 - in /kylin/site: ./ blog/ blog/2016/ blog/2016/02/
blog/2016/02/03/ blog/2016/02/03/streaming-cubing/ development/
Author: mahongbin
Date: Wed Feb 3 08:35:20 2016
New Revision: 1728270
URL: http://svn.apache.org/viewvc?rev=1728270&view=rev
Log:
UPDATE MESSAGE
Added:
kylin/site/blog/2016/
kylin/site/blog/2016/02/
kylin/site/blog/2016/02/03/
kylin/site/blog/2016/02/03/streaming-cubing/
kylin/site/blog/2016/02/03/streaming-cubing/index.html
Modified:
kylin/site/blog/index.html
kylin/site/development/about_temp_files.html
kylin/site/development/dev_env.html
kylin/site/development/howto_contribute.html
kylin/site/development/howto_docs.html
kylin/site/development/howto_package.html
kylin/site/development/howto_release.html
kylin/site/development/howto_test.html
kylin/site/development/index.html
kylin/site/development/new_metadata.html
kylin/site/development/web_tech.html
kylin/site/feed.xml
Added: kylin/site/blog/2016/02/03/streaming-cubing/index.html
URL: http://svn.apache.org/viewvc/kylin/site/blog/2016/02/03/streaming-cubing/index.html?rev=1728270&view=auto
==============================================================================
--- kylin/site/blog/2016/02/03/streaming-cubing/index.html (added)
+++ kylin/site/blog/2016/02/03/streaming-cubing/index.html Wed Feb 3 08:35:20 2016
@@ -0,0 +1,272 @@
+<!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+<!doctype html>
+<html>
+ <!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <title>Apache Kylin | Streaming cubing (Prototype)</title>
+ <meta name="description" content="One of the most important features in 2.x branches is streaming cubing which enables OLAP analysis on streaming data. Streaming cubing delivers faster insigh...">
+ <meta name="author" content="Apache Kylin">
+ <link rel="shortcut icon" href="fav.png" type="image/png">
+
+
+
+<link rel="stylesheet" href="/assets/css/animate.css">
+<!-- Bootstrap -->
+<link rel="stylesheet" href="/assets/css/bootstrap.min.css">
+
+<!-- Fonts -->
+<!-- <link rel="stylesheet" href="http://fonts.googleapis.com/css?family=Alice|Open+Sans:400,300,700"> -->
+
+<!-- Icons -->
+<link rel="stylesheet" href="/assets/css/font-awesome.min.css">
+
+ <!-- Custom styles -->
+ <link rel="stylesheet" href="/assets/css/styles.css">
+ <link rel="stylesheet" href="/assets/css/docs.css">
+ <link rel="stylesheet" href="/assets/css/pygments.css">
+
+ <link rel="canonical" href="http://kylin.apache.org/blog/2016/02/03/streaming-cubing/">
+ <link rel="alternate" type="application/rss+xml" title="Apache Kylin" href="http://kylin.apache.org/feed.xml" />
+
+<!--[if lt IE 9]> <script src="assets/js/html5shiv.js"></script> <![endif]-->
+<script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ //oringal tracker for kylin.io
+ ga('create', 'UA-55534813-1', 'auto');
+ //new tracker for kylin.apache.org
+ ga('create', 'UA-55534813-2', 'auto', {'name':'toplevel'});
+
+ ga('send', 'pageview');
+ ga('toplevel.send', 'pageview');
+
+
+</script>
+<script type="text/javascript" src="/assets/js/jquery-1.9.1.min.js"></script>
+<script type="text/javascript" src="/assets/js/nside.js"></script> </script>
+<script type="text/javascript" src="/assets/js/nnav.js"></script> </script>
+</head>
+
+ <body>
+ <!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+
+<header id="header" >
+
+ <div id="head" class="parallax" parallax-speed="3" >
+ <div id="logo" class="text-center"> <img class="img-circle" id="circlelogo" src="/assets/images/kylin_logo.jpg"> <span class="title" >Apache Kylinâ¢</span> <span class="tagline">Extreme OLAP Engine for Big Data</span>
+ </div>
+ </div>
+
+
+ <!-- Main Menu -->
+ <nav class="navbar navbar-default" role="navigation" id="nav-wrapper">
+ <div class="container-fluid" id="nav">
+ <!--
+ <img class="img-circle" width="40px" height="40px" id="circlelogo" src="/assets/images/kylin_logo.jpg">
+ -->
+ <!-- Brand and toggle get grouped for better mobile display -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ </div>
+
+ <!-- Collect the nav links, forms, and other content for toggling -->
+ <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav">
+ <li><a href="/">Home</a></li>
+ <li><a href="/docs" >Docs</a></li>
+ <li><a href="/download">Download</li>
+ <li><a href="/community" >Community</a></li>
+ <li><a href="/development" >Development</a></li>
+ <li><a href="/blog">Blog</li>
+ <li><a href="/cn" >ä¸æç</a></li>
+ <li><a href="https://twitter.com/apachekylin" target="_blank" class="fa fa-twitter fa-lg" title="Twitter: @ApacheKylin" ></a></li>
+ <li><a href="https://github.com/apache/kylin" target="_blank" class="fa fa-github-alt fa-lg" title="Github: apache/kylin" ></a></li>
+ <li><a href="https://www.facebook.com/kylinio" target="_blank" class="fa fa-facebook fa-lg" title="Facebook: kylin.io" ></a></li>
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </div><!-- /.container-fluid -->
+</nav>
+ </header>
+
+ <div class="page-content">
+ <header style=" padding:2em 0 0 0">
+ <div class="container" >
+ <h4 class="section-title"><span>Apache Kylin⢠Technical Blog</span></h4>
+ </div>
+ </div>
+
+ <div class="container">
+ <div>
+ <article class="post-content" >
+ <!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+
+<div class="post" style=" padding:2em 4em 4em 4em">
+
+ <header class="post-header">
+ <h1 class="post-title">Streaming cubing (Prototype)</h1>
+ <p class="post-meta" >Feb 3, 2016 ⢠Hongbin Ma</p>
+ </header>
+
+ <article class="post-content" >
+ <p>One of the most important features in 2.x branches is streaming cubing which enables OLAP analysis on streaming data. Streaming cubing delivers faster insights on the data to help more promptly business decisions. Even though there are already many real time analysis tools in open source community, Kylin Streaming cubing still differs from them in multiple angles:</p>
+
+<p>Firstly, Kylin Streaming Cubing aligns with Kylin traditional cubing to provided unified, ANSI SQL interface. Actually Kylin Streaming shares the storage engine and query engine with traditional Kylin cubes, so in theory all of the optimization techniques to save storage and speed up query performance can also be applied on streaming cubes. Besides, all the supported aggregations/filters/UDFs still work for streaming cubes. By unifying the storage engine and query engine we also get freed from double amount of maintaince work.</p>
+
+<p>Secondly, Kylin Streaming Cubing does not require large amount of memory to store real time data, nor does it attempts to provide truly âreal timeâ analysis. By our customer survey we found that minutes of visualization latency is acceptable for OLAP analysts. So our streaming cubing adopts the micro batch approach. Incoming streaming data are partitioned into different time windows and we build a micro batch for each time window. The cube output for each micro batch is directly saved to HBase. The query engine goes to HBase for data retrieving instead of the data ingestion server. The benefit of such design is that we donât have to maintain large amount of in-memory index which could easily require tens of gigabytes of memory. In the future Kylin might need to consider truly âreal timeâ support, too.</p>
+
+<p>Thirdly, Kylin Streaming Cubing data will be persistent and gradually be converted to traditional cubes, thus customers can still query âcold dataâ without any compromise on performance. As discussed above the output of streaming cubing is directly saved to HBase as a new segment. The traditional job engine will be notified of the new segment and take over to schedule merge jobs when then segments accumulates. Day after day the segments of the streaming cube got merged and become a very large traditional cube.</p>
+
+<p><img src="/images/develop/streaming.png" alt="Kylin Streaming Framework Architecture" /></p>
+
+<p>With the major difference in mind we will introduce the modules for Kylin Streaming cubing. Kylin Streaming cubing consist of three major parts:</p>
+
+<ul>
+ <li>Streaming Input to retrieve data from a replayable data queue (currently it is Kafka) within given time window. Streaming Input is also responsible for primary data cleaning and normalization. By default Kylin Streaming provides a default implementation to parse the messages from the source queue. Customers can choose to configure the parser or provide a brand new one based on their requirements.</li>
+ <li>Streaming Batch Ingestion to ingest the incoming data batch and transform it into a micro cube. Thanks to the latest Kylin In-memory cubing technology, this step is now times faster and space-saving than previous. The micro cube is directly saved to HBase.</li>
+ <li>Job Scheduling Module to trigger Streaming Batch Ingestion. Kylin does not put too much efforts in job scheduling, streaming cubing is not a exception. Currently we provided a simple implementation based on Linux Crontab.</li>
+</ul>
+
+<p>Weâll publish more detailed documents on how to use Kylin Streaming soon. In latest 2.x branch we are also working on more complicated load balancing schemes for streaming cubing. Please stay tuned.</p>
+
+
+ </article>
+
+</div>
+
+
+
+
+
+ </article>
+ </div>
+ </div>
+ <!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+
+<footer id="underfooter">
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12 widget">
+ <div class="widget-body" style="text-align:center">
+ <a href="http://www.apache.org">
+ <img id="asf-logo" alt="Apache Software Foundation" src="/assets/images/feather-small.gif">
+ </a>
+
+ <div>
+ The contents of this website are © 2015 Apache Software Foundation under the terms of the <a
+ href="http://www.apache.org/licenses/LICENSE-2.0"> Apache License v2 </a>. Apache Kylin and
+ its logo are trademarks of the Apache Software Foundation.
+ </div>
+
+ </div>
+ </div>
+ </div>
+ <!-- /row of widgets -->
+
+ </div>
+ <div></div>
+
+</footer>
+
+ <script src="/assets/js/jquery-1.9.1.min.js"></script>
+ <script src="/assets/js/bootstrap.min.js"></script>
+ <script src="/assets/js/main.js"></script>
+ </body>
+</html>
+
+
+
+
Modified: kylin/site/blog/index.html
URL: http://svn.apache.org/viewvc/kylin/site/blog/index.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/blog/index.html (original)
+++ kylin/site/blog/index.html Wed Feb 3 08:35:20 2016
@@ -174,6 +174,12 @@
<li>
<h2 align="left" style="margin:0px">
+ <a class="post-link" href="/blog/2016/02/03/streaming-cubing/">Streaming cubing (Prototype)</a></h2><div align="left" class="post-meta">posted: Feb 3, 2016</div>
+
+ </li>
+
+ <li>
+ <h2 align="left" style="margin:0px">
<a class="post-link" href="/blog/2015/12/25/support-powerbi-tableau9/">Apache Kylin supports Tableau 9 and MS Excel, Power BI now</a></h2><div align="left" class="post-meta">posted: Dec 25, 2015</div>
</li>
Modified: kylin/site/development/about_temp_files.html
URL: http://svn.apache.org/viewvc/kylin/site/development/about_temp_files.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/about_temp_files.html (original)
+++ kylin/site/development/about_temp_files.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html" class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing (Prototype)</a></li>
-
-
-
Modified: kylin/site/development/dev_env.html
URL: http://svn.apache.org/viewvc/kylin/site/development/dev_env.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/dev_env.html (original)
+++ kylin/site/development/dev_env.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html" class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing (Prototype)</a></li>
-
-
-
Modified: kylin/site/development/howto_contribute.html
URL: http://svn.apache.org/viewvc/kylin/site/development/howto_contribute.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/howto_contribute.html (original)
+++ kylin/site/development/howto_contribute.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html" class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing (Prototype)</a></li>
-
-
-
Modified: kylin/site/development/howto_docs.html
URL: http://svn.apache.org/viewvc/kylin/site/development/howto_docs.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/howto_docs.html (original)
+++ kylin/site/development/howto_docs.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html" class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing (Prototype)</a></li>
-
-
-
Modified: kylin/site/development/howto_package.html
URL: http://svn.apache.org/viewvc/kylin/site/development/howto_package.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/howto_package.html (original)
+++ kylin/site/development/howto_package.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html" class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing (Prototype)</a></li>
-
-
-
Modified: kylin/site/development/howto_release.html
URL: http://svn.apache.org/viewvc/kylin/site/development/howto_release.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/howto_release.html (original)
+++ kylin/site/development/howto_release.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html" class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing (Prototype)</a></li>
-
-
-
Modified: kylin/site/development/howto_test.html
URL: http://svn.apache.org/viewvc/kylin/site/development/howto_test.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/howto_test.html (original)
+++ kylin/site/development/howto_test.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html" class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing (Prototype)</a></li>
-
-
-
Modified: kylin/site/development/index.html
URL: http://svn.apache.org/viewvc/kylin/site/development/index.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/index.html (original)
+++ kylin/site/development/index.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html" class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing (Prototype)</a></li>
-
-
-
Modified: kylin/site/development/new_metadata.html
URL: http://svn.apache.org/viewvc/kylin/site/development/new_metadata.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/new_metadata.html (original)
+++ kylin/site/development/new_metadata.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html" class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing (Prototype)</a></li>
-
-
-
Modified: kylin/site/development/web_tech.html
URL: http://svn.apache.org/viewvc/kylin/site/development/web_tech.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/web_tech.html (original)
+++ kylin/site/development/web_tech.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html" class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing (Prototype)</a></li>
-
-
-
Modified: kylin/site/feed.xml
URL: http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Wed Feb 3 08:35:20 2016
@@ -19,11 +19,43 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Wed, 03 Feb 2016 00:25:14 -0800</pubDate>
- <lastBuildDate>Wed, 03 Feb 2016 00:25:14 -0800</lastBuildDate>
+ <pubDate>Wed, 03 Feb 2016 00:34:04 -0800</pubDate>
+ <lastBuildDate>Wed, 03 Feb 2016 00:34:04 -0800</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>
+ <title>Streaming cubing (Prototype)</title>
+ <description><p>One of the most important features in 2.x branches is streaming cubing which enables OLAP analysis on streaming data. Streaming cubing delivers faster insights on the data to help more promptly business decisions. Even though there are already many real time analysis tools in open source community, Kylin Streaming cubing still differs from them in multiple angles:</p>
+
+<p>Firstly, Kylin Streaming Cubing aligns with Kylin traditional cubing to provided unified, ANSI SQL interface. Actually Kylin Streaming shares the storage engine and query engine with traditional Kylin cubes, so in theory all of the optimization techniques to save storage and speed up query performance can also be applied on streaming cubes. Besides, all the supported aggregations/filters/UDFs still work for streaming cubes. By unifying the storage engine and query engine we also get freed from double amount of maintaince work.</p>
+
+<p>Secondly, Kylin Streaming Cubing does not require large amount of memory to store real time data, nor does it attempts to provide truly âreal timeâ analysis. By our customer survey we found that minutes of visualization latency is acceptable for OLAP analysts. So our streaming cubing adopts the micro batch approach. Incoming streaming data are partitioned into different time windows and we build a micro batch for each time window. The cube output for each micro batch is directly saved to HBase. The query engine goes to HBase for data retrieving instead of the data ingestion server. The benefit of such design is that we donât have to maintain large amount of in-memory index which could easily require tens of gigabytes of memory. In the future Kylin might need to consider truly âreal timeâ support, too.</p>
+
+<p>Thirdly, Kylin Streaming Cubing data will be persistent and gradually be converted to traditional cubes, thus customers can still query âcold dataâ without any compromise on performance. As discussed above the output of streaming cubing is directly saved to HBase as a new segment. The traditional job engine will be notified of the new segment and take over to schedule merge jobs when then segments accumulates. Day after day the segments of the streaming cube got merged and become a very large traditional cube.</p>
+
+<p><img src="/images/develop/streaming.png" alt="Kylin Streaming Framework Architecture" /></p>
+
+<p>With the major difference in mind we will introduce the modules for Kylin Streaming cubing. Kylin Streaming cubing consist of three major parts:</p>
+
+<ul>
+ <li>Streaming Input to retrieve data from a replayable data queue (currently it is Kafka) within given time window. Streaming Input is also responsible for primary data cleaning and normalization. By default Kylin Streaming provides a default implementation to parse the messages from the source queue. Customers can choose to configure the parser or provide a brand new one based on their requirements.</li>
+ <li>Streaming Batch Ingestion to ingest the incoming data batch and transform it into a micro cube. Thanks to the latest Kylin In-memory cubing technology, this step is now times faster and space-saving than previous. The micro cube is directly saved to HBase.</li>
+ <li>Job Scheduling Module to trigger Streaming Batch Ingestion. Kylin does not put too much efforts in job scheduling, streaming cubing is not a exception. Currently we provided a simple implementation based on Linux Crontab.</li>
+</ul>
+
+<p>Weâll publish more detailed documents on how to use Kylin Streaming soon. In latest 2.x branch we are also working on more complicated load balancing schemes for streaming cubing. Please stay tuned.</p>
+
+</description>
+ <pubDate>Wed, 03 Feb 2016 08:30:00 -0800</pubDate>
+ <link>http://kylin.apache.org/blog/2016/02/03/streaming-cubing/</link>
+ <guid isPermaLink="true">http://kylin.apache.org/blog/2016/02/03/streaming-cubing/</guid>
+
+
+ <category>blog</category>
+
+ </item>
+
+ <item>
<title>Apache Kylin supports Tableau 9 and MS Excel, Power BI now</title>
<description><p>Apache Kylin Community has rolled out an updated ODBC Driver (v1.2), which enforced more support for Business Intelligence clients, including Microsoft Excel, Microsoft Power BI and Tableau 9. This update did bring more convenience and selectivity for analysts and end users.</p>
@@ -88,67 +120,6 @@ With sub-seconds query latency feature o
</item>
<item>
- <title>Apache Kylin v1.2 Release Announcement</title>
- <description><p>The Apache Kylin community is pleased to announce the release of Apache Kylin v1.2, the first release after graduation.</p>
-
-<p>Apache Kylin is an open source Distributed Analytics Engine designed to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets, original contributed from eBay Inc.</p>
-
-<p>To download Apache Kylin v1.2 source code or binary package: <br />
-please visit the <a href="http://kylin.apache.org/download">download</a> page.</p>
-
-<p>This is a major release which brings more stable, robust and well management version, Apache Kylin community resolved about 44 issues including bug fixes, improvements, and few new features.</p>
-
-<h2 id="change-highlights">Change Highlights</h2>
-
-<p><strong>Kylin Core Improvement</strong></p>
-
-<ul>
- <li>Support Excel, Power BI and Tableau 9.1 <a href="https://issues.apache.org/jira/browse/KYLIN-596">KYLIN-596</a>,<a href="https://issues.apache.org/jira/browse/KYLIN-1065">KYLIN-1065</a></li>
- <li>Improve small file management on HDFS <a href="https://issues.apache.org/jira/browse/KYLIN-702">KYLIN-702</a></li>
- <li>Env shell script enhance for Hive HCatalog <a href="https://issues.apache.org/jira/browse/KYLIN-1081">KYLIN-1081</a>, <a href="https://issues.apache.org/jira/browse/KYLIN-1119">KYLIN-1119</a></li>
- <li>Dimenion column supports high cardinality over 10 million <a href="https://issues.apache.org/jira/browse/KYLIN-1099">KYLIN-1099</a></li>
- <li>Enhance job page loading performance <a href="https://issues.apache.org/jira/browse/KYLIN-1154">KYLIN-1154</a></li>
- <li>Make memory budget per query configurable <a href="https://issues.apache.org/jira/browse/KYLIN-1190">KYLIN-1190</a></li>
-</ul>
-
-<p><strong>Main Bug Fixes</strong></p>
-
-<ul>
- <li>Save cube issue in edit model <a href="https://issues.apache.org/jira/browse/KYLIN-1168">KYLIN-1168</a></li>
- <li>Couldnât change a cubeâs name after it be created <a href="https://issues.apache.org/jira/browse/KYLIN-693">KYLIN-693</a></li>
- <li>Cube list missing under project <a href="https://issues.apache.org/jira/browse/KYLIN-930">KYLIN-930</a></li>
- <li>Error when join two sub-query <a href="https://issues.apache.org/jira/browse/KYLIN-1033">KYLIN-1033</a></li>
- <li>Filter like (A or false) yields wrong result <a href="https://issues.apache.org/jira/browse/KYLIN-1039">KYLIN-1039</a></li>
- <li>Support get MapReduce Job status for ResourceManager HA Env <a href="https://issues.apache.org/jira/browse/KYLIN-1067">KYLIN-1067</a></li>
- <li>Can not send email caused by Build Base Cuboid Data step failed <a href="https://issues.apache.org/jira/browse/KYLIN-1106">KYLIN-1106</a></li>
- <li>ResourceTool download/upload does not work in binary package <a href="https://issues.apache.org/jira/browse/KYLIN-1121">KYLIN-1121</a></li>
- <li>Kylinâs sample cube âkylin_sales_cubeâ couldnât be saved <a href="https://issues.apache.org/jira/browse/KYLIN-1140">KYLIN-1140</a></li>
- <li>Unit test with minicluster doesnât work on 1.x <a href="https://issues.apache.org/jira/browse/KYLIN-1155">KYLIN-1155</a></li>
- <li>Canât parse DateFormat like âYYYYMMDDâ correctly in query <a href="https://issues.apache.org/jira/browse/KYLIN-1216">KYLIN-1216</a></li>
-</ul>
-
-<p><strong>Upgrade</strong> <br />
-We recommend to upgrade to this version for better performance, stability and bug fixes.<br />
-Also to keep up to date with community with latest features and supports.</p>
-
-<p><strong>Support</strong> <br />
-Any issue or question during upgrade, please <br />
-open JIRA to Kylin project: <a href="https://issues.apache.org/jira/browse/KYLIN/">https://issues.apache.org/jira/browse/KYLIN/</a> <br />
-or <br />
-send mail to Apache Kylin dev mailing list: <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a></p>
-
-<p><em>Great thanks to everyone who contributed!</em></p>
-</description>
- <pubDate>Wed, 23 Dec 2015 14:28:00 -0800</pubDate>
- <link>http://kylin.apache.org/blog/2015/12/23/release-v1.2/</link>
- <guid isPermaLink="true">http://kylin.apache.org/blog/2015/12/23/release-v1.2/</guid>
-
-
- <category>blog</category>
-
- </item>
-
- <item>
<title>Apache Kylin v1.2 æ£å¼åå¸</title>
<description><p>Apache Kylin社åºé常é«å
´å®£å¸Apache Kylin v1.2æ£å¼åå¸ï¼è¿æ¯èªé¡ºå©æ¯ä¸æApache顶级项ç®åç第ä¸ä¸ªåå¸çæ¬ã</p>
@@ -210,6 +181,67 @@ send mail to Apache Kylin dev mailing li
</item>
<item>
+ <title>Apache Kylin v1.2 Release Announcement</title>
+ <description><p>The Apache Kylin community is pleased to announce the release of Apache Kylin v1.2, the first release after graduation.</p>
+
+<p>Apache Kylin is an open source Distributed Analytics Engine designed to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets, original contributed from eBay Inc.</p>
+
+<p>To download Apache Kylin v1.2 source code or binary package: <br />
+please visit the <a href="http://kylin.apache.org/download">download</a> page.</p>
+
+<p>This is a major release which brings more stable, robust and well management version, Apache Kylin community resolved about 44 issues including bug fixes, improvements, and few new features.</p>
+
+<h2 id="change-highlights">Change Highlights</h2>
+
+<p><strong>Kylin Core Improvement</strong></p>
+
+<ul>
+ <li>Support Excel, Power BI and Tableau 9.1 <a href="https://issues.apache.org/jira/browse/KYLIN-596">KYLIN-596</a>,<a href="https://issues.apache.org/jira/browse/KYLIN-1065">KYLIN-1065</a></li>
+ <li>Improve small file management on HDFS <a href="https://issues.apache.org/jira/browse/KYLIN-702">KYLIN-702</a></li>
+ <li>Env shell script enhance for Hive HCatalog <a href="https://issues.apache.org/jira/browse/KYLIN-1081">KYLIN-1081</a>, <a href="https://issues.apache.org/jira/browse/KYLIN-1119">KYLIN-1119</a></li>
+ <li>Dimenion column supports high cardinality over 10 million <a href="https://issues.apache.org/jira/browse/KYLIN-1099">KYLIN-1099</a></li>
+ <li>Enhance job page loading performance <a href="https://issues.apache.org/jira/browse/KYLIN-1154">KYLIN-1154</a></li>
+ <li>Make memory budget per query configurable <a href="https://issues.apache.org/jira/browse/KYLIN-1190">KYLIN-1190</a></li>
+</ul>
+
+<p><strong>Main Bug Fixes</strong></p>
+
+<ul>
+ <li>Save cube issue in edit model <a href="https://issues.apache.org/jira/browse/KYLIN-1168">KYLIN-1168</a></li>
+ <li>Couldnât change a cubeâs name after it be created <a href="https://issues.apache.org/jira/browse/KYLIN-693">KYLIN-693</a></li>
+ <li>Cube list missing under project <a href="https://issues.apache.org/jira/browse/KYLIN-930">KYLIN-930</a></li>
+ <li>Error when join two sub-query <a href="https://issues.apache.org/jira/browse/KYLIN-1033">KYLIN-1033</a></li>
+ <li>Filter like (A or false) yields wrong result <a href="https://issues.apache.org/jira/browse/KYLIN-1039">KYLIN-1039</a></li>
+ <li>Support get MapReduce Job status for ResourceManager HA Env <a href="https://issues.apache.org/jira/browse/KYLIN-1067">KYLIN-1067</a></li>
+ <li>Can not send email caused by Build Base Cuboid Data step failed <a href="https://issues.apache.org/jira/browse/KYLIN-1106">KYLIN-1106</a></li>
+ <li>ResourceTool download/upload does not work in binary package <a href="https://issues.apache.org/jira/browse/KYLIN-1121">KYLIN-1121</a></li>
+ <li>Kylinâs sample cube âkylin_sales_cubeâ couldnât be saved <a href="https://issues.apache.org/jira/browse/KYLIN-1140">KYLIN-1140</a></li>
+ <li>Unit test with minicluster doesnât work on 1.x <a href="https://issues.apache.org/jira/browse/KYLIN-1155">KYLIN-1155</a></li>
+ <li>Canât parse DateFormat like âYYYYMMDDâ correctly in query <a href="https://issues.apache.org/jira/browse/KYLIN-1216">KYLIN-1216</a></li>
+</ul>
+
+<p><strong>Upgrade</strong> <br />
+We recommend to upgrade to this version for better performance, stability and bug fixes.<br />
+Also to keep up to date with community with latest features and supports.</p>
+
+<p><strong>Support</strong> <br />
+Any issue or question during upgrade, please <br />
+open JIRA to Kylin project: <a href="https://issues.apache.org/jira/browse/KYLIN/">https://issues.apache.org/jira/browse/KYLIN/</a> <br />
+or <br />
+send mail to Apache Kylin dev mailing list: <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a></p>
+
+<p><em>Great thanks to everyone who contributed!</em></p>
+</description>
+ <pubDate>Wed, 23 Dec 2015 14:28:00 -0800</pubDate>
+ <link>http://kylin.apache.org/blog/2015/12/23/release-v1.2/</link>
+ <guid isPermaLink="true">http://kylin.apache.org/blog/2015/12/23/release-v1.2/</guid>
+
+
+ <category>blog</category>
+
+ </item>
+
+ <item>
<title>Apache Kylin v1.1 (incubating) Release Announcement</title>
<description><p>The Apache Kylin community is pleased to announce the release of Apache Kylin v1.1 (incubating).</p>
@@ -652,58 +684,6 @@ So the environment setup is as below:<
<category>blog</category>
-
- </item>
-
- <item>
- <title>Apache Kylin v1.0 (incubating) Release Announcement</title>
- <description><p>The Apache Kylin team is pleased to announce the release of Apache Kylin v1.0 (incubating). Apache Kylin is an open source Distributed Analytics Engine designed to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets.</p>
-
-<p>To download Apache Kylin v1.0 (incubating) visit the <a href="http://kylin.apache.org/download">download</a> page.</p>
-
-<p>This is a major release which brings more stable, robust and well management version, Apache Kylin team resolved about 47 issues including bug fixes, improvements, and few new features.</p>
-
-<h2 id="change-highlights">Change Highlights</h2>
-
-<p><strong>Kylin Core Improvement</strong></p>
-
-<ul>
- <li>Dynamic Data Model has been added to supporting adding or removing column in data model without rebuild cube from the beginning <a href="https://issues.apache.org/jira/browse/KYLIN-867">KYLIN-867</a></li>
- <li>Upgraded Apache Calcite to 1.3 for more bug fixes and new SQL functions <a href="https://issues.apache.org/jira/browse/KYLIN-881">KYLIN-881</a></li>
- <li>Cleanup job enhanced to make sure thereâs no garbage files left in OS and HDFS/HBase after job build <a href="https://issues.apache.org/jira/browse/KYLIN-926">KYLIN-926</a></li>
- <li>Added setting option for Hive intermediate tables created by Kylin <a href="https://issues.apache.org/jira/browse/KYLIN-883">KYLIN-883</a></li>
- <li>HBase coprocessor enhanced to imrpove query performance <a href="https://issues.apache.org/jira/browse/KYLIN-857">KYLIN-857</a></li>
- <li>Kylin System Dashboard for usage, storage, performance <a href="https://issues.apache.org/jira/browse/KYLIN-792">KYLIN-792</a></li>
-</ul>
-
-<p><strong>Main Bug Fixes</strong></p>
-
-<ul>
- <li>Canât get source record size, especially when using other Hadoop distribution rather than HDP, like Cloudera and MapR <a href="https://issues.apache.org/jira/browse/KYLIN-404">KYLIN-404</a></li>
- <li>Front-end cache cleanup issue <a href="https://issues.apache.org/jira/browse/KYLIN-757">KYLIN-757</a></li>
- <li>Useless hive intermediate tables and HBase tables will be dropped after cube build/merge <a href="https://issues.apache.org/jira/browse/KYLIN-805">KYLIN-805</a></li>
- <li>Support More than one HDFS files of lookup table, especially for large lookup table <a href="https://issues.apache.org/jira/browse/KYLIN-889">KYLIN-889</a></li>
- <li>JDBC driver bug fixes <a href="https://issues.apache.org/jira/browse/KYLIN-945">KYLIN-945</a>, <a href="https://issues.apache.org/jira/browse/KYLIN-626">KYLIN-626</a></li>
- <li>UI Bug fixes <a href="https://issues.apache.org/jira/browse/KYLIN-946">KYLIN-946</a>, <a href="https://issues.apache.org/jira/browse/KYLIN-935">KYLIN-935</a></li>
-</ul>
-
-<p><strong>Zeppelin Integration</strong></p>
-
-<p><a href="http://zeppelin.incubator.apache.org/">Apache Zeppelin</a> is a web-based notebook that enables interactive data analytics. The Apache Kylin team has contributed Kylin Interpreter which enables Zeppelin interaction with Kylin from notebook using ANSI SQL, this interpreter could be found from Zeppelin master code repo <a href="https://github.com/apache/incubator-zeppelin/tree/master/kylin">here</a>.</p>
-
-<p><strong>Upgrade</strong></p>
-
-<p>We recommend to upgrade to this version from v0.7.x or even more early version for better performance, stablility and clear one (most of the intermediate files will be cleaned up automatically). Also to keep up to date with community with latest features and supports.<br />
-Any issue or question during upgrade, please send to Apache Kylin dev mailing list: <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a></p>
-
-<p><em>Great thanks to everyone who contributed!</em></p>
-</description>
- <pubDate>Sun, 06 Sep 2015 10:28:00 -0700</pubDate>
- <link>http://kylin.apache.org/blog/2015/09/06/release-v1.0-incubating/</link>
- <guid isPermaLink="true">http://kylin.apache.org/blog/2015/09/06/release-v1.0-incubating/</guid>
-
-
- <category>blog</category>
</item>