You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by uc...@apache.org on 2015/05/15 16:10:51 UTC
[52/52] [abbrv] flink-web git commit: Address PR comments
Address PR comments
Project: http://git-wip-us.apache.org/repos/asf/flink-web/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink-web/commit/5e1410e5
Tree: http://git-wip-us.apache.org/repos/asf/flink-web/tree/5e1410e5
Diff: http://git-wip-us.apache.org/repos/asf/flink-web/diff/5e1410e5
Branch: refs/heads/asf-site
Commit: 5e1410e56ec340777bc579c5b5e97afc56d5416d
Parents: 1f533de
Author: Ufuk Celebi <uc...@apache.org>
Authored: Wed May 13 17:55:40 2015 +0200
Committer: Ufuk Celebi <uc...@apache.org>
Committed: Fri May 15 13:31:06 2015 +0200
----------------------------------------------------------------------
_config.yml | 2 +
_includes/navbar.html | 34 +-
_layouts/plain.html | 6 +-
_layouts/post.html | 32 +-
.../2015-05-11-Juggling-with-Bits-and-Bytes.md | 114 +-
blog/index.html | 7 +-
content/blog/feed.xml | 2797 ++++++++
content/blog/index.html | 647 ++
content/blog/page2/index.html | 641 ++
content/blog/page3/index.html | 681 ++
content/coding-guidelines.html | 213 +
content/community.html | 442 ++
content/css/flink.css | 153 +
content/css/syntax.css | 60 +
content/downloads.html | 323 +
content/faq.html | 199 +
content/favicon.ico | Bin 0 -> 1150 bytes
content/features.html | 259 +
content/how-to-contribute.html | 358 +
content/img/assets/WhatIsFlink.png | Bin 0 -> 248892 bytes
content/img/assets/grep.png | Bin 0 -> 121239 bytes
content/img/assets/hadoop-img.png | Bin 0 -> 67841 bytes
content/img/assets/optimizer-visual.png | Bin 0 -> 204452 bytes
content/img/assets/pagerank.pdf | Bin 0 -> 21052 bytes
content/img/assets/pagerank.png | Bin 0 -> 135851 bytes
content/img/blog/data-serialization.png | Bin 0 -> 395455 bytes
content/img/blog/flinkSer-int-gc.png | Bin 0 -> 12405 bytes
content/img/blog/flinkSer-int-mem.png | Bin 0 -> 15689 bytes
content/img/blog/kryoSer-int-gc.png | Bin 0 -> 12018 bytes
content/img/blog/kryoSer-int-mem.png | Bin 0 -> 16747 bytes
content/img/blog/memory-alloc.png | Bin 0 -> 299878 bytes
content/img/blog/memory-mgmt.png | Bin 0 -> 360975 bytes
content/img/blog/objHeap-int-gc.png | Bin 0 -> 22921 bytes
content/img/blog/objHeap-int-mem.png | Bin 0 -> 17259 bytes
content/img/blog/sort-benchmark.png | Bin 0 -> 73111 bytes
content/img/blog/sorting-binary-data-1.png | Bin 0 -> 251023 bytes
content/img/blog/sorting-binary-data-2.png | Bin 0 -> 335940 bytes
content/img/blog/sorting-binary-data-3.png | Bin 0 -> 278178 bytes
content/img/flink-stack-small.png | Bin 0 -> 55877 bytes
content/img/flink-stack.png | Bin 0 -> 87387 bytes
content/img/logo.zip | Bin 0 -> 7708436 bytes
.../logo/png/100/flink_squirrel_100_black.png | Bin 0 -> 7516 bytes
.../logo/png/100/flink_squirrel_100_color.png | Bin 0 -> 16446 bytes
.../logo/png/100/flink_squirrel_100_white.png | Bin 0 -> 6096 bytes
content/img/logo/png/1000/flink1000_black.png | Bin 0 -> 79026 bytes
.../img/logo/png/1000/flink1000_color_black.png | Bin 0 -> 192934 bytes
.../img/logo/png/1000/flink1000_color_white.png | Bin 0 -> 192809 bytes
content/img/logo/png/1000/flink1000_white.png | Bin 0 -> 69039 bytes
.../img/logo/png/1000/flink_squirrel_1000.png | Bin 0 -> 258648 bytes
.../logo/png/1000/flink_squirrel_black_1000.png | Bin 0 -> 105074 bytes
.../logo/png/1000/flink_squirrel_white_1000.png | Bin 0 -> 91158 bytes
content/img/logo/png/200/flink2_200_black.png | Bin 0 -> 8958 bytes
.../img/logo/png/200/flink2_200_color_black.png | Bin 0 -> 18470 bytes
.../img/logo/png/200/flink2_200_color_white.png | Bin 0 -> 18377 bytes
content/img/logo/png/200/flink2_200_white.png | Bin 0 -> 7180 bytes
.../logo/png/200/flink_squirrel_200_black.png | Bin 0 -> 16353 bytes
.../logo/png/200/flink_squirrel_200_color.png | Bin 0 -> 40591 bytes
.../logo/png/200/flink_squirrel_200_white.png | Bin 0 -> 13315 bytes
content/img/logo/png/50/black_50.png | Bin 0 -> 3487 bytes
content/img/logo/png/50/color_50.png | Bin 0 -> 6182 bytes
content/img/logo/png/50/white_50.png | Bin 0 -> 2457 bytes
content/img/logo/png/500/flink2_500_black.png | Bin 0 -> 25677 bytes
.../img/logo/png/500/flink2_500_color_black.png | Bin 0 -> 59849 bytes
.../img/logo/png/500/flink2_500_color_white.png | Bin 0 -> 59550 bytes
content/img/logo/png/500/flink2_500_white.png | Bin 0 -> 20590 bytes
content/img/logo/png/500/flink500_black.png | Bin 0 -> 36289 bytes
.../img/logo/png/500/flink500_color_black.png | Bin 0 -> 89112 bytes
.../img/logo/png/500/flink500_color_white.png | Bin 0 -> 89131 bytes
content/img/logo/png/500/flink500_white.png | Bin 0 -> 30192 bytes
content/img/logo/png/500/flink_3_500.png | Bin 0 -> 14804 bytes
content/img/logo/png/500/flink_squirrel_500.png | Bin 0 -> 119199 bytes
.../logo/png/500/flink_squirrel_500_black.png | Bin 0 -> 48815 bytes
.../logo/png/500/flink_squirrel_500_white.png | Bin 0 -> 41430 bytes
content/img/logo/psd/flink1000.psd | Bin 0 -> 1966600 bytes
content/img/logo/psd/flink50.psd | Bin 0 -> 407654 bytes
content/img/logo/psd/flink5000.psd | Bin 0 -> 1528938 bytes
content/img/logo/psd/flink_3_500.psd | Bin 0 -> 844305 bytes
content/img/logo/psd/flink_squirrel.psd | Bin 0 -> 1970196 bytes
content/img/logo/psd/flink_squirrel_1000.psd | Bin 0 -> 2283989 bytes
content/img/logo/rsz_1flink-stack.png | Bin 0 -> 55877 bytes
content/img/logo/svg/black_outline.svg | 473 ++
content/img/logo/svg/color_black.svg | 1561 +++++
content/img/logo/svg/color_white.svg | 1563 +++++
content/img/logo/svg/flink_logos.svg | 6425 ++++++++++++++++++
content/img/logo/svg/flink_logotypes.svg | 3835 +++++++++++
content/img/logo/svg/white_filled.svg | 231 +
content/img/managed-state.png | Bin 0 -> 22870 bytes
content/img/navbar-brand-logo.jpg | Bin 0 -> 18470 bytes
content/img/runtime.png | Bin 0 -> 20205 bytes
content/index.html | 297 +
content/js/codetabs.js | 121 +
content/material.html | 288 +
content/news/2012/08/21/release02.html | 192 +
content/news/2012/10/15/icde2013.html | 193 +
content/news/2012/11/12/btw2013demo.html | 191 +
content/news/2012/11/21/previewICDE2013.html | 186 +
content/news/2013/03/27/www-demo-paper.html | 190 +
content/news/2013/10/21/cikm2013-paper.html | 224 +
.../2013/12/13/humboldt-innovation-award.html | 195 +
.../2014/01/10/stratosphere-hadoop-summit.html | 185 +
.../news/2014/01/12/0.4-migration-guide.html | 261 +
.../2014/01/13/stratosphere-release-0.4.html | 254 +
.../26/optimizer_plan_visualization_tool.html | 203 +
content/news/2014/01/28/querying_mongodb.html | 276 +
.../18/amazon-elastic-mapreduce-cloud-yarn.html | 365 +
...stratosphere-google-summer-of-code-2014.html | 192 +
.../16/stratosphere-goes-apache-incubator.html | 187 +
content/news/2014/05/31/release-0.5.html | 264 +
content/news/2014/08/26/release-0.6.html | 258 +
content/news/2014/09/26/release-0.6.1.html | 189 +
content/news/2014/10/03/upcoming_events.html | 274 +
content/news/2014/11/04/release-0.7.0.html | 247 +
.../news/2014/11/18/hadoop-compatibility.html | 262 +
content/news/2015/01/06/december-in-flink.html | 237 +
content/news/2015/01/21/release-0.8.html | 261 +
content/news/2015/02/04/january-in-flink.html | 228 +
content/news/2015/02/09/streaming-example.html | 820 +++
.../news/2015/03/02/february-2015-in-flink.html | 291 +
.../peeking-into-Apache-Flinks-Engine-Room.html | 365 +
content/news/2015/04/07/march-in-flink.html | 254 +
.../2015/04/13/release-0.9.0-milestone1.html | 427 ++
.../05/11/Juggling-with-Bits-and-Bytes.html | 366 +
content/privacy-policy.html | 191 +
content/project.html | 173 +
css/flink.css | 35 +-
css/syntax.css | 60 +
favicon.ico | Bin 0 -> 1150 bytes
features.md | 99 +-
img/assets/WhatIsFlink.png | Bin 0 -> 248892 bytes
img/assets/grep.png | Bin 0 -> 121239 bytes
img/assets/hadoop-img.png | Bin 0 -> 67841 bytes
img/assets/optimizer-visual.png | Bin 0 -> 204452 bytes
img/assets/pagerank.pdf | Bin 0 -> 21052 bytes
img/assets/pagerank.png | Bin 0 -> 135851 bytes
img/flink-stack-small.png | Bin 0 -> 55877 bytes
img/flink-stack.png | Bin 44990 -> 87387 bytes
img/logo/png/100/flink_squirrel_100_black.png | Bin 0 -> 7516 bytes
img/logo/png/100/flink_squirrel_100_color.png | Bin 0 -> 16446 bytes
img/logo/png/100/flink_squirrel_100_white.png | Bin 0 -> 6096 bytes
img/logo/png/1000/flink1000_black.png | Bin 0 -> 79026 bytes
img/logo/png/1000/flink1000_color_black.png | Bin 0 -> 192934 bytes
img/logo/png/1000/flink1000_color_white.png | Bin 0 -> 192809 bytes
img/logo/png/1000/flink1000_white.png | Bin 0 -> 69039 bytes
img/logo/png/1000/flink_squirrel_1000.png | Bin 0 -> 258648 bytes
img/logo/png/1000/flink_squirrel_black_1000.png | Bin 0 -> 105074 bytes
img/logo/png/1000/flink_squirrel_white_1000.png | Bin 0 -> 91158 bytes
img/logo/png/200/flink2_200_black.png | Bin 0 -> 8958 bytes
img/logo/png/200/flink2_200_color_black.png | Bin 0 -> 18470 bytes
img/logo/png/200/flink2_200_color_white.png | Bin 0 -> 18377 bytes
img/logo/png/200/flink2_200_white.png | Bin 0 -> 7180 bytes
img/logo/png/200/flink_squirrel_200_black.png | Bin 0 -> 16353 bytes
img/logo/png/200/flink_squirrel_200_color.png | Bin 0 -> 40591 bytes
img/logo/png/200/flink_squirrel_200_white.png | Bin 0 -> 13315 bytes
img/logo/png/50/black_50.png | Bin 0 -> 3487 bytes
img/logo/png/50/color_50.png | Bin 0 -> 6182 bytes
img/logo/png/50/white_50.png | Bin 0 -> 2457 bytes
img/logo/png/500/flink2_500_black.png | Bin 0 -> 25677 bytes
img/logo/png/500/flink2_500_color_black.png | Bin 0 -> 59849 bytes
img/logo/png/500/flink2_500_color_white.png | Bin 0 -> 59550 bytes
img/logo/png/500/flink2_500_white.png | Bin 0 -> 20590 bytes
img/logo/png/500/flink500_black.png | Bin 0 -> 36289 bytes
img/logo/png/500/flink500_color_black.png | Bin 0 -> 89112 bytes
img/logo/png/500/flink500_color_white.png | Bin 0 -> 89131 bytes
img/logo/png/500/flink500_white.png | Bin 0 -> 30192 bytes
img/logo/png/500/flink_3_500.png | Bin 0 -> 14804 bytes
img/logo/png/500/flink_squirrel_500.png | Bin 0 -> 119199 bytes
img/logo/png/500/flink_squirrel_500_black.png | Bin 0 -> 48815 bytes
img/logo/png/500/flink_squirrel_500_white.png | Bin 0 -> 41430 bytes
img/logo/psd/flink1000.psd | Bin 0 -> 1966600 bytes
img/logo/psd/flink50.psd | Bin 0 -> 407654 bytes
img/logo/psd/flink5000.psd | Bin 0 -> 1528938 bytes
img/logo/psd/flink_3_500.psd | Bin 0 -> 844305 bytes
img/logo/psd/flink_squirrel.psd | Bin 0 -> 1970196 bytes
img/logo/psd/flink_squirrel_1000.psd | Bin 0 -> 2283989 bytes
img/logo/rsz_1flink-stack.png | Bin 0 -> 55877 bytes
img/logo/svg/black_outline.svg | 473 ++
img/logo/svg/color_black.svg | 1561 +++++
img/logo/svg/color_white.svg | 1563 +++++
img/logo/svg/flink_logos.svg | 6425 ++++++++++++++++++
img/logo/svg/flink_logotypes.svg | 3835 +++++++++++
img/logo/svg/white_filled.svg | 231 +
index.md | 85 +-
js/codetabs.js | 121 +
183 files changed, 44524 insertions(+), 127 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5e1410e5/_config.yml
----------------------------------------------------------------------
diff --git a/_config.yml b/_config.yml
index ea72c5f..c50bd97 100644
--- a/_config.yml
+++ b/_config.yml
@@ -52,6 +52,8 @@ twitter: "https://twitter.com/apacheflink"
github: "https://github.com/apache/flink"
wiki: "https://cwiki.apache.org/confluence/display/FLINK/Apache+Flink+Home"
+twitter-handle: "ApacheFlink"
+
#baseurl: /incubator-flink-website
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5e1410e5/_includes/navbar.html
----------------------------------------------------------------------
diff --git a/_includes/navbar.html b/_includes/navbar.html
index cd90cd0..e4e00a9 100644
--- a/_includes/navbar.html
+++ b/_includes/navbar.html
@@ -9,7 +9,7 @@
<span class="icon-bar"></span>
</button>
<div class="navbar-logo">
- <a href="{{ site.baseurl }}/"><img alt="Apache Flink" src="{{ site.baseurl }}/img/navbar-brand-logo.jpg"></a>
+ <a href="{{ site.baseurl }}/"><img alt="Apache Flink" src="{{ site.baseurl }}/img/navbar-brand-logo.jpg" width="78px" height="40px"></a>
</div>
</div><!-- /.navbar-header -->
@@ -17,13 +17,13 @@
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav">
<!-- Overview -->
- <li{% if page.url == '/index.html' %} class="active"{% endif %}><a href="{{ site.baseurl }}/index.html">Overview</span></a></li>
+ <li{% if page.url == '/index.html' %} class="active"{% endif %}><a href="{{ site.baseurl }}/index.html">Overview</a></li>
<!-- Features -->
- <li{% if page.url == '/features.html' %} class="hidden-sm active"{% endif %}><a href="{{ site.baseurl }}/features.html">Features</span></a></li>
+ <li{% if page.url == '/features.html' %} class="hidden-sm active"{% endif %}><a href="{{ site.baseurl }}/features.html">Features</a></li>
<!-- Downloads -->
- <li{% if page.url == '/downloads.html' %} class="active"{% endif %}><a href="{{ site.baseurl }}/downloads.html">Downloads</span></a></li>
+ <li{% if page.url == '/downloads.html' %} class="active"{% endif %}><a href="{{ site.baseurl }}/downloads.html">Downloads</a></li>
<!-- Documentation -->
<li class="dropdown">
@@ -56,18 +56,10 @@
<!-- Blog -->
<li{% if page.url contains '/blog/' or page.url contains '/news/' %} class="active"{% endif %}><a href="{{ site.baseurl }}/blog/">Blog</a></li>
- <li class="dropdown{% if page.url == '/material.html' or page.url == '/community.html' or page.url == '/how-to-contribute.html' or page.url == '/coding-guidelines.html'%} active{% endif %}">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Project <span class="caret"></span></a>
- <ul class="dropdown-menu" role="menu">
- <!-- Project -->
- <li role="presentation" class="dropdown-header"><strong>Project</strong></li>
- <li><a href="{{ site.baseurl }}/material.html">Material</a></li>
- <li><a href="{{ site.twitter }}"><small><span class="glyphicon glyphicon-new-window"></span></small> Twitter</a></li>
- <li><a href="{{ site.github }}"><small><span class="glyphicon glyphicon-new-window"></span></small> GitHub</a></li>
- <li><a href="{{ site.wiki }}"><small><span class="glyphicon glyphicon-new-window"></span></small> Wiki</a></li>
-
+ <li class="dropdown{% if page.url == '/community.html' or page.url == '/how-to-contribute.html' or page.url == '/coding-guidelines.html' %} active{% endif %}">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Community <span class="caret"></span></a>
+ <ul class="dropdown-menu" role="menu">
<!-- Community -->
- <li class="divider"></li>
<li role="presentation" class="dropdown-header"><strong>Community</strong></li>
<li><a href="{{ site.baseurl }}/community.html#mailing-lists">Mailing Lists</a></li>
<li><a href="{{ site.baseurl }}/community.html#irc">IRC</a></li>
@@ -82,6 +74,18 @@
<li><a href="{{ site.baseurl }}/coding-guidelines.html">Coding Guidelines</a></li>
</ul>
</li>
+
+ <li class="dropdown{% if page.url == '/material.html' %} active{% endif %}">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Project <span class="caret"></span></a>
+ <ul class="dropdown-menu" role="menu">
+ <!-- Project -->
+ <li role="presentation" class="dropdown-header"><strong>Project</strong></li>
+ <li><a href="{{ site.baseurl }}/material.html">Material</a></li>
+ <li><a href="{{ site.twitter }}"><small><span class="glyphicon glyphicon-new-window"></span></small> Twitter</a></li>
+ <li><a href="{{ site.github }}"><small><span class="glyphicon glyphicon-new-window"></span></small> GitHub</a></li>
+ <li><a href="{{ site.wiki }}"><small><span class="glyphicon glyphicon-new-window"></span></small> Wiki</a></li>
+ </ul>
+ </li>
</ul>
</div><!-- /.navbar-collapse -->
</div><!-- /.container -->
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5e1410e5/_layouts/plain.html
----------------------------------------------------------------------
diff --git a/_layouts/plain.html b/_layouts/plain.html
index 15ab458..6f48be0 100644
--- a/_layouts/plain.html
+++ b/_layouts/plain.html
@@ -2,8 +2,10 @@
layout: base
---
<div class="row">
- <div class="col-sm-12">
- <h1>{{ page.title }}</h1>
+ <div class="col-sm-8 col-sm-offset-2">
+ <div class="row">
+ <div class="col-sm-12"><h1>{{ page.title }}</h1></div>
+ </div>
{{ content }}
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5e1410e5/_layouts/post.html
----------------------------------------------------------------------
diff --git a/_layouts/post.html b/_layouts/post.html
index 895221c..09221f5 100644
--- a/_layouts/post.html
+++ b/_layouts/post.html
@@ -1,8 +1,32 @@
---
-layout: plain
+layout: base
---
-<article>
- <p>{{ page.date | date_to_string }}{% if page.author %} by {{ page.author }}{% if page.author-twitter %} (<a href="https://twitter.com/{{ page.author-twitter }}">@{{ page.author-twitter }}</a>){% endif %}{% endif %}</p>
+
+<div class="row">
+ <div class="col-sm-8 col-sm-offset-2">
+ <div class="row">
+ <h1>{{ page.title }}</h1>
+
+ <article>
+ <p>{{ page.date | date_to_string }}{% if page.author %} by {{ page.author }}{% if page.author-twitter %} (<a href="https://twitter.com/{{ page.author-twitter }}">@{{ page.author-twitter }}</a>){% endif %}{% endif %}</p>
{{ content }}
-</article>
\ No newline at end of file
+ </article>
+ </div>
+
+ <div class="row">
+ <div id="disqus_thread"></div>
+ <script type="text/javascript">
+ /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
+ var disqus_shortname = 'stratosphere-eu'; // required: replace example with your forum shortname
+
+ /* * * DON'T EDIT BELOW THIS LINE * * */
+ (function() {
+ var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
+ dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
+ (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
+ })();
+ </script>
+ </div>
+ </div>
+</div>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5e1410e5/_posts/2015-05-11-Juggling-with-Bits-and-Bytes.md
----------------------------------------------------------------------
diff --git a/_posts/2015-05-11-Juggling-with-Bits-and-Bytes.md b/_posts/2015-05-11-Juggling-with-Bits-and-Bytes.md
index f0a6f87..7ac5013 100644
--- a/_posts/2015-05-11-Juggling-with-Bits-and-Bytes.md
+++ b/_posts/2015-05-11-Juggling-with-Bits-and-Bytes.md
@@ -2,22 +2,26 @@
layout: post
title: "Juggling with Bits and Bytes"
date: 2015-05-11 10:00:00
+author: "Fabian Hüske"
+author-twitter: "fhueske"
+excerpt: "<p>Nowadays, a lot of open-source systems for analyzing large data sets are implemented in Java or other JVM-based programming languages. The most well-known example is Apache Hadoop, but also newer frameworks such as Apache Spark, Apache Drill, and also Apache Flink run on JVMs. A common challenge that JVM-based data analysis engines face is to store large amounts of data in memory - both for caching and for efficient processing such as sorting and joining of data. Managing the JVM memory well makes the difference between a system that is hard to configure and has unpredictable reliability and performance and a system that behaves robustly with few configuration knobs.</p>
+
+<p>In this blog post we discuss how Apache Flink manages memory, talk about its custom data de/serialization stack, and show how it operates on binary data.</p>"
categories: news
---
-###How Apache Flink operates on binary data
+## How Apache Flink operates on binary data
Nowadays, a lot of open-source systems for analyzing large data sets are implemented in Java or other JVM-based programming languages. The most well-known example is Apache Hadoop, but also newer frameworks such as Apache Spark, Apache Drill, and also Apache Flink run on JVMs. A common challenge that JVM-based data analysis engines face is to store large amounts of data in memory - both for caching and for efficient processing such as sorting and joining of data. Managing the JVM memory well makes the difference between a system that is hard to configure and has unpredictable reliability and performance and a system that behaves robustly with few configuration knobs.
In this blog post we discuss how Apache Flink manages memory, talk about its custom data de/serialization stack, and show how it operates on binary data.
-###Data Objects? Let’s put them on the heap!
+## Data Objects? Let’s put them on the heap!
The most straight-forward approach to process lots of data in a JVM is to put it as objects on the heap and operate on these objects. Caching a data set as objects would be as simple as maintaining a list containing an object for each record. An in-memory sort would simply sort the list of objects.
However, this approach has a few notable drawbacks. First of all it is not trivial to watch and control heap memory usage when a lot of objects are created and invalidated constantly. Memory overallocation instantly kills the JVM with an `OutOfMemoryError`. Another aspect is garbage collection on multi-GB JVMs which are flooded with new objects. The overhead of garbage collection in such environments can easily reach 50% and more. Finally, Java objects come with a certain space overhead depending on the JVM and platform. For data sets with many small objects this can significantly reduce the effectively usable amount of memory. Given proficient system design and careful, use-case specific system parameter tuning, heap memory usage can be more or less controlled and `OutOfMemoryErrors` avoided. However, such setups are rather fragile especially if data characteristics or the execution environment change.
-
-###What is Flink doing about that?
+## What is Flink doing about that?
Apache Flink has its roots at a research project which aimed to combine the best technologies of MapReduce-based systems and parallel database systems. Coming from this background, Flink has always had its own way of processing data in-memory. Instead of putting lots of objects on the heap, Flink serializes objects into a fixed number of pre-allocated memory segments. Its DBMS-style sort and join algorithms operate as much as possible on this binary data to keep the de/serialization overhead at a minimum. If more data needs to be processed than can be kept in memory, Flink’s operators partially spill data to disk. In fact, a lot of Flink’s internal implementations look more like C/C++ rather than common Java. The following figure gives a high-level overview of how Flink stores data serialized in memory segments and spills to disk if necessary.
@@ -37,7 +41,7 @@ These properties of active memory management are very desirable in a data proces
In the following we discuss in detail how Flink allocates memory, de/serializes objects, and operates on binary data. We will also show some performance numbers comparing processing objects on the heap and operating on binary data.
-###How does Flink allocate memory?
+## How does Flink allocate memory?
A Flink worker, called TaskManager, is composed of several internal components such as an actor system for coordination with the Flink master, an IOManager that takes care of spilling data to disk and reading it back, and a MemoryManager that coordinates memory usage. In the context of this blog post, the MemoryManager is of most interest.
@@ -49,7 +53,7 @@ MemorySegments are allocated once at TaskManager start-up time and are destroyed
<img src="{{ site.baseurl }}/img/blog/memory-alloc.png" style="width:60%;margin:15px">
</center>
-###How does Flink serialize objects?
+## How does Flink serialize objects?
The Java ecosystem offers several libraries to convert objects into a binary representation and back. Common alternatives are standard Java serialization, [Kryo](https://github.com/EsotericSoftware/kryo), [Apache Avro](http://avro.apache.org/), [Apache Thrift](http://thrift.apache.org/), or Google’s [Protobuf](https://github.com/google/protobuf). Flink includes its own custom serialization framework in order to control the binary representation of data. This is important because operating on binary data such as comparing or even manipulating binary data requires exact knowledge of the serialization layout. Further, configuring the serialization layout with respect to operations that are performed on binary data can yield a significant performance boost. Flink’s serialization stack also leverages the fact, that the type of the objects which are going through de/serialization are exactly known before a program is executed.
@@ -80,8 +84,7 @@ public class Person {
Flink’s type system can be easily extended by providing custom TypeInformations, Serializers, and Comparators to improve the performance of serializing and comparing custom data types.
-
-###How does Flink operate on binary data?
+## How does Flink operate on binary data?
Similar to many other data processing APIs (including SQL), Flink’s APIs provide transformations to group, sort, and join data sets. These transformations operate on potentially very large data sets. Relational database systems feature very efficient algorithms for these purposes since several decades including external merge-sort, merge-join, and hybrid hash-join. Flink builds on this technology, but generalizes it to handle arbitrary objects using its custom serialization and comparison stack. In the following, we show how Flink operates with binary data by the example of Flink’s in-memory sort algorithm.
@@ -106,8 +109,7 @@ The sort buffer compares two elements by comparing their binary fix-length sort
The sorted data is returned by sequentially reading the pointer region of the sort buffer, skipping the sort keys and following the sorted pointers to the actual data. This data is either deserialized and returned as objects or the binary representation is copied and written to disk in case of an external merge-sort (see this [blog post on joins in Flink](http://flink.apache.org/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html)).
-
-###Show me numbers!
+## Show me numbers!
So, what does operating on binary data mean for performance? We’ll run a benchmark that sorts 10 million `Tuple2<Integer, String>` objects to find out. The values of the Integer field are sampled from a uniform distribution. The String field values have a length of 12 characters and are sampled from a long-tail distribution. The input data is provided by an iterator that returns a mutable object, i.e., the same tuple object instance is returned with different field values. Flink uses this technique when reading data from memory, network, or disk to avoid unnecessary object instantiations. The benchmarks are run in a JVM with 900 MB heap size which is approximately the required amount of memory to store and sort 10 million tuple objects on the heap without dying of an `OutOfMemoryError`. We sort the tuples on the Integer field and on the String field using three sorting methods:
@@ -123,70 +125,66 @@ All sort methods are implemented using a single thread. The reported times are a
We see that Flink’s sort on binary data using its own serializers significantly outperforms the other two methods. Comparing to the object-on-heap method, we see that loading the data into memory is much faster. Since we actually collect the objects, there is no opportunity to reuse the object instances, but have to re-create every tuple. This is less efficient than Flink’s serializers (or Kryo serialization). On the other hand, reading objects from the heap comes for free compared to deserialization. In our benchmark, object cloning was more expensive than serialization and deserialization combined. Looking at the sorting time, we see that also sorting on the binary representation is faster than Java’s collection sort. Sorting data that was serialized using Kryo without binary sort key, is much slower than both other methods. This is due to the heavy deserialization overhead. Sorting the tuples on their String field is faster than sorting on the Integer field due to the long-
tailed value distribution which significantly reduces the number of pair-wise comparisons. To get a better feeling of what is happening during sorting we monitored the executing JVM using VisualVM. The following screenshots show heap memory usage, garbage collection activity and CPU usage over the execution of 10 runs.
-<table>
-<tr>
- <td> </td>
- <th><center><b>Garbage Collection</b></center></td>
- <th><center><b>Memory Usage</b></center></td>
-</tr>
-<tr>
- <td><b>Object-on-Heap (int)</b></td>
- <td><img src="{{ site.baseurl }}/img/blog/objHeap-int-gc.png" style="width:80%;margin:15px"></td>
- <td><img src="{{ site.baseurl }}/img/blog/objHeap-int-mem.png" style="width:80%;margin:15px"></td>
-</tr>
-<tr>
- <td><b>Flink-Serialized (int)</b></td>
- <td><img src="{{ site.baseurl }}/img/blog/flinkSer-int-gc.png" style="width:80%;margin:15px"></td>
- <td><img src="{{ site.baseurl }}/img/blog/flinkSer-int-mem.png" style="width:80%;margin:15px"></td>
-</tr>
-<tr>
- <td><b>Kryo-Serialized (int)</b></td>
- <td><img src="{{ site.baseurl }}/img/blog/kryoSer-int-gc.png" style="width:80%;margin:15px"></td>
- <td><img src="{{ site.baseurl }}/img/blog/kryoSer-int-mem.png" style="width:80%;margin:15px"></td>
-</tr>
+<table width="100%">
+ <tr>
+ <th></th>
+ <th><center><b>Garbage Collection</b></center></th>
+ <th><center><b>Memory Usage</b></center></th>
+ </tr>
+ <tr>
+ <td><b>Object-on-Heap (int)</b></td>
+ <td><img src="{{ site.baseurl }}/img/blog/objHeap-int-gc.png" style="width:80%"></td>
+ <td><img src="{{ site.baseurl }}/img/blog/objHeap-int-mem.png" style="width:80%"></td>
+ </tr>
+ <tr>
+ <td><b>Flink-Serialized (int)</b></td>
+ <td><img src="{{ site.baseurl }}/img/blog/flinkSer-int-gc.png" style="width:80%"></td>
+ <td><img src="{{ site.baseurl }}/img/blog/flinkSer-int-mem.png" style="width:80%"></td>
+ </tr>
+ <tr>
+ <td><b>Kryo-Serialized (int)</b></td>
+ <td><img src="{{ site.baseurl }}/img/blog/kryoSer-int-gc.png" style="width:80%"></td>
+ <td><img src="{{ site.baseurl }}/img/blog/kryoSer-int-mem.png" style="width:80%"></td>
+ </tr>
</table>
The experiments run single-threaded on an 8-core machine, so full utilization of one core only corresponds to a 12.5% overall utilization. The screenshots show that operating on binary data significantly reduces garbage collection activity. For the object-on-heap approach, the garbage collector runs in very short intervals while filling the sort buffer and causes a lot of CPU usage even for a single processing thread (sorting itself does not trigger the garbage collector). The JVM garbage collects with multiple parallel threads, explaining the high overall CPU utilization. On the other hand, the methods that operate on serialized data rarely trigger the garbage collector and have a much lower CPU utilization. In fact the garbage collector does not run at all if the tuples are sorted on the Integer field using the flink-serialized method because no objects need to be deserialized for pair-wise comparisons. The kryo-serialized method requires slightly more garbage collection since it
does not use binary sort keys and deserializes two objects for each comparison.
The memory usage charts shows that the flink-serialized and kryo-serialized constantly occupy a high amount of memory (plus some objects for operation). This is due to the pre-allocation of MemorySegments. The actual memory usage is much lower, because the sort buffers are not completely filled. The following table shows the memory consumption of each method. 10 million records result in about 280 MB of binary data (object data plus pointers and sort keys) depending on the used serializer and presence and size of a binary sort key. Comparing this to the memory requirements of the object-on-heap approach we see that operating on binary data can significantly improve memory efficiency. In our benchmark more than twice as much data can be sorted in-memory if serialized into a sort buffer instead of holding it as objects on the heap.
-
<table width="100%">
-<tr><td><b>Occupied Memory</b></td>
- <td><b>Object-on-Heap</b></td>
- <td><b>Flink-Serialized</b></td>
- <td><b>Kryo-Serialized</b></td>
-</tr>
-<tr>
- <td><b>Sort on Integer</b></td>
- <td>approx. 700 MB (heap)</td>
- <td>277 MB (sort buffer)</td>
- <td>266 MB (sort buffer)</td>
-</tr>
-<tr>
- <td><b>Sort on String</b></td>
- <td>approx. 700 MB (heap)</td>
- <td>315 MB (sort buffer)</td>
- <td>266 MB (sort buffer)</td>
-</tr>
-</table><br>
+ <tr>
+ <th>Occupied Memory</th>
+ <th>Object-on-Heap</th>
+ <th>Flink-Serialized</th>
+ <th>Kryo-Serialized</th>
+ </tr>
+ <tr>
+ <td><b>Sort on Integer</b></td>
+ <td>approx. 700 MB (heap)</td>
+ <td>277 MB (sort buffer)</td>
+ <td>266 MB (sort buffer)</td>
+ </tr>
+ <tr>
+ <td><b>Sort on String</b></td>
+ <td>approx. 700 MB (heap)</td>
+ <td>315 MB (sort buffer)</td>
+ <td>266 MB (sort buffer)</td>
+ </tr>
+</table>
-To summarize, the experiments verify the previously stated benefits of operating on binary data.
+<br>
+To summarize, the experiments verify the previously stated benefits of operating on binary data.
-###We’re not done yet!
+## We’re not done yet!
Apache Flink features quite a bit of advanced techniques to safely and efficiently process huge amounts of data with limited memory resources. However, there are a few points that could make Flink even more efficient. The Flink community is working on moving the managed memory to off-heap memory. This will allow for smaller JVMs, lower garbage collection overhead, and also easier system configuration. With Flink’s Table API, the semantics of all operations such as aggregations and projections are known (in contrast to black-box user-defined functions). Hence we can generate code for Table API operations that directly operates on binary data. Further improvements include serialization layouts which are tailored towards the operations that are applied on the binary data and code generation for serializers and comparators.
The groundwork (and a lot more) for operating on binary data is done but there is still some room for making Flink even better and faster. If you are crazy about performance and like to juggle with lot of bits and bytes, join the Flink community!
-
-###TL;DR; Give me three things to remember!
+## TL;DR; Give me three things to remember!
* Flink’s active memory management avoids nasty `OutOfMemoryErrors` that kill your JVMs and reduces garbage collection overhead.
* Flink features a highly efficient data de/serialization stack that facilitates operations on binary data and makes more data fit into memory.
* Flink’s DBMS-style operators operate natively on binary data yielding high performance in-memory and destage gracefully to disk if necessary.
-
-
-<br>
-<small>Written by Fabian Hueske ([@fhueske](https://twitter.com/fhueske)).</small>
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5e1410e5/blog/index.html
----------------------------------------------------------------------
diff --git a/blog/index.html b/blog/index.html
index c3553f7..bbe42f6 100644
--- a/blog/index.html
+++ b/blog/index.html
@@ -1,8 +1,13 @@
---
title: Blog
+layout: base
---
<div class="row">
+ <div class="col-sm-12"><h1>Blog</h1></div>
+</div>
+
+<div class="row">
<div class="col-sm-8">
<!-- Blog posts -->
{% for post in paginator.posts %}
@@ -53,7 +58,7 @@ title: Blog
<ul id="markdown-toc">
{% endif %}
- <li><a href="{{ post.url }}">{{ post.title }}</a></li>
+ <li><a href="{{ site.baseurl }}{{ post.url }}">{{ post.title }}</a></li>
{% if forloop.last %}
</ul>