You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by fh...@apache.org on 2019/02/13 15:48:07 UTC
[flink-web] branch asf-site updated (5b1b3e9 -> 990c983)
This is an automated email from the ASF dual-hosted git repository.
fhueske pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git.
from 5b1b3e9 Rebuilt website
new 9c08c5a Add blog post about Blink contribution.
new 990c983 Rebuild website
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
_posts/2019-02-13-blink-flink-merge.md | 142 +++++++++
content/blog/feed.xml | 152 +++++++++
content/blog/index.html | 45 ++-
content/blog/page2/index.html | 49 +--
content/blog/page3/index.html | 49 ++-
content/blog/page4/index.html | 47 ++-
content/blog/page5/index.html | 47 ++-
content/blog/page6/index.html | 47 ++-
content/blog/page7/index.html | 47 ++-
content/blog/page8/index.html | 32 +-
.../blink-flink-tpch.png | Bin 0 -> 483500 bytes
.../blink-spark-tpcds.png | Bin 0 -> 50441 bytes
.../bounded-unbounded.png | Bin
.../sort-performance.png | Bin 0 -> 24953 bytes
.../stream-batch-joins.png | Bin 0 -> 47644 bytes
content/index.html | 8 +-
content/news/2019/02/13/blink-flink-merge.html | 352 +++++++++++++++++++++
content/zh/index.html | 8 +-
.../blink-flink-tpch.png | Bin 0 -> 483500 bytes
.../blink-spark-tpcds.png | Bin 0 -> 50441 bytes
.../bounded-unbounded.png | Bin
.../sort-performance.png | Bin 0 -> 24953 bytes
.../stream-batch-joins.png | Bin 0 -> 47644 bytes
23 files changed, 900 insertions(+), 125 deletions(-)
create mode 100644 _posts/2019-02-13-blink-flink-merge.md
create mode 100644 content/img/blog/unified-batch-streaming-blink/blink-flink-tpch.png
create mode 100644 content/img/blog/unified-batch-streaming-blink/blink-spark-tpcds.png
copy {img => content/img/blog/unified-batch-streaming-blink}/bounded-unbounded.png (100%)
create mode 100644 content/img/blog/unified-batch-streaming-blink/sort-performance.png
create mode 100644 content/img/blog/unified-batch-streaming-blink/stream-batch-joins.png
create mode 100644 content/news/2019/02/13/blink-flink-merge.html
create mode 100644 img/blog/unified-batch-streaming-blink/blink-flink-tpch.png
create mode 100644 img/blog/unified-batch-streaming-blink/blink-spark-tpcds.png
copy img/{ => blog/unified-batch-streaming-blink}/bounded-unbounded.png (100%)
create mode 100644 img/blog/unified-batch-streaming-blink/sort-performance.png
create mode 100644 img/blog/unified-batch-streaming-blink/stream-batch-joins.png
[flink-web] 01/02: Add blog post about Blink contribution.
Posted by fh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fhueske pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 9c08c5a167337e2f01c2070ccbfac758dfd79f56
Author: Fabian Hueske <fh...@apache.org>
AuthorDate: Wed Feb 13 15:35:05 2019 +0100
Add blog post about Blink contribution.
---
_posts/2019-02-13-blink-flink-merge.md | 142 +++++++++++++++++++++
.../blink-flink-tpch.png | Bin 0 -> 483500 bytes
.../blink-spark-tpcds.png | Bin 0 -> 50441 bytes
.../bounded-unbounded.png | Bin 0 -> 176524 bytes
.../sort-performance.png | Bin 0 -> 24953 bytes
.../stream-batch-joins.png | Bin 0 -> 47644 bytes
6 files changed, 142 insertions(+)
diff --git a/_posts/2019-02-13-blink-flink-merge.md b/_posts/2019-02-13-blink-flink-merge.md
new file mode 100644
index 0000000..62cb671
--- /dev/null
+++ b/_posts/2019-02-13-blink-flink-merge.md
@@ -0,0 +1,142 @@
+---
+layout: post
+title: "Batch as a Special Case of Streaming and Alibaba's contribution of Blink"
+date: 2019-02-13T12:00:00.000Z
+authors:
+- stephan:
+ name: "Stephan Ewen"
+ twitter: "stephanewen"
+- fabian:
+ name: "Fabian Hueske"
+ twitter: "fhueske"
+- xiaowei:
+ name: "Xiaowei Jiang"
+ twitter: "XiaoweiJ"
+excerpt: A few weeks ago, Alibaba contributed its Flink-fork 'Blink' back to Apache Flink. In this blog post we discuss how Blink's features will help the Flink community to make a big step towards its vision to build a truly unified system for stream and batch processing.
+categories: news
+---
+
+Last week, we [broke the news](https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E) that Alibaba decided to contribute its Flink-fork, called Blink, back to the Apache Flink project. Why is that a big thing for Flink, what will it mean for users and the community, and how does it fit into Flink’s overall vision? Let's take a step back to understand this better...
+
+## A Unified Approach to Batch and Streaming
+
+Since its early days, Apache Flink has followed the philosophy of taking a unified approach to batch and streaming data processing. The core building block is *"continuous processing of unbounded data streams"*: if you can do that, you can also do offline processing of bounded data sets (batch processing use cases), because these are just streams that happen to end at some point.
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/unified-batch-streaming-blink/bounded-unbounded.png" width="600px" alt="Processing of bounded and unbounded data."/>
+</center>
+<br>
+
+The *"streaming first, with batch as a special case of streaming"* philosophy is supported by various projects (for example [Flink](https://flink.apache.org), [Beam](https://beam.apache.org), etc.) and often been cited as a powerful way to build data applications that [generalize across real-time and offline processing](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101) and to help greatly reduce the complexity of data infrastructures.
+
+### Why are there still batch processors?
+
+However, *"batch is just a special case of streaming"* does not mean that any stream processor is now the right tool for your batch processing use cases - the introduction of stream processors did not render batch processors obsolete:
+
+* Pure stream processing systems are very slow at batch processing workloads. No one would consider it a good idea to use a stream processor that shuffles through message queues to analyze large amounts of available data.
+
+* Unified APIs like [Apache Beam](https://beam.apache.org) often delegate to different runtimes depending on whether the data is continuous/unbounded of fix/bounded. For example, the implementations of the batch and streaming runtime of Google Cloud Dataflow are different, to get the desired performance and resilience in each case.
+
+* *Apache Flink* has a streaming API that can do bounded/unbounded use cases, but still offers a separate DataSet API and runtime stack that is faster for batch use cases.
+
+What is the reason for the above? Where did *"batch is just a special case of streaming"* go wrong?
+
+The answer is simple, nothing is wrong with that paradigm. Unifying batch and streaming in the API is one aspect. One needs to also exploit certain characteristics of the special case “bounded data” in the runtime to competitively handle batch processing use cases. After all, batch processors have been built specifically for that special case.
+
+## Batch on top of a Streaming Runtime
+
+We always believed that it is possible to have a runtime that is state-of-the-art for both stream processing and batch processing use cases at the same time. A runtime that is streaming-first, but can exploit just the right amount of special properties of bounded streams to be as fast for batch use cases as dedicated batch processors. **This is the unique approach that Flink takes.**
+
+Apache Flink has a network stack that supports both [low-latency/high-throughput streaming data exchanges](https://www.ververica.com/flink-forward-berlin/resources/improving-throughput-and-latency-with-flinks-network-stack), as well as high-throughput batch shuffles. Flink has streaming runtime operators for many operations, but also specialized operators for bounded inputs, which get used when you choose the DataSet API or select the batch environment in the Table API.
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/unified-batch-streaming-blink/stream-batch-joins.png" width="500px" alt="Streaming and batch joins."/>
+<br>
+<i>The figure illustrates a streaming join and a batch join. The batch join can read one input fully into a hash table and then probe with the other input. The stream join needs to build tables for both sides, because it needs to continuously process both inputs.
+For data larger than memory, the batch join can partition both data sets into subsets that fit in memory (data hits disk once) whereas the continuous nature of the stream join requires it to always keep all data in the table and repeatedly hit disk on cache misses.</i>
+</center>
+<br>
+
+Because of that, Apache Flink has been actually demonstrating some pretty impressive batch processing performance since its early days. The below benchmark is a bit older, but validated our architectural approach early on.
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/unified-batch-streaming-blink/sort-performance.png" width="500px" alt="Sort performance."/>
+<br>
+<i>Time to sort 3.2 TB (80 GB/node), in seconds<br>
+(<a href="https://www.slideshare.net/FlinkForward/dongwon-kim-a-comparative-performance-evaluation-of-flink" target="blank">Presentation by Dongwon Kim, Flink Forward Berlin 2015</a>.)</i>
+</center>
+<br>
+
+## What is still missing?
+
+To conclude the approach and make Flink's experience on bounded data (batch) state-of-the-art, we need to add a few more enhancements. We believe that these features are key to realizing our vision:
+
+**(1) A truly unified runtime operator stack**: Currently the bounded and unbounded operators have a different network and threading model and don't mix and match. The original reason was that batch operators followed a "pull model" (easier for batch algorithms), while streaming operators followed a "push model" (better latency/throughput characteristics). In a unified stack, continuous streaming operators are the foundation. When operating on bounded data without latency constraints, th [...]
+
+**(2) Exploiting bounded streams to reduce the scope of fault tolerance**: When input data is bounded, it is possible to completely buffer data during shuffles (memory or disk) and replay that data after a failure. This makes recovery more fine grained and thus much more efficient.
+
+**(3) Exploiting bounded stream operator properties for scheduling**: A continuous unbounded streaming application needs (by definition) all operators running at the same time. An application on bounded data can schedule operations after another, depending on how the operators consume data (e.g., first build hash table, then probe hash table). This increases resource efficiency.
+
+**(4) Enabling these special case optimizations for the DataStream API**: Currently, only the Table API (which is unified across bounded/unbounded streams) activates these optimizations when working on bounded data.
+
+**(5) Performance and coverage for SQL**: SQL is the de-facto standard data language, and while it is also being rapidly adopted for continuous streaming use cases, there is absolutely no way past it for bounded/batch use cases. To be competitive with the best batch engines, Flink needs more coverage and performance for the SQL query execution. While the core data-plane in Flink is high performance, the speed of SQL execution ultimately depends a lot also on optimizer rules, a rich set o [...]
+
+## Enter Blink
+
+Blink is a fork of Apache Flink, originally created inside Alibaba to improve Flink’s behavior for internal use cases. Blink adds a series of improvements and integrations (see the [Readme](https://github.com/apache/flink/blob/blink/README.md) for details), many of which fall into the category of improved bounded-data/batch processing and SQL. In fact, of the above list of features for a unified batch/streaming system, Blink implements significant steps forward in all except (4):
+
+**Unified Stream Operators:** Blink extends the Flink streaming runtime operator model to support selectively reading from different inputs, while keeping the push model for very low latency. This control over the inputs helps to now support algorithms like hybrid hash-joins on the same operator and threading model as continuous symmetric joins through RocksDB. These operators also form the basis for future features like [“Side Inputs”](https://cwiki.apache.org/confluence/display/FLINK/F [...]
+
+**Table API & SQL Query Processor:** The SQL query processor is the component that evolved the changed most compared to the latest Flink master branch:
+
+- While Flink currently translates queries either into DataSet or DataStream programs (depending on the characteristics of their inputs), Blink translates queries to a data flow of the aforementioned stream operators.
+
+- Blink adds many more runtime operators for common SQL operations like semi-joins, anti-joins, etc.
+
+- The query planner (optimizer) is still based on Apache Calcite, but has many more optimization rules (incl. join reordering) and uses a proper cost model for planning.
+
+- Stream operators are more aggressively chained.
+
+- The common data structures (sorters, hash tables) and serializers are extended to go even further in operating on binary data and saving serialization overhead. Code generation is used for the row serializers.
+
+**Improved Scheduling and Failure Recovery:** Finally, Blink implements several improvements for task scheduling and fault tolerance. The scheduling strategies use resources better by exploiting how the operators process their input data. The failover strategies recover more fine-grained along the boundaries of persistent shuffles. A failed JobManager can be replaced without restarting a running application.
+
+The changes in Blink result in a big improvement in performance. The below numbers were reported by the developers of Blink to give a rough impression of the performance gains.
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/unified-batch-streaming-blink/blink-flink-tpch.png" width="600px" alt="TPC-H performance of Blink and Flink."/>
+<br>
+<i>Relative performance of Blink versus Flink 1.6.0 in the TPC-H benchmark, query by query.<br>
+The performance improvement is in average 10x.<br>
+<a href="https://www.ververica.com/flink-forward-berlin/resources/unified-engine-for-data-processing-and-ai" target="blank">Presentation by Xiaowei Jiang at Flink Forward Berlin, 2018</a>.)</i>
+</center>
+<br>
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/unified-batch-streaming-blink/blink-spark-tpcds.png" width="600px" alt="TPC-DS performace of Blink and Spark."/>
+<br>
+<i>Performance of Blink versus Spark in the TPC-DS benchmark, aggregate time for all queries together.<br>
+<a href="https://www.bilibili.com/video/av42325467/?p=3" target="blank">Presentation by Xiaowei Jiang at Flink Forward Beijing, 2018</a>.</i>
+</center>
+<br>
+
+## How do we plan to merge Blink and Flink?
+
+Blink’s code is currently available as a [branch](https://github.com/apache/flink/tree/blink) in the Apache Flink repository. It is a challenge to merge a such big amount of changes, while making the merge process as non-disruptive as possible and keeping public APIs as stable as possible.
+
+The community’s [merge plan](https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E) focuses initially on the bounded/batch processing features mentioned above and follows the following approach to ensure a smooth integration:
+
+* To merge Blink’s _SQL/Table API query processor_ enhancements, we exploit the fact that both Flink and Blink have the same APIs: SQL and the Table API.
+Following some restructuring of the Table/SQL module ([FLIP-32](https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions)) we plan to merge the Blink query planner (optimizer) and runtime (operators) as an additional query processor next to the current SQL runtime. Think of it as two different runners for the same APIs.<br>
+Initially, users will be able to select which query processor to use. After a transition period in which the new query processor will be developed to subsume the current query processor, the current processor will most likely be deprecated and eventually dropped. Given that SQL is such a well defined interface, we anticipate that this transition has little friction for users. Mostly a pleasant surprise to have broader SQL feature coverage and a boost in performance.
+
+* To support the merge of Blink’s _enhancements to scheduling and recovery_ for jobs on bounded data, the Flink community is already working on refactoring its current schedule and adding support for [pluggable scheduling and fail-over strategies](https://issues.apache.org/jira/browse/FLINK-10429).<br>
+Once this effort is finished, we can add Blink’s scheduling and recovery strategies as a new scheduling strategy that is used by the new query processor. Eventually, we plan to use the new scheduling strategy also for bounded DataStream programs.
+
+* The extended catalog support, DDL support, as well as support for Hive’s catalog and integrations is currently going through separate design discussions. We plan to leverage existing code here whenever it makes sense.
+
+## Summary
+
+We believe that the data processing stack of the future is based on stream processing: The elegance of stream processing with its ability to model offline processing (batch), real-time data processing, and event-driven applications in the same way, while offering high performance and consistency is simply too compelling.
+
+Exploiting certain properties of bounded data is important for a stream processor to achieve the same performance as dedicated batch processors. While Flink always supported batch processing, the project is taking the next step in building a unified runtime and towards **becoming a stream processor that is competitive with batch processing systems even on their home turf: OLAP SQL.** The contribution of Alibaba’s Blink code helps the Flink community to pick up the speed on this development.
\ No newline at end of file
diff --git a/img/blog/unified-batch-streaming-blink/blink-flink-tpch.png b/img/blog/unified-batch-streaming-blink/blink-flink-tpch.png
new file mode 100644
index 0000000..77e6354
Binary files /dev/null and b/img/blog/unified-batch-streaming-blink/blink-flink-tpch.png differ
diff --git a/img/blog/unified-batch-streaming-blink/blink-spark-tpcds.png b/img/blog/unified-batch-streaming-blink/blink-spark-tpcds.png
new file mode 100644
index 0000000..e6ef8bc
Binary files /dev/null and b/img/blog/unified-batch-streaming-blink/blink-spark-tpcds.png differ
diff --git a/img/blog/unified-batch-streaming-blink/bounded-unbounded.png b/img/blog/unified-batch-streaming-blink/bounded-unbounded.png
new file mode 100644
index 0000000..29dfe8a
Binary files /dev/null and b/img/blog/unified-batch-streaming-blink/bounded-unbounded.png differ
diff --git a/img/blog/unified-batch-streaming-blink/sort-performance.png b/img/blog/unified-batch-streaming-blink/sort-performance.png
new file mode 100644
index 0000000..c1e60ba
Binary files /dev/null and b/img/blog/unified-batch-streaming-blink/sort-performance.png differ
diff --git a/img/blog/unified-batch-streaming-blink/stream-batch-joins.png b/img/blog/unified-batch-streaming-blink/stream-batch-joins.png
new file mode 100644
index 0000000..3591ae7
Binary files /dev/null and b/img/blog/unified-batch-streaming-blink/stream-batch-joins.png differ
[flink-web] 02/02: Rebuild website
Posted by fh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fhueske pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 990c9830b7c5f1b5f9a578d3b5ce97acebfd3e0e
Author: Fabian Hueske <fh...@apache.org>
AuthorDate: Wed Feb 13 16:47:09 2019 +0100
Rebuild website
---
content/blog/feed.xml | 152 +++++++++
content/blog/index.html | 45 ++-
content/blog/page2/index.html | 49 +--
content/blog/page3/index.html | 49 ++-
content/blog/page4/index.html | 47 ++-
content/blog/page5/index.html | 47 ++-
content/blog/page6/index.html | 47 ++-
content/blog/page7/index.html | 47 ++-
content/blog/page8/index.html | 32 +-
.../blink-flink-tpch.png | Bin 0 -> 483500 bytes
.../blink-spark-tpcds.png | Bin 0 -> 50441 bytes
.../bounded-unbounded.png | Bin 0 -> 176524 bytes
.../sort-performance.png | Bin 0 -> 24953 bytes
.../stream-batch-joins.png | Bin 0 -> 47644 bytes
content/index.html | 8 +-
content/news/2019/02/13/blink-flink-merge.html | 352 +++++++++++++++++++++
content/zh/index.html | 8 +-
17 files changed, 758 insertions(+), 125 deletions(-)
diff --git a/content/blog/feed.xml b/content/blog/feed.xml
index 756523d..21bc6b8 100644
--- a/content/blog/feed.xml
+++ b/content/blog/feed.xml
@@ -7,6 +7,158 @@
<atom:link href="https://flink.apache.org/blog/feed.xml" rel="self" type="application/rss+xml" />
<item>
+<title>Batch as a Special Case of Streaming and Alibaba's contribution of Blink</title>
+<description><p>Last week, we <a href="https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E">broke the news</a> that Alibaba decided to contribute its Flink-fork, called Blink, back to the Apache Flink project. Why is that a big thing for Flink, what will it mean for users and the community, and how does it fit into Flink’s overall vision? Let’s take a step back to understand this better…</p>
+
+<h2 id="a-unified-approach-to-batch-and-streaming">A Unified Approach to Batch and Streaming</h2>
+
+<p>Since its early days, Apache Flink has followed the philosophy of taking a unified approach to batch and streaming data processing. The core building block is <em>“continuous processing of unbounded data streams”</em>: if you can do that, you can also do offline processing of bounded data sets (batch processing use cases), because these are just streams that happen to end at some point.</p>
+
+<center>
+<img src="/img/blog/unified-batch-streaming-blink/bounded-unbounded.png" width="600px" alt="Processing of bounded and unbounded data." />
+</center>
+<p><br /></p>
+
+<p>The <em>“streaming first, with batch as a special case of streaming”</em> philosophy is supported by various projects (for example <a href="https://flink.apache.org">Flink</a>, <a href="https://beam.apache.org">Beam</a>, etc.) and often been cited as a powerful way to build data applications that <a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101">generalize across real-time and offl [...]
+
+<h3 id="why-are-there-still-batch-processors">Why are there still batch processors?</h3>
+
+<p>However, <em>“batch is just a special case of streaming”</em> does not mean that any stream processor is now the right tool for your batch processing use cases - the introduction of stream processors did not render batch processors obsolete:</p>
+
+<ul>
+ <li>
+ <p>Pure stream processing systems are very slow at batch processing workloads. No one would consider it a good idea to use a stream processor that shuffles through message queues to analyze large amounts of available data.</p>
+ </li>
+ <li>
+ <p>Unified APIs like <a href="https://beam.apache.org">Apache Beam</a> often delegate to different runtimes depending on whether the data is continuous/unbounded of fix/bounded. For example, the implementations of the batch and streaming runtime of Google Cloud Dataflow are different, to get the desired performance and resilience in each case.</p>
+ </li>
+ <li>
+ <p><em>Apache Flink</em> has a streaming API that can do bounded/unbounded use cases, but still offers a separate DataSet API and runtime stack that is faster for batch use cases.</p>
+ </li>
+</ul>
+
+<p>What is the reason for the above? Where did <em>“batch is just a special case of streaming”</em> go wrong?</p>
+
+<p>The answer is simple, nothing is wrong with that paradigm. Unifying batch and streaming in the API is one aspect. One needs to also exploit certain characteristics of the special case “bounded data” in the runtime to competitively handle batch processing use cases. After all, batch processors have been built specifically for that special case.</p>
+
+<h2 id="batch-on-top-of-a-streaming-runtime">Batch on top of a Streaming Runtime</h2>
+
+<p>We always believed that it is possible to have a runtime that is state-of-the-art for both stream processing and batch processing use cases at the same time. A runtime that is streaming-first, but can exploit just the right amount of special properties of bounded streams to be as fast for batch use cases as dedicated batch processors. <strong>This is the unique approach that Flink takes.</strong></p>
+
+<p>Apache Flink has a network stack that supports both <a href="https://www.ververica.com/flink-forward-berlin/resources/improving-throughput-and-latency-with-flinks-network-stack">low-latency/high-throughput streaming data exchanges</a>, as well as high-throughput batch shuffles. Flink has streaming runtime operators for many operations, but also specialized operators for bounded inputs, which get used when you choose the DataSet API or select the batch envir [...]
+
+<center>
+<img src="/img/blog/unified-batch-streaming-blink/stream-batch-joins.png" width="500px" alt="Streaming and batch joins." />
+<br />
+<i>The figure illustrates a streaming join and a batch join. The batch join can read one input fully into a hash table and then probe with the other input. The stream join needs to build tables for both sides, because it needs to continuously process both inputs.
+For data larger than memory, the batch join can partition both data sets into subsets that fit in memory (data hits disk once) whereas the continuous nature of the stream join requires it to always keep all data in the table and repeatedly hit disk on cache misses.</i>
+</center>
+<p><br /></p>
+
+<p>Because of that, Apache Flink has been actually demonstrating some pretty impressive batch processing performance since its early days. The below benchmark is a bit older, but validated our architectural approach early on.</p>
+
+<center>
+<img src="/img/blog/unified-batch-streaming-blink/sort-performance.png" width="500px" alt="Sort performance." />
+<br />
+<i>Time to sort 3.2 TB (80 GB/node), in seconds<br />
+(<a href="https://www.slideshare.net/FlinkForward/dongwon-kim-a-comparative-performance-evaluation-of-flink" target="blank">Presentation by Dongwon Kim, Flink Forward Berlin 2015</a>.)</i>
+</center>
+<p><br /></p>
+
+<h2 id="what-is-still-missing">What is still missing?</h2>
+
+<p>To conclude the approach and make Flink’s experience on bounded data (batch) state-of-the-art, we need to add a few more enhancements. We believe that these features are key to realizing our vision:</p>
+
+<p><strong>(1) A truly unified runtime operator stack</strong>: Currently the bounded and unbounded operators have a different network and threading model and don’t mix and match. The original reason was that batch operators followed a “pull model” (easier for batch algorithms), while streaming operators followed a “push model” (better latency/throughput characteristics). In a unified stack, continuous streaming operators are the foundation. When operating on bounded da [...]
+
+<p><strong>(2) Exploiting bounded streams to reduce the scope of fault tolerance</strong>: When input data is bounded, it is possible to completely buffer data during shuffles (memory or disk) and replay that data after a failure. This makes recovery more fine grained and thus much more efficient.</p>
+
+<p><strong>(3) Exploiting bounded stream operator properties for scheduling</strong>: A continuous unbounded streaming application needs (by definition) all operators running at the same time. An application on bounded data can schedule operations after another, depending on how the operators consume data (e.g., first build hash table, then probe hash table). This increases resource efficiency.</p>
+
+<p><strong>(4) Enabling these special case optimizations for the DataStream API</strong>: Currently, only the Table API (which is unified across bounded/unbounded streams) activates these optimizations when working on bounded data.</p>
+
+<p><strong>(5) Performance and coverage for SQL</strong>: SQL is the de-facto standard data language, and while it is also being rapidly adopted for continuous streaming use cases, there is absolutely no way past it for bounded/batch use cases. To be competitive with the best batch engines, Flink needs more coverage and performance for the SQL query execution. While the core data-plane in Flink is high performance, the speed of SQL execution ultimately depends a lot als [...]
+
+<h2 id="enter-blink">Enter Blink</h2>
+
+<p>Blink is a fork of Apache Flink, originally created inside Alibaba to improve Flink’s behavior for internal use cases. Blink adds a series of improvements and integrations (see the <a href="https://github.com/apache/flink/blob/blink/README.md">Readme</a> for details), many of which fall into the category of improved bounded-data/batch processing and SQL. In fact, of the above list of features for a unified batch/streaming system, Blink implements significan [...]
+
+<p><strong>Unified Stream Operators:</strong> Blink extends the Flink streaming runtime operator model to support selectively reading from different inputs, while keeping the push model for very low latency. This control over the inputs helps to now support algorithms like hybrid hash-joins on the same operator and threading model as continuous symmetric joins through RocksDB. These operators also form the basis for future features like <a href="https://cwiki.ap [...]
+
+<p><strong>Table API &amp; SQL Query Processor:</strong> The SQL query processor is the component that evolved the changed most compared to the latest Flink master branch:</p>
+
+<ul>
+ <li>
+ <p>While Flink currently translates queries either into DataSet or DataStream programs (depending on the characteristics of their inputs), Blink translates queries to a data flow of the aforementioned stream operators.</p>
+ </li>
+ <li>
+ <p>Blink adds many more runtime operators for common SQL operations like semi-joins, anti-joins, etc.</p>
+ </li>
+ <li>
+ <p>The query planner (optimizer) is still based on Apache Calcite, but has many more optimization rules (incl. join reordering) and uses a proper cost model for planning.</p>
+ </li>
+ <li>
+ <p>Stream operators are more aggressively chained.</p>
+ </li>
+ <li>
+ <p>The common data structures (sorters, hash tables) and serializers are extended to go even further in operating on binary data and saving serialization overhead. Code generation is used for the row serializers.</p>
+ </li>
+</ul>
+
+<p><strong>Improved Scheduling and Failure Recovery:</strong> Finally, Blink implements several improvements for task scheduling and fault tolerance. The scheduling strategies use resources better by exploiting how the operators process their input data. The failover strategies recover more fine-grained along the boundaries of persistent shuffles. A failed JobManager can be replaced without restarting a running application.</p>
+
+<p>The changes in Blink result in a big improvement in performance. The below numbers were reported by the developers of Blink to give a rough impression of the performance gains.</p>
+
+<center>
+<img src="/img/blog/unified-batch-streaming-blink/blink-flink-tpch.png" width="600px" alt="TPC-H performance of Blink and Flink." />
+<br />
+<i>Relative performance of Blink versus Flink 1.6.0 in the TPC-H benchmark, query by query.<br />
+The performance improvement is in average 10x.<br />
+<a href="https://www.ververica.com/flink-forward-berlin/resources/unified-engine-for-data-processing-and-ai" target="blank">Presentation by Xiaowei Jiang at Flink Forward Berlin, 2018</a>.)</i>
+</center>
+<p><br /></p>
+
+<center>
+<img src="/img/blog/unified-batch-streaming-blink/blink-spark-tpcds.png" width="600px" alt="TPC-DS performace of Blink and Spark." />
+<br />
+<i>Performance of Blink versus Spark in the TPC-DS benchmark, aggregate time for all queries together.<br />
+<a href="https://www.bilibili.com/video/av42325467/?p=3" target="blank">Presentation by Xiaowei Jiang at Flink Forward Beijing, 2018</a>.</i>
+</center>
+<p><br /></p>
+
+<h2 id="how-do-we-plan-to-merge-blink-and-flink">How do we plan to merge Blink and Flink?</h2>
+
+<p>Blink’s code is currently available as a <a href="https://github.com/apache/flink/tree/blink">branch</a> in the Apache Flink repository. It is a challenge to merge a such big amount of changes, while making the merge process as non-disruptive as possible and keeping public APIs as stable as possible.</p>
+
+<p>The community’s <a href="https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E">merge plan</a> focuses initially on the bounded/batch processing features mentioned above and follows the following approach to ensure a smooth integration:</p>
+
+<ul>
+ <li>
+ <p>To merge Blink’s <em>SQL/Table API query processor</em> enhancements, we exploit the fact that both Flink and Blink have the same APIs: SQL and the Table API.
+Following some restructuring of the Table/SQL module (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions">FLIP-32</a>) we plan to merge the Blink query planner (optimizer) and runtime (operators) as an additional query processor next to the current SQL runtime. Think of it as two different runners for the same APIs.<br />
+Initially, users will be able to select which query processor to use. After a transition period in which the new query processor will be developed to subsume the current query processor, the current processor will most likely be deprecated and eventually dropped. Given that SQL is such a well defined interface, we anticipate that this transition has little friction for users. Mostly a pleasant surprise to have broader SQL feature coverage and a boost in performance.</p>
+ </li>
+ <li>
+ <p>To support the merge of Blink’s <em>enhancements to scheduling and recovery</em> for jobs on bounded data, the Flink community is already working on refactoring its current schedule and adding support for <a href="https://issues.apache.org/jira/browse/FLINK-10429">pluggable scheduling and fail-over strategies</a>.<br />
+Once this effort is finished, we can add Blink’s scheduling and recovery strategies as a new scheduling strategy that is used by the new query processor. Eventually, we plan to use the new scheduling strategy also for bounded DataStream programs.</p>
+ </li>
+ <li>
+ <p>The extended catalog support, DDL support, as well as support for Hive’s catalog and integrations is currently going through separate design discussions. We plan to leverage existing code here whenever it makes sense.</p>
+ </li>
+</ul>
+
+<h2 id="summary">Summary</h2>
+
+<p>We believe that the data processing stack of the future is based on stream processing: The elegance of stream processing with its ability to model offline processing (batch), real-time data processing, and event-driven applications in the same way, while offering high performance and consistency is simply too compelling.</p>
+
+<p>Exploiting certain properties of bounded data is important for a stream processor to achieve the same performance as dedicated batch processors. While Flink always supported batch processing, the project is taking the next step in building a unified runtime and towards <strong>becoming a stream processor that is competitive with batch processing systems even on their home turf: OLAP SQL.</strong> The contribution of Alibaba’s Blink code helps the Flink community to p [...]
+</description>
+<pubDate>Wed, 13 Feb 2019 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2019/02/13/blink-flink-merge.html</link>
+<guid isPermaLink="true">/news/2019/02/13/blink-flink-merge.html</guid>
+</item>
+
+<item>
<title>Apache Flink 1.5.6 Released</title>
<description><p>The Apache Flink community released the sixth and last bugfix version of the Apache Flink 1.5 series.</p>
diff --git a/content/blog/index.html b/content/blog/index.html
index e4424f2..68e76b1 100644
--- a/content/blog/index.html
+++ b/content/blog/index.html
@@ -155,6 +155,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2019/02/13/blink-flink-merge.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></h2>
+
+ <p>13 Feb 2019
+ Stephan Ewen (<a href="https://twitter.com/stephanewen">@stephanewen</a>), Fabian Hueske (<a href="https://twitter.com/fhueske">@fhueske</a>), & Xiaowei Jiang (<a href="https://twitter.com/XiaoweiJ">@XiaoweiJ</a>)</p>
+
+ <p>A few weeks ago, Alibaba contributed its Flink-fork 'Blink' back to Apache Flink. In this blog post we discuss how Blink's features will help the Flink community to make a big step towards its vision to build a truly unified system for stream and batch processing.</p>
+
+ <p><a href="/news/2019/02/13/blink-flink-merge.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2018/12/26/release-1.5.6.html">Apache Flink 1.5.6 Released</a></h2>
<p>26 Dec 2018
@@ -291,21 +304,6 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2018/08/09/release-1.6.0.html">Apache Flink 1.6.0 Release Announcement</a></h2>
-
- <p>09 Aug 2018
- Till Rohrmann (<a href="https://twitter.com/stsffap">@stsffap</a>)</p>
-
- <p><p>The Apache Flink community is proud to announce the 1.6.0 release. Over the past 2 months, the Flink community has worked hard to resolve more than 360 issues. Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12342760">complete changelog</a> for more details.</p>
-
-</p>
-
- <p><a href="/news/2018/08/09/release-1.6.0.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -334,9 +332,24 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
- <h2>2018</h2>
+ <h2>2019</h2>
+
+ <ul id="markdown-toc">
+
+ <li><a href="/news/2019/02/13/blink-flink-merge.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></li>
+
+
+ </ul>
+ <hr>
+ <h2>2018</h2>
<ul id="markdown-toc">
+
+
+
+
+
+
<li><a href="/news/2018/12/26/release-1.5.6.html">Apache Flink 1.5.6 Released</a></li>
diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html
index 7dc290f..173d192 100644
--- a/content/blog/page2/index.html
+++ b/content/blog/page2/index.html
@@ -155,6 +155,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2018/08/09/release-1.6.0.html">Apache Flink 1.6.0 Release Announcement</a></h2>
+
+ <p>09 Aug 2018
+ Till Rohrmann (<a href="https://twitter.com/stsffap">@stsffap</a>)</p>
+
+ <p><p>The Apache Flink community is proud to announce the 1.6.0 release. Over the past 2 months, the Flink community has worked hard to resolve more than 360 issues. Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12342760">complete changelog</a> for more details.</p>
+
+</p>
+
+ <p><a href="/news/2018/08/09/release-1.6.0.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2018/07/31/release-1.5.2.html">Apache Flink 1.5.2 Released</a></h2>
<p>31 Jul 2018
@@ -283,23 +298,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2017/12/12/release-1.4.0.html">Apache Flink 1.4.0 Release Announcement</a></h2>
-
- <p>12 Dec 2017
- Aljoscha Krettek (<a href="https://twitter.com/aljoscha">@aljoscha</a>) & Mike Winters (<a href="https://twitter.com/wints">@wints</a>)</p>
-
- <p><p>The Apache Flink community is pleased to announce the 1.4.0 release. Over the past 5 months, the
-Flink community has been working hard to resolve more than 900 issues. See the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12340533">complete changelog</a>
-for more detail.</p>
-
-</p>
-
- <p><a href="/news/2017/12/12/release-1.4.0.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -328,9 +326,24 @@ for more detail.</p>
- <h2>2018</h2>
+ <h2>2019</h2>
+
+ <ul id="markdown-toc">
+
+ <li><a href="/news/2019/02/13/blink-flink-merge.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></li>
+
+
+ </ul>
+ <hr>
+ <h2>2018</h2>
<ul id="markdown-toc">
+
+
+
+
+
+
<li><a href="/news/2018/12/26/release-1.5.6.html">Apache Flink 1.5.6 Released</a></li>
diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html
index 9d9e5e3..1578f50 100644
--- a/content/blog/page3/index.html
+++ b/content/blog/page3/index.html
@@ -155,6 +155,23 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2017/12/12/release-1.4.0.html">Apache Flink 1.4.0 Release Announcement</a></h2>
+
+ <p>12 Dec 2017
+ Aljoscha Krettek (<a href="https://twitter.com/aljoscha">@aljoscha</a>) & Mike Winters (<a href="https://twitter.com/wints">@wints</a>)</p>
+
+ <p><p>The Apache Flink community is pleased to announce the 1.4.0 release. Over the past 5 months, the
+Flink community has been working hard to resolve more than 900 issues. See the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12340533">complete changelog</a>
+for more detail.</p>
+
+</p>
+
+ <p><a href="/news/2017/12/12/release-1.4.0.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2017/11/22/release-1.4-and-1.5-timeline.html">Looking Ahead to Apache Flink 1.4.0 and 1.5.0</a></h2>
<p>22 Nov 2017
@@ -287,21 +304,6 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2017/03/23/release-1.1.5.html">Apache Flink 1.1.5 Released</a></h2>
-
- <p>23 Mar 2017
- </p>
-
- <p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series.</p>
-
-</p>
-
- <p><a href="/news/2017/03/23/release-1.1.5.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -330,10 +332,25 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community
- <h2>2018</h2>
+ <h2>2019</h2>
<ul id="markdown-toc">
+ <li><a href="/news/2019/02/13/blink-flink-merge.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></li>
+
+
+
+ </ul>
+ <hr>
+ <h2>2018</h2>
+ <ul id="markdown-toc">
+
+
+
+
+
+
+
<li><a href="/news/2018/12/26/release-1.5.6.html">Apache Flink 1.5.6 Released</a></li>
diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html
index a31e10a..a066bfe 100644
--- a/content/blog/page4/index.html
+++ b/content/blog/page4/index.html
@@ -155,6 +155,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2017/03/23/release-1.1.5.html">Apache Flink 1.1.5 Released</a></h2>
+
+ <p>23 Mar 2017
+ </p>
+
+ <p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series.</p>
+
+</p>
+
+ <p><a href="/news/2017/03/23/release-1.1.5.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2017/02/06/release-1.2.0.html">Announcing Apache Flink 1.2.0</a></h2>
<p>06 Feb 2017 by Robert Metzger
@@ -284,21 +299,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2016/05/11/release-1.0.3.html">Flink 1.0.3 Released</a></h2>
-
- <p>11 May 2016
- </p>
-
- <p><p>Today, the Flink community released Flink version <strong>1.0.3</strong>, the third bugfix release of the 1.0 series.</p>
-
-</p>
-
- <p><a href="/news/2016/05/11/release-1.0.3.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -327,10 +327,25 @@
- <h2>2018</h2>
+ <h2>2019</h2>
<ul id="markdown-toc">
+ <li><a href="/news/2019/02/13/blink-flink-merge.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></li>
+
+
+
+ </ul>
+ <hr>
+ <h2>2018</h2>
+ <ul id="markdown-toc">
+
+
+
+
+
+
+
<li><a href="/news/2018/12/26/release-1.5.6.html">Apache Flink 1.5.6 Released</a></li>
diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html
index 3d64255..d84210d 100644
--- a/content/blog/page5/index.html
+++ b/content/blog/page5/index.html
@@ -155,6 +155,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2016/05/11/release-1.0.3.html">Flink 1.0.3 Released</a></h2>
+
+ <p>11 May 2016
+ </p>
+
+ <p><p>Today, the Flink community released Flink version <strong>1.0.3</strong>, the third bugfix release of the 1.0 series.</p>
+
+</p>
+
+ <p><a href="/news/2016/05/11/release-1.0.3.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2016/04/22/release-1.0.2.html">Flink 1.0.2 Released</a></h2>
<p>22 Apr 2016
@@ -282,21 +297,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2015/11/27/release-0.10.1.html">Flink 0.10.1 released</a></h2>
-
- <p>27 Nov 2015
- </p>
-
- <p><p>Today, the Flink community released the first bugfix release of the 0.10 series of Flink.</p>
-
-</p>
-
- <p><a href="/news/2015/11/27/release-0.10.1.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -325,10 +325,25 @@
- <h2>2018</h2>
+ <h2>2019</h2>
<ul id="markdown-toc">
+ <li><a href="/news/2019/02/13/blink-flink-merge.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></li>
+
+
+
+ </ul>
+ <hr>
+ <h2>2018</h2>
+ <ul id="markdown-toc">
+
+
+
+
+
+
+
<li><a href="/news/2018/12/26/release-1.5.6.html">Apache Flink 1.5.6 Released</a></li>
diff --git a/content/blog/page6/index.html b/content/blog/page6/index.html
index 905b2cf..167f9fa 100644
--- a/content/blog/page6/index.html
+++ b/content/blog/page6/index.html
@@ -155,6 +155,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2015/11/27/release-0.10.1.html">Flink 0.10.1 released</a></h2>
+
+ <p>27 Nov 2015
+ </p>
+
+ <p><p>Today, the Flink community released the first bugfix release of the 0.10 series of Flink.</p>
+
+</p>
+
+ <p><a href="/news/2015/11/27/release-0.10.1.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2015/11/16/release-0.10.0.html">Announcing Apache Flink 0.10.0</a></h2>
<p>16 Nov 2015
@@ -297,21 +312,6 @@ release is a preview release that contains known issues.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2015/04/07/march-in-flink.html">March 2015 in the Flink community</a></h2>
-
- <p>07 Apr 2015
- </p>
-
- <p><p>March has been a busy month in the Flink community.</p>
-
-</p>
-
- <p><a href="/news/2015/04/07/march-in-flink.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -340,10 +340,25 @@ release is a preview release that contains known issues.</p>
- <h2>2018</h2>
+ <h2>2019</h2>
<ul id="markdown-toc">
+ <li><a href="/news/2019/02/13/blink-flink-merge.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></li>
+
+
+
+ </ul>
+ <hr>
+ <h2>2018</h2>
+ <ul id="markdown-toc">
+
+
+
+
+
+
+
<li><a href="/news/2018/12/26/release-1.5.6.html">Apache Flink 1.5.6 Released</a></li>
diff --git a/content/blog/page7/index.html b/content/blog/page7/index.html
index ff5a783..ced1bae 100644
--- a/content/blog/page7/index.html
+++ b/content/blog/page7/index.html
@@ -155,6 +155,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2015/04/07/march-in-flink.html">March 2015 in the Flink community</a></h2>
+
+ <p>07 Apr 2015
+ </p>
+
+ <p><p>March has been a busy month in the Flink community.</p>
+
+</p>
+
+ <p><a href="/news/2015/04/07/march-in-flink.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html">Peeking into Apache Flink's Engine Room</a></h2>
<p>13 Mar 2015 by Fabian Hüske (<a href="https://twitter.com/">@fhueske</a>)
@@ -294,21 +309,6 @@ and offers a new API including definition of flexible windows.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2014/09/26/release-0.6.1.html">Apache Flink 0.6.1 available</a></h2>
-
- <p>26 Sep 2014
- </p>
-
- <p><p>We are happy to announce the availability of Flink 0.6.1.</p>
-
-</p>
-
- <p><a href="/news/2014/09/26/release-0.6.1.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -337,10 +337,25 @@ and offers a new API including definition of flexible windows.</p>
- <h2>2018</h2>
+ <h2>2019</h2>
<ul id="markdown-toc">
+ <li><a href="/news/2019/02/13/blink-flink-merge.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></li>
+
+
+
+ </ul>
+ <hr>
+ <h2>2018</h2>
+ <ul id="markdown-toc">
+
+
+
+
+
+
+
<li><a href="/news/2018/12/26/release-1.5.6.html">Apache Flink 1.5.6 Released</a></li>
diff --git a/content/blog/page8/index.html b/content/blog/page8/index.html
index 8cb4952..f546a04 100644
--- a/content/blog/page8/index.html
+++ b/content/blog/page8/index.html
@@ -155,6 +155,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2014/09/26/release-0.6.1.html">Apache Flink 0.6.1 available</a></h2>
+
+ <p>26 Sep 2014
+ </p>
+
+ <p><p>We are happy to announce the availability of Flink 0.6.1.</p>
+
+</p>
+
+ <p><a href="/news/2014/09/26/release-0.6.1.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2014/08/26/release-0.6.html">Apache Flink 0.6 available</a></h2>
<p>26 Aug 2014
@@ -200,9 +215,24 @@ academic and open source project that Flink originates from.</p>
- <h2>2018</h2>
+ <h2>2019</h2>
+
+ <ul id="markdown-toc">
+
+ <li><a href="/news/2019/02/13/blink-flink-merge.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></li>
+
+
+ </ul>
+ <hr>
+ <h2>2018</h2>
<ul id="markdown-toc">
+
+
+
+
+
+
<li><a href="/news/2018/12/26/release-1.5.6.html">Apache Flink 1.5.6 Released</a></li>
diff --git a/content/img/blog/unified-batch-streaming-blink/blink-flink-tpch.png b/content/img/blog/unified-batch-streaming-blink/blink-flink-tpch.png
new file mode 100644
index 0000000..77e6354
Binary files /dev/null and b/content/img/blog/unified-batch-streaming-blink/blink-flink-tpch.png differ
diff --git a/content/img/blog/unified-batch-streaming-blink/blink-spark-tpcds.png b/content/img/blog/unified-batch-streaming-blink/blink-spark-tpcds.png
new file mode 100644
index 0000000..e6ef8bc
Binary files /dev/null and b/content/img/blog/unified-batch-streaming-blink/blink-spark-tpcds.png differ
diff --git a/content/img/blog/unified-batch-streaming-blink/bounded-unbounded.png b/content/img/blog/unified-batch-streaming-blink/bounded-unbounded.png
new file mode 100644
index 0000000..29dfe8a
Binary files /dev/null and b/content/img/blog/unified-batch-streaming-blink/bounded-unbounded.png differ
diff --git a/content/img/blog/unified-batch-streaming-blink/sort-performance.png b/content/img/blog/unified-batch-streaming-blink/sort-performance.png
new file mode 100644
index 0000000..c1e60ba
Binary files /dev/null and b/content/img/blog/unified-batch-streaming-blink/sort-performance.png differ
diff --git a/content/img/blog/unified-batch-streaming-blink/stream-batch-joins.png b/content/img/blog/unified-batch-streaming-blink/stream-batch-joins.png
new file mode 100644
index 0000000..3591ae7
Binary files /dev/null and b/content/img/blog/unified-batch-streaming-blink/stream-batch-joins.png differ
diff --git a/content/index.html b/content/index.html
index 84718f1..e31d38c 100644
--- a/content/index.html
+++ b/content/index.html
@@ -433,6 +433,9 @@
<dl>
+ <dt> <a href="/news/2019/02/13/blink-flink-merge.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></dt>
+ <dd>A few weeks ago, Alibaba contributed its Flink-fork 'Blink' back to Apache Flink. In this blog post we discuss how Blink's features will help the Flink community to make a big step towards its vision to build a truly unified system for stream and batch processing.</dd>
+
<dt> <a href="/news/2018/12/26/release-1.5.6.html">Apache Flink 1.5.6 Released</a></dt>
<dd><p>The Apache Flink community released the sixth and last bugfix version of the Apache Flink 1.5 series.</p>
@@ -454,11 +457,6 @@ The latest release includes more than 420 resolved issues and some exciting addi
Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12343585">complete changelog</a> for more details.</p>
</dd>
-
- <dt> <a href="/news/2018/10/29/release-1.6.2.html">Apache Flink 1.6.2 Released</a></dt>
- <dd><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.6 series.</p>
-
-</dd>
</dl>
diff --git a/content/news/2019/02/13/blink-flink-merge.html b/content/news/2019/02/13/blink-flink-merge.html
new file mode 100644
index 0000000..8a4fe6e
--- /dev/null
+++ b/content/news/2019/02/13/blink-flink-merge.html
@@ -0,0 +1,352 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>Apache Flink: Batch as a Special Case of Streaming and Alibaba's contribution of Blink</title>
+ <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
+ <link rel="icon" href="/favicon.ico" type="image/x-icon">
+
+ <!-- Bootstrap -->
+ <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css">
+ <link rel="stylesheet" href="/css/flink.css">
+ <link rel="stylesheet" href="/css/syntax.css">
+
+ <!-- Blog RSS feed -->
+ <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml" title="Apache Flink Blog: RSS feed" />
+
+ <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
+ <!-- We need to load Jquery in the header for custom google analytics event tracking-->
+ <script src="/js/jquery.min.js"></script>
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ </head>
+ <body>
+
+
+ <!-- Main content. -->
+ <div class="container">
+ <div class="row">
+
+
+ <div id="sidebar" class="col-sm-3">
+
+
+<!-- Top navbar. -->
+ <nav class="navbar navbar-default">
+ <!-- The logo. -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <div class="navbar-logo">
+ <a href="/">
+ <img alt="Apache Flink" src="/img/flink-header-logo.svg" width="147px" height="73px">
+ </a>
+ </div>
+ </div><!-- /.navbar-header -->
+
+ <!-- The navigation links. -->
+ <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-main">
+
+ <!-- First menu section explains visitors what Flink is -->
+
+ <!-- What is Stream Processing? -->
+ <!--
+ <li><a href="/streamprocessing1.html">What is Stream Processing?</a></li>
+ -->
+
+ <!-- What is Flink? -->
+ <li><a href="/flink-architecture.html">What is Apache Flink?</a></li>
+
+ <!-- Use cases -->
+ <li><a href="/usecases.html">Use Cases</a></li>
+
+ <!-- Powered by -->
+ <li><a href="/poweredby.html">Powered By</a></li>
+
+ <!-- FAQ -->
+ <li><a href="/faq.html">FAQ</a></li>
+
+
+ <!-- Second menu section aims to support Flink users -->
+
+ <!-- Downloads -->
+ <li><a href="/downloads.html">Downloads</a></li>
+
+ <!-- Quickstart -->
+ <li>
+ <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/quickstart/setup_quickstart.html" target="_blank">Tutorials <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+ <!-- Documentation -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7" target="_blank">1.7 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-master" target="_blank">1.8 (Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ </ul>
+ </li>
+
+ <!-- getting help -->
+ <li><a href="/gettinghelp.html">Getting Help</a></li>
+
+ <!-- Blog -->
+ <li class="active"><a href="/blog/"><b>Flink Blog</b></a></li>
+
+
+
+ <!-- Third menu section aim to support community and contributors -->
+
+ <!-- Community -->
+ <li><a href="/community.html">Community & Project Info</a></li>
+
+ <!-- Contribute -->
+ <li><a href="/how-to-contribute.html">How to Contribute</a></li>
+
+ <!-- GitHub -->
+ <li>
+ <a href="https://github.com/apache/flink" target="_blank">Flink on GitHub <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+
+
+ <!-- Language Switcher -->
+ <li>
+
+
+ <a href="/zh/news/2019/02/13/blink-flink-merge.html">中文版</a>
+
+
+ </li>
+
+ </ul>
+
+ <ul class="nav navbar-nav navbar-bottom">
+ <hr />
+
+ <!-- Twitter -->
+ <li><a href="https://twitter.com/apacheflink" target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <!-- Visualizer -->
+ <li class=" hidden-md hidden-sm"><a href="/visualizer/" target="_blank">Plan Visualizer <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </nav>
+ </div>
+ <div class="col-sm-9">
+ <div class="row-fluid">
+ <div class="col-sm-12">
+ <div class="row">
+ <h1>Batch as a Special Case of Streaming and Alibaba's contribution of Blink</h1>
+
+ <article>
+ <p>13 Feb 2019 Stephan Ewen (<a href="https://twitter.com/stephanewen">@stephanewen</a>), Fabian Hueske (<a href="https://twitter.com/fhueske">@fhueske</a>), & Xiaowei Jiang (<a href="https://twitter.com/XiaoweiJ">@XiaoweiJ</a>)</p>
+
+<p>Last week, we <a href="https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E">broke the news</a> that Alibaba decided to contribute its Flink-fork, called Blink, back to the Apache Flink project. Why is that a big thing for Flink, what will it mean for users and the community, and how does it fit into Flink’s overall vision? Let’s take a step back to understand this better…</p>
+
+<h2 id="a-unified-approach-to-batch-and-streaming">A Unified Approach to Batch and Streaming</h2>
+
+<p>Since its early days, Apache Flink has followed the philosophy of taking a unified approach to batch and streaming data processing. The core building block is <em>“continuous processing of unbounded data streams”</em>: if you can do that, you can also do offline processing of bounded data sets (batch processing use cases), because these are just streams that happen to end at some point.</p>
+
+<center>
+<img src="/img/blog/unified-batch-streaming-blink/bounded-unbounded.png" width="600px" alt="Processing of bounded and unbounded data." />
+</center>
+<p><br /></p>
+
+<p>The <em>“streaming first, with batch as a special case of streaming”</em> philosophy is supported by various projects (for example <a href="https://flink.apache.org">Flink</a>, <a href="https://beam.apache.org">Beam</a>, etc.) and often been cited as a powerful way to build data applications that <a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101">generalize across real-time and offline processing</a> and to help greatly reduce the complexity of data infrastru [...]
+
+<h3 id="why-are-there-still-batch-processors">Why are there still batch processors?</h3>
+
+<p>However, <em>“batch is just a special case of streaming”</em> does not mean that any stream processor is now the right tool for your batch processing use cases - the introduction of stream processors did not render batch processors obsolete:</p>
+
+<ul>
+ <li>
+ <p>Pure stream processing systems are very slow at batch processing workloads. No one would consider it a good idea to use a stream processor that shuffles through message queues to analyze large amounts of available data.</p>
+ </li>
+ <li>
+ <p>Unified APIs like <a href="https://beam.apache.org">Apache Beam</a> often delegate to different runtimes depending on whether the data is continuous/unbounded of fix/bounded. For example, the implementations of the batch and streaming runtime of Google Cloud Dataflow are different, to get the desired performance and resilience in each case.</p>
+ </li>
+ <li>
+ <p><em>Apache Flink</em> has a streaming API that can do bounded/unbounded use cases, but still offers a separate DataSet API and runtime stack that is faster for batch use cases.</p>
+ </li>
+</ul>
+
+<p>What is the reason for the above? Where did <em>“batch is just a special case of streaming”</em> go wrong?</p>
+
+<p>The answer is simple, nothing is wrong with that paradigm. Unifying batch and streaming in the API is one aspect. One needs to also exploit certain characteristics of the special case “bounded data” in the runtime to competitively handle batch processing use cases. After all, batch processors have been built specifically for that special case.</p>
+
+<h2 id="batch-on-top-of-a-streaming-runtime">Batch on top of a Streaming Runtime</h2>
+
+<p>We always believed that it is possible to have a runtime that is state-of-the-art for both stream processing and batch processing use cases at the same time. A runtime that is streaming-first, but can exploit just the right amount of special properties of bounded streams to be as fast for batch use cases as dedicated batch processors. <strong>This is the unique approach that Flink takes.</strong></p>
+
+<p>Apache Flink has a network stack that supports both <a href="https://www.ververica.com/flink-forward-berlin/resources/improving-throughput-and-latency-with-flinks-network-stack">low-latency/high-throughput streaming data exchanges</a>, as well as high-throughput batch shuffles. Flink has streaming runtime operators for many operations, but also specialized operators for bounded inputs, which get used when you choose the DataSet API or select the batch environment in the Table API.</p>
+
+<center>
+<img src="/img/blog/unified-batch-streaming-blink/stream-batch-joins.png" width="500px" alt="Streaming and batch joins." />
+<br />
+<i>The figure illustrates a streaming join and a batch join. The batch join can read one input fully into a hash table and then probe with the other input. The stream join needs to build tables for both sides, because it needs to continuously process both inputs.
+For data larger than memory, the batch join can partition both data sets into subsets that fit in memory (data hits disk once) whereas the continuous nature of the stream join requires it to always keep all data in the table and repeatedly hit disk on cache misses.</i>
+</center>
+<p><br /></p>
+
+<p>Because of that, Apache Flink has been actually demonstrating some pretty impressive batch processing performance since its early days. The below benchmark is a bit older, but validated our architectural approach early on.</p>
+
+<center>
+<img src="/img/blog/unified-batch-streaming-blink/sort-performance.png" width="500px" alt="Sort performance." />
+<br />
+<i>Time to sort 3.2 TB (80 GB/node), in seconds<br />
+(<a href="https://www.slideshare.net/FlinkForward/dongwon-kim-a-comparative-performance-evaluation-of-flink" target="blank">Presentation by Dongwon Kim, Flink Forward Berlin 2015</a>.)</i>
+</center>
+<p><br /></p>
+
+<h2 id="what-is-still-missing">What is still missing?</h2>
+
+<p>To conclude the approach and make Flink’s experience on bounded data (batch) state-of-the-art, we need to add a few more enhancements. We believe that these features are key to realizing our vision:</p>
+
+<p><strong>(1) A truly unified runtime operator stack</strong>: Currently the bounded and unbounded operators have a different network and threading model and don’t mix and match. The original reason was that batch operators followed a “pull model” (easier for batch algorithms), while streaming operators followed a “push model” (better latency/throughput characteristics). In a unified stack, continuous streaming operators are the foundation. When operating on bounded data without latency [...]
+
+<p><strong>(2) Exploiting bounded streams to reduce the scope of fault tolerance</strong>: When input data is bounded, it is possible to completely buffer data during shuffles (memory or disk) and replay that data after a failure. This makes recovery more fine grained and thus much more efficient.</p>
+
+<p><strong>(3) Exploiting bounded stream operator properties for scheduling</strong>: A continuous unbounded streaming application needs (by definition) all operators running at the same time. An application on bounded data can schedule operations after another, depending on how the operators consume data (e.g., first build hash table, then probe hash table). This increases resource efficiency.</p>
+
+<p><strong>(4) Enabling these special case optimizations for the DataStream API</strong>: Currently, only the Table API (which is unified across bounded/unbounded streams) activates these optimizations when working on bounded data.</p>
+
+<p><strong>(5) Performance and coverage for SQL</strong>: SQL is the de-facto standard data language, and while it is also being rapidly adopted for continuous streaming use cases, there is absolutely no way past it for bounded/batch use cases. To be competitive with the best batch engines, Flink needs more coverage and performance for the SQL query execution. While the core data-plane in Flink is high performance, the speed of SQL execution ultimately depends a lot also on optimizer rul [...]
+
+<h2 id="enter-blink">Enter Blink</h2>
+
+<p>Blink is a fork of Apache Flink, originally created inside Alibaba to improve Flink’s behavior for internal use cases. Blink adds a series of improvements and integrations (see the <a href="https://github.com/apache/flink/blob/blink/README.md">Readme</a> for details), many of which fall into the category of improved bounded-data/batch processing and SQL. In fact, of the above list of features for a unified batch/streaming system, Blink implements significant steps forward in all excep [...]
+
+<p><strong>Unified Stream Operators:</strong> Blink extends the Flink streaming runtime operator model to support selectively reading from different inputs, while keeping the push model for very low latency. This control over the inputs helps to now support algorithms like hybrid hash-joins on the same operator and threading model as continuous symmetric joins through RocksDB. These operators also form the basis for future features like <a href="https://cwiki.apache.org/confluence/displa [...]
+
+<p><strong>Table API & SQL Query Processor:</strong> The SQL query processor is the component that evolved the changed most compared to the latest Flink master branch:</p>
+
+<ul>
+ <li>
+ <p>While Flink currently translates queries either into DataSet or DataStream programs (depending on the characteristics of their inputs), Blink translates queries to a data flow of the aforementioned stream operators.</p>
+ </li>
+ <li>
+ <p>Blink adds many more runtime operators for common SQL operations like semi-joins, anti-joins, etc.</p>
+ </li>
+ <li>
+ <p>The query planner (optimizer) is still based on Apache Calcite, but has many more optimization rules (incl. join reordering) and uses a proper cost model for planning.</p>
+ </li>
+ <li>
+ <p>Stream operators are more aggressively chained.</p>
+ </li>
+ <li>
+ <p>The common data structures (sorters, hash tables) and serializers are extended to go even further in operating on binary data and saving serialization overhead. Code generation is used for the row serializers.</p>
+ </li>
+</ul>
+
+<p><strong>Improved Scheduling and Failure Recovery:</strong> Finally, Blink implements several improvements for task scheduling and fault tolerance. The scheduling strategies use resources better by exploiting how the operators process their input data. The failover strategies recover more fine-grained along the boundaries of persistent shuffles. A failed JobManager can be replaced without restarting a running application.</p>
+
+<p>The changes in Blink result in a big improvement in performance. The below numbers were reported by the developers of Blink to give a rough impression of the performance gains.</p>
+
+<center>
+<img src="/img/blog/unified-batch-streaming-blink/blink-flink-tpch.png" width="600px" alt="TPC-H performance of Blink and Flink." />
+<br />
+<i>Relative performance of Blink versus Flink 1.6.0 in the TPC-H benchmark, query by query.<br />
+The performance improvement is in average 10x.<br />
+<a href="https://www.ververica.com/flink-forward-berlin/resources/unified-engine-for-data-processing-and-ai" target="blank">Presentation by Xiaowei Jiang at Flink Forward Berlin, 2018</a>.)</i>
+</center>
+<p><br /></p>
+
+<center>
+<img src="/img/blog/unified-batch-streaming-blink/blink-spark-tpcds.png" width="600px" alt="TPC-DS performace of Blink and Spark." />
+<br />
+<i>Performance of Blink versus Spark in the TPC-DS benchmark, aggregate time for all queries together.<br />
+<a href="https://www.bilibili.com/video/av42325467/?p=3" target="blank">Presentation by Xiaowei Jiang at Flink Forward Beijing, 2018</a>.</i>
+</center>
+<p><br /></p>
+
+<h2 id="how-do-we-plan-to-merge-blink-and-flink">How do we plan to merge Blink and Flink?</h2>
+
+<p>Blink’s code is currently available as a <a href="https://github.com/apache/flink/tree/blink">branch</a> in the Apache Flink repository. It is a challenge to merge a such big amount of changes, while making the merge process as non-disruptive as possible and keeping public APIs as stable as possible.</p>
+
+<p>The community’s <a href="https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E">merge plan</a> focuses initially on the bounded/batch processing features mentioned above and follows the following approach to ensure a smooth integration:</p>
+
+<ul>
+ <li>
+ <p>To merge Blink’s <em>SQL/Table API query processor</em> enhancements, we exploit the fact that both Flink and Blink have the same APIs: SQL and the Table API.
+Following some restructuring of the Table/SQL module (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions">FLIP-32</a>) we plan to merge the Blink query planner (optimizer) and runtime (operators) as an additional query processor next to the current SQL runtime. Think of it as two different runners for the same APIs.<br />
+Initially, users will be able to select which query processor to use. After a transition period in which the new query processor will be developed to subsume the current query processor, the current processor will most likely be deprecated and eventually dropped. Given that SQL is such a well defined interface, we anticipate that this transition has little friction for users. Mostly a pleasant surprise to have broader SQL feature coverage and a boost in performance.</p>
+ </li>
+ <li>
+ <p>To support the merge of Blink’s <em>enhancements to scheduling and recovery</em> for jobs on bounded data, the Flink community is already working on refactoring its current schedule and adding support for <a href="https://issues.apache.org/jira/browse/FLINK-10429">pluggable scheduling and fail-over strategies</a>.<br />
+Once this effort is finished, we can add Blink’s scheduling and recovery strategies as a new scheduling strategy that is used by the new query processor. Eventually, we plan to use the new scheduling strategy also for bounded DataStream programs.</p>
+ </li>
+ <li>
+ <p>The extended catalog support, DDL support, as well as support for Hive’s catalog and integrations is currently going through separate design discussions. We plan to leverage existing code here whenever it makes sense.</p>
+ </li>
+</ul>
+
+<h2 id="summary">Summary</h2>
+
+<p>We believe that the data processing stack of the future is based on stream processing: The elegance of stream processing with its ability to model offline processing (batch), real-time data processing, and event-driven applications in the same way, while offering high performance and consistency is simply too compelling.</p>
+
+<p>Exploiting certain properties of bounded data is important for a stream processor to achieve the same performance as dedicated batch processors. While Flink always supported batch processing, the project is taking the next step in building a unified runtime and towards <strong>becoming a stream processor that is competitive with batch processing systems even on their home turf: OLAP SQL.</strong> The contribution of Alibaba’s Blink code helps the Flink community to pick up the speed o [...]
+
+ </article>
+ </div>
+
+ <div class="row">
+ <div id="disqus_thread"></div>
+ <script type="text/javascript">
+ /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
+ var disqus_shortname = 'stratosphere-eu'; // required: replace example with your forum shortname
+
+ /* * * DON'T EDIT BELOW THIS LINE * * */
+ (function() {
+ var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
+ dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
+ (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
+ })();
+ </script>
+ </div>
+ </div>
+</div>
+ </div>
+ </div>
+
+ <hr />
+
+ <div class="row">
+ <div class="footer text-center col-sm-12">
+ <p>Copyright © 2014-2019 <a href="http://apache.org">The Apache Software Foundation</a>. All Rights Reserved.</p>
+ <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p>
+ <p><a href="/privacy-policy.html">Privacy Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p>
+ </div>
+ </div>
+ </div><!-- /.container -->
+
+ <!-- Include all compiled plugins (below), or include individual files as needed -->
+ <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery.matchHeight/0.7.0/jquery.matchHeight-min.js"></script>
+ <script src="/js/codetabs.js"></script>
+ <script src="/js/stickysidebar.js"></script>
+
+ <!-- Google Analytics -->
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-52545728-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+ </body>
+</html>
diff --git a/content/zh/index.html b/content/zh/index.html
index 4543d7f..30aa15f 100644
--- a/content/zh/index.html
+++ b/content/zh/index.html
@@ -431,6 +431,9 @@
<dl>
+ <dt> <a href="/news/2019/02/13/blink-flink-merge.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></dt>
+ <dd>A few weeks ago, Alibaba contributed its Flink-fork 'Blink' back to Apache Flink. In this blog post we discuss how Blink's features will help the Flink community to make a big step towards its vision to build a truly unified system for stream and batch processing.</dd>
+
<dt> <a href="/news/2018/12/26/release-1.5.6.html">Apache Flink 1.5.6 Released</a></dt>
<dd><p>The Apache Flink community released the sixth and last bugfix version of the Apache Flink 1.5 series.</p>
@@ -452,11 +455,6 @@ The latest release includes more than 420 resolved issues and some exciting addi
Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12343585">complete changelog</a> for more details.</p>
</dd>
-
- <dt> <a href="/news/2018/10/29/release-1.6.2.html">Apache Flink 1.6.2 Released</a></dt>
- <dd><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.6 series.</p>
-
-</dd>
</dl>